From b4423799269c9df5a2b0258055c1d213f4f57dcd Mon Sep 17 00:00:00 2001 From: Honglei Huang Date: Fri, 29 May 2026 10:23:17 +0800 Subject: [PATCH 001/464] drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2158267 commit 342981fff32802a819d6fc7cf3c9fedf9f3d9d60 upstream. Since commit c08972f55594 ("drm/amdgpu: fix amdgpu_hmm_range_get_pages") moved mmu_interval_read_begin() out of the per-chunk loop, the captured notifier_seq is no longer refreshed across retries. As a result, the existing -EBUSY retry path can never make progress: hmm_range_fault() returns -EBUSY only when mmu_interval_check_retry(notifier, notifier_seq) reports that the sequence is stale. Once the sequence has advanced, the stored seq will never match again, so every subsequent call within the same invocation returns -EBUSY immediately. The "goto retry" therefore degenerates into a busy spin that simply burns CPU for the full HMM_RANGE_DEFAULT_TIMEOUT (~1s) window before finally bailing out with -EAGAIN. This is pure latency with no chance of recovery, and it actively hurts the KFD userptr stack: the caller ends up blocked for a second while holding mmap_lock, only to return -EAGAIN to the restore worker (or to userspace) which would have re-driven the operation immediately anyway. Drop the retry/timeout entirely and let -EBUSY propagate straight to out_free_pfns, where it is already translated to -EAGAIN. Recovery is handled at a higher level: the KFD restore_userptr_worker reschedules itself, and the userptr ioctl path returns -EAGAIN to userspace. No functional regression: the previous behaviour on -EBUSY was already to fail with -EAGAIN after a 1s stall; we just skip the stall. Reviewed-by: Christian König Signed-off-by: Honglei Huang Signed-off-by: Alex Deucher Signed-off-by: Greg Kroah-Hartman Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c index 5bfa5a84b09cb..f75790a5c6c45 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c @@ -172,7 +172,6 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, const u64 max_bytes = SZ_2G; struct hmm_range *hmm_range = &range->hmm_range; - unsigned long timeout; unsigned long *pfns; unsigned long end; int r; @@ -199,15 +198,9 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, pr_debug("hmm range: start = 0x%lx, end = 0x%lx", hmm_range->start, hmm_range->end); - timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); - -retry: r = hmm_range_fault(hmm_range); - if (unlikely(r)) { - if (r == -EBUSY && !time_after(jiffies, timeout)) - goto retry; + if (unlikely(r)) goto out_free_pfns; - } if (hmm_range->end == end) break; From eac82d6e02dd9e3111d9daa333ac5546bf39ac5d Mon Sep 17 00:00:00 2001 From: Kuniyuki Iwashima Date: Mon, 1 Jun 2026 18:20:55 +0000 Subject: [PATCH 002/464] tcp: Add preempt_{disable,enable}_nested() in reqsk_queue_hash_req(). [ Upstream commit e10902df24488ca722303133acfc82490f7d59ad ] syzbot reported a weird reqsk->rsk_refcnt underflow in __inet_csk_reqsk_queue_drop(). The captured reqsk_put() in __inet_csk_reqsk_queue_drop() is called only when it successfully removes reqsk from ehash. Moreover, reqsk_timer_handler() calls another reqsk_put() after that. This indicates that the reqsk was missing both refcnts for ehash and the timer itself. Since all the syzbot reports had PREEMPT_RT enabled, the only possible scenario is that reqsk_queue_hash_req() is preempted after mod_timer() and before refcount_set(), and then the timer triggered after 1s aborts the reqsk due to its listener's close(). Let's wrap mod_timer() and refcount_set() with preempt_disable_nested() and preempt_enable_nested(). Note that inet_ehash_insert() holds the normal spin_lock() (mutex in PREEMPT_RT), so it must be called outside of preempt_disable_nested(), but this is fine. The lookup path just ignores 0 sk_refcnt entries in ehash and tries to create another reqsk, but this will fail at inet_ehash_insert(). [0]: refcount_t: underflow; use-after-free. WARNING: lib/refcount.c:28 at refcount_warn_saturate+0xb2/0x110 lib/refcount.c:28, CPU#0: ktimers/0/16 Modules linked in: CPU: 0 UID: 0 PID: 16 Comm: ktimers/0 Tainted: G L syzkaller #0 PREEMPT_{RT,(full)} Tainted: [L]=SOFTLOCKUP Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/18/2026 RIP: 0010:refcount_warn_saturate+0xb2/0x110 lib/refcount.c:28 Code: e4 7d d1 0a 67 48 0f b9 3a eb 4a e8 38 3d 23 fd 48 8d 3d e1 7d d1 0a 67 48 0f b9 3a eb 37 e8 25 3d 23 fd 48 8d 3d de 7d d1 0a <67> 48 0f b9 3a eb 24 e8 12 3d 23 fd 48 8d 3d db 7d d1 0a 67 48 0f RSP: 0000:ffffc90000157948 EFLAGS: 00010246 RAX: ffffffff84a1301b RBX: 0000000000000003 RCX: ffff88801ca98000 RDX: 0000000000000100 RSI: 0000000000000000 RDI: ffffffff8f72ae00 RBP: ffffffff99ae3b01 R08: ffff88801ca98000 R09: 0000000000000005 R10: 0000000000000100 R11: 0000000000000004 R12: ffff8880425ef568 R13: ffff8880425ef4f8 R14: ffff8880425ef578 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff888126386000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7b46710e9c CR3: 000000000dbb6000 CR4: 00000000003526f0 Call Trace: __refcount_sub_and_test include/linux/refcount.h:400 [inline] __refcount_dec_and_test include/linux/refcount.h:432 [inline] refcount_dec_and_test include/linux/refcount.h:450 [inline] reqsk_put include/net/request_sock.h:136 [inline] __inet_csk_reqsk_queue_drop+0x3ce/0x440 net/ipv4/inet_connection_sock.c:1007 reqsk_timer_handler+0x651/0xdf0 net/ipv4/inet_connection_sock.c:1137 call_timer_fn+0x192/0x5e0 kernel/time/timer.c:1748 expire_timers kernel/time/timer.c:1799 [inline] __run_timers kernel/time/timer.c:2374 [inline] __run_timer_base+0x6a3/0x9f0 kernel/time/timer.c:2386 run_timer_base kernel/time/timer.c:2395 [inline] run_timer_softirq+0x67/0x170 kernel/time/timer.c:2403 handle_softirqs+0x1de/0x6d0 kernel/softirq.c:622 __do_softirq kernel/softirq.c:656 [inline] run_ktimerd+0x69/0x100 kernel/softirq.c:1151 smpboot_thread_fn+0x541/0xa50 kernel/smpboot.c:160 kthread+0x388/0x470 kernel/kthread.c:436 ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245 Fixes: d2d6422f8bd1 ("x86: Allow to enable PREEMPT_RT.") Reported-by: syzbot+e809069bc15f26300526@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a1a7bcf.0a9e871e.332604.000b.GAE@google.com/ Signed-off-by: Kuniyuki Iwashima Reviewed-by: Eric Dumazet Reviewed-by: Sebastian Andrzej Siewior Link: https://patch.msgid.link/20260601182101.3183993-1-kuniyu@google.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53260 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/ipv4/inet_connection_sock.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index f1988fd503540..d9f6c8d4d7e63 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -1151,6 +1151,9 @@ static bool reqsk_queue_hash_req(struct request_sock *req) /* The timer needs to be setup after a successful insertion. */ req->timeout = tcp_timeout_init((struct sock *)req); timer_setup(&req->rsk_timer, reqsk_timer_handler, TIMER_PINNED); + + preempt_disable_nested(); + mod_timer(&req->rsk_timer, jiffies + req->timeout); /* before letting lookups find us, make sure all req fields @@ -1158,6 +1161,9 @@ static bool reqsk_queue_hash_req(struct request_sock *req) */ smp_wmb(); refcount_set(&req->rsk_refcnt, 2 + 1); + + preempt_enable_nested(); + return true; } From 44937621bf374440c45d1cb6111da8a22037eae3 Mon Sep 17 00:00:00 2001 From: Lorenzo Bianconi Date: Tue, 2 Jun 2026 11:21:05 +0200 Subject: [PATCH 003/464] net: ethernet: mtk_eth_soc: Fix use-after-free in metadata dst teardown [ Upstream commit 80df409e1a483676826a6c66e693dba6ac507751 ] mtk_free_dev() calls metadata_dst_free() which frees the metadata_dst with kfree() immediately, bypassing the RCU grace period. In the RX path, skb_dst_set_noref() sets a non-refcounted pointer from the skb to the metadata_dst. This function requires RCU read-side protection and the dst must remain valid until all RCU readers complete. Since metadata_dst_free() calls kfree() directly, a use-after-free can occur if any skb still holds a noref pointer to the dst when the driver tears it down. Replace metadata_dst_free() with dst_release() which properly goes through the refcount path: when the refcount drops to zero, it schedules the actual free via call_rcu_hurry(), ensuring all RCU readers have completed before the memory is freed. Fixes: 2d7605a72906 ("net: ethernet: mtk_eth_soc: enable hardware DSA untagging") Signed-off-by: Lorenzo Bianconi Link: https://patch.msgid.link/20260602-airoha-mtk-metadata-uaf-fix-v1-2-3aaa99d83351@kernel.org Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53247 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 796f79088f366..2eca2b2a964fe 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -4469,7 +4469,7 @@ static int mtk_free_dev(struct mtk_eth *eth) for (i = 0; i < ARRAY_SIZE(eth->dsa_meta); i++) { if (!eth->dsa_meta[i]) break; - metadata_dst_free(eth->dsa_meta[i]); + dst_release(ð->dsa_meta[i]->dst); } return 0; From ca0a57ec2613c15d1e2b467204e1b0b3aaf0ddda Mon Sep 17 00:00:00 2001 From: Xin Long Date: Mon, 1 Jun 2026 21:06:06 -0400 Subject: [PATCH 004/464] sctp: validate cached peer INIT chunk length in COOKIE_ECHO processing [ Upstream commit 0861615c28de668669d748ef4eb913ea9262d13b ] When a listening SCTP server processes a COOKIE_ECHO chunk, the cached peer INIT chunk embedded after the cookie is parsed and its parameters are later walked by sctp_process_init() using sctp_walk_params(). However, the chunk header length of this cached INIT chunk was not validated against the remaining buffer in the COOKIE_ECHO payload. If the length field is inflated, the parameter walk can run beyond the actual received data, leading to out-of-bounds reads and potential memory corruption during later parameter handling (e.g. STATE_COOKIE processing and kmemdup() copies). Add a bounds check in sctp_unpack_cookie() to ensure the cached INIT chunk length does not exceed the available data in the COOKIE_ECHO buffer before it is used. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Brian Geffon Signed-off-by: Xin Long Link: https://patch.msgid.link/eb60825fa22d6f9e663c7d4dbb69f397b5d34d42.1780362366.git.lucien.xin@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53246 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/sctp/sm_make_chunk.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c index 2c0017d058d40..9014b095f52dd 100644 --- a/net/sctp/sm_make_chunk.c +++ b/net/sctp/sm_make_chunk.c @@ -1730,6 +1730,7 @@ struct sctp_association *sctp_unpack_cookie( struct sctp_signed_cookie *cookie; struct sk_buff *skb = chunk->skb; struct sctp_cookie *bear_cookie; + struct sctp_chunkhdr *ch; enum sctp_scope scope; unsigned int len; ktime_t kt; @@ -1759,6 +1760,10 @@ struct sctp_association *sctp_unpack_cookie( cookie = chunk->subh.cookie_hdr; bear_cookie = &cookie->c; + ch = (struct sctp_chunkhdr *)(bear_cookie + 1); + if (ntohs(ch->length) > len - fixed_size) + goto malformed; + /* Verify the cookie's MAC, if cookie authentication is enabled. */ if (sctp_sk(ep->base.sk)->cookie_auth_enable) { u8 mac[SHA256_DIGEST_SIZE]; From 70b100e33629ea6e0e2466ba45d03e23a8cd31a8 Mon Sep 17 00:00:00 2001 From: Xin Long Date: Wed, 3 Jun 2026 14:11:44 -0400 Subject: [PATCH 005/464] sctp: purge outqueue on stale COOKIE-ECHO handling [ Upstream commit e374b22e9b07b72a25909621464ff74096151bfb ] sctp_stream_update() is only invoked when the association is moved into COOKIE_WAIT during association setup/reconfiguration. In this path, the outbound stream scheduler state (stream->out_curr) is expected to be clean, since no user data should have been transmitted yet unless the state machine has already partially progressed. However, a corner case exists in sctp_sf_do_5_2_6_stale(): when a Stale Cookie ERROR is received, the association is rolled back from COOKIE_ECHOED to COOKIE_WAIT. In this scenario, user data may already have been queued and even bundled with the COOKIE-ECHO chunk. During the rollback, sctp_stream_update() frees the old stream table and installs a new one, but it does not invalidate stream->out_curr. As a result, out_curr may still point to a freed sctp_stream_out entry from the previous stream state. Later, SCTP scheduler dequeue paths (FCFS, RR, PRIO, etc.) rely on stream->out_curr->ext, which can lead to use-after-free once the old stream state has been released via sctp_stream_free(). This results in crashes such as (reported by Yuqi): BUG: KASAN: slab-use-after-free in sctp_sched_fcfs_dequeue+0x13a/0x140 Read of size 8 at addr ff1100004d4d3208 by task mini_poc/9312 CPU: 1 UID: 1001 PID: 9312 Comm: mini_poc Not tainted 7.1.0-rc1-00305-gbd3a4795d574 #5 PREEMPT(full) sctp_sched_fcfs_dequeue+0x13a/0x140 sctp_outq_flush+0x1603/0x33e0 sctp_do_sm+0x31c9/0x5d30 sctp_assoc_bh_rcv+0x392/0x6f0 sctp_inq_push+0x1db/0x270 sctp_rcv+0x138d/0x3c10 Fix this by fully purging the association outqueue when handling the Stale Cookie case. This ensures all pending transmit and retransmit state is dropped, and any scheduler cached pointers are invalidated, making it safe to rebuild stream state during COOKIE_WAIT restart. Updating only stream->out_curr would be insufficient, since queued and retransmittable data would still reference the old stream state and trigger later use-after-free in dequeue paths. Fixes: 5bbbbe32a431 ("sctp: introduce stream scheduler foundations") Reported-by: Yuan Tan Reported-by: Yifan Wu Reported-by: Juefei Pu Reported-by: Zhengchuan Liang Reported-by: Xin Liu Reported-by: Yuqi Xu Reported-by: Ren Wei Signed-off-by: Xin Long Link: https://patch.msgid.link/94318159b9052907a6cbb7256aee8b5f8dfbfccb.1780510304.git.lucien.xin@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-52924 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/sctp/sm_statefuns.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c index 8e89a870780c4..9b23c11cbb9ea 100644 --- a/net/sctp/sm_statefuns.c +++ b/net/sctp/sm_statefuns.c @@ -2598,11 +2598,7 @@ static enum sctp_disposition sctp_sf_do_5_2_6_stale( */ sctp_add_cmd_sf(commands, SCTP_CMD_DEL_NON_PRIMARY, SCTP_NULL()); - /* If we've sent any data bundled with COOKIE-ECHO we will need to - * resend - */ - sctp_add_cmd_sf(commands, SCTP_CMD_T1_RETRAN, - SCTP_TRANSPORT(asoc->peer.primary_path)); + sctp_add_cmd_sf(commands, SCTP_CMD_PURGE_OUTQUEUE, SCTP_NULL()); /* Cast away the const modifier, as we want to just * rerun it through as a sideffect. From 43102eff6d4892f7b728b8f3470f669d3ffbade6 Mon Sep 17 00:00:00 2001 From: Kyle Zeng Date: Fri, 5 Jun 2026 00:34:48 -0700 Subject: [PATCH 006/464] ipv6: sit: reload inner IPv6 header after GSO offloads [ Upstream commit f0e42f0c4337b1f220de1ddd63f47197c7dee4de ] ipip6_tunnel_xmit() caches the inner IPv6 header pointer at function entry and continues using it after iptunnel_handle_offloads(). For GSO skbs, iptunnel_handle_offloads() calls skb_header_unclone(). When the skb header is cloned, skb_header_unclone() can call pskb_expand_head(), which may move the skb head. The pskb_expand_head() contract requires pointers into the skb header to be reloaded after the call. If the later skb_realloc_headroom() branch is not taken, SIT uses the stale iph6 pointer to read the inner hop limit and DS field. That can read from a freed skb head after the old head's remaining clone is released. Reload iph6 after the offload helper succeeds and before subsequent reads from the inner IPv6 header. Keep the existing reload after skb_realloc_headroom(), since that branch can also replace the skb. Fixes: 14909664e4e1 ("sit: Setup and TX path for sit/UDP foo-over-udp encapsulation") Signed-off-by: Kyle Zeng Reviewed-by: Eric Dumazet Reported-by: syzbot+6eb9ca986d80f6f88cf9@syzkaller.appspotmail.com Link: https://patch.msgid.link/20260605073448.6524-1-kylebot@openai.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53228 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/ipv6/sit.c | 1 + 1 file changed, 1 insertion(+) diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c index 6a7b8abb04772..7c317c105bdcb 100644 --- a/net/ipv6/sit.c +++ b/net/ipv6/sit.c @@ -960,6 +960,7 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb, ip_rt_put(rt); goto tx_error; } + iph6 = ipv6_hdr(skb); if (df) { mtu = dst4_mtu(&rt->dst) - t_hlen; From 4606b0af4bbf99e3c7792953925d359f98764d69 Mon Sep 17 00:00:00 2001 From: Michael Bommarito Date: Mon, 8 Jun 2026 08:22:34 -0400 Subject: [PATCH 007/464] sctp: fix uninit-value in __sctp_rcv_asconf_lookup() [ Upstream commit f8373d7090b745728de66308deeecc67e8d319ce ] __sctp_rcv_asconf_lookup() in net/sctp/input.c only checks that the ASCONF chunk can hold the ADDIP header and a parameter header, then calls af->from_addr_param(), which reads the full address (16 bytes for IPv6) trusting the parameter's declared length. An unauthenticated peer can send a truncated trailing ASCONF chunk that declares an IPv6 address parameter but stops after the 4-byte parameter header; reached from the no-association lookup path, from_addr_param() then reads uninitialized bytes past the parameter. Impact: an unauthenticated SCTP peer makes the receive path read up to 16 bytes of uninitialized memory past a truncated ASCONF address parameter. The sibling __sctp_rcv_init_lookup() bounds parameters with sctp_walk_params(); this path open-codes the fetch and omits the bound. Verify the whole address parameter lies within the chunk before from_addr_param() reads it, the same class of fix as commit 51e5ad549c43 ("net: sctp: fix KMSAN uninit-value in sctp_inq_pop"). Fixes: df2185771439 ("[SCTP]: Update association lookup to look at ASCONF chunks as well") Signed-off-by: Michael Bommarito Acked-by: Xin Long Link: https://patch.msgid.link/20260608122234.459098-1-michael.bommarito@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53225 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/sctp/input.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/net/sctp/input.c b/net/sctp/input.c index e119e460ccde0..864741fae4187 100644 --- a/net/sctp/input.c +++ b/net/sctp/input.c @@ -1204,6 +1204,14 @@ static struct sctp_association *__sctp_rcv_asconf_lookup( /* Skip over the ADDIP header and find the Address parameter */ param = (union sctp_addr_param *)(asconf + 1); + /* The whole address parameter must lie within the chunk before + * af->from_addr_param() reads the variable-length address; otherwise a + * truncated trailing ASCONF chunk lets it read uninitialized bytes past + * the parameter. + */ + if (sizeof(*asconf) + ntohs(param->p.length) > ntohs(ch->length)) + return NULL; + af = sctp_get_af_specific(param_type2af(param->p.type)); if (unlikely(!af)) return NULL; From 5892990b9129d0fb62f7441c08e910fef4cbd553 Mon Sep 17 00:00:00 2001 From: Xin Long Date: Sun, 7 Jun 2026 19:03:47 -0400 Subject: [PATCH 008/464] sctp: validate embedded INIT chunk and address list lengths in cookie [ Upstream commit 6f4c80a2a7e6d06753b89a578b710a2499a5e62b ] sctp_unpack_cookie() only checked that the embedded INIT chunk length did not exceed the remaining cookie payload, but did not ensure that the INIT chunk is large enough to contain a complete INIT header. A malformed COOKIE_ECHO can therefore carry a truncated INIT chunk whose length field is smaller than sizeof(struct sctp_init_chunk). Later, sctp_process_init() accesses INIT parameters unconditionally, which may lead to out-of-bounds reads. In addition, raw_addr_list_len is not fully validated against the remaining cookie payload. When cookie authentication is disabled, an attacker can supply an oversized raw_addr_list_len and cause sctp_raw_to_bind_addrs() to read beyond the end of the cookie. The address parser also lacks sufficient bounds checks for parameter headers and lengths, allowing malformed address parameters to trigger out-of-bounds reads. Fix this by: - requiring the embedded INIT chunk length to be at least sizeof(struct sctp_init_chunk); - validating that the INIT chunk and raw address list together fit within the cookie payload; - verifying sufficient data exists for each address parameter header and payload before parsing it. Note that sctp_verify_init() must be called after sctp_unpack_cookie() and before sctp_process_init() when cookie authentication is disabled. This will be addressed in a separate patch. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Reported-by: Sashiko Signed-off-by: Xin Long Link: https://patch.msgid.link/75af23a89adf881a0895d511775e4770da367cbf.1780873427.git.lucien.xin@gmail.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53224 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/sctp/bind_addr.c | 11 ++++++++++- net/sctp/sm_make_chunk.c | 9 +++++++-- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/net/sctp/bind_addr.c b/net/sctp/bind_addr.c index 75e3e61d494e0..31737f144c7f0 100644 --- a/net/sctp/bind_addr.c +++ b/net/sctp/bind_addr.c @@ -275,6 +275,16 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list, param = (struct sctp_paramhdr *)raw_addr_list; rawaddr = (union sctp_addr_param *)raw_addr_list; + if (addrs_len < sizeof(*param)) { + retval = -EINVAL; + goto out_err; + } + len = ntohs(param->length); + if (addrs_len < len) { + retval = -EINVAL; + goto out_err; + } + af = sctp_get_af_specific(param_type2af(param->type)); if (unlikely(!af) || !af->from_addr_param(&addr, rawaddr, htons(port), 0)) { @@ -291,7 +301,6 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list, goto out_err; next: - len = ntohs(param->length); addrs_len -= len; raw_addr_list += len; } diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c index 9014b095f52dd..51affa4fd396b 100644 --- a/net/sctp/sm_make_chunk.c +++ b/net/sctp/sm_make_chunk.c @@ -1731,8 +1731,8 @@ struct sctp_association *sctp_unpack_cookie( struct sk_buff *skb = chunk->skb; struct sctp_cookie *bear_cookie; struct sctp_chunkhdr *ch; + unsigned int len, chlen; enum sctp_scope scope; - unsigned int len; ktime_t kt; /* Header size is static data prior to the actual cookie, including @@ -1761,7 +1761,12 @@ struct sctp_association *sctp_unpack_cookie( bear_cookie = &cookie->c; ch = (struct sctp_chunkhdr *)(bear_cookie + 1); - if (ntohs(ch->length) > len - fixed_size) + chlen = ntohs(ch->length); + if (chlen < sizeof(struct sctp_init_chunk)) + goto malformed; + if (chlen > len - fixed_size) + goto malformed; + if (bear_cookie->raw_addr_list_len > len - fixed_size - chlen) goto malformed; /* Verify the cookie's MAC, if cookie authentication is enabled. */ From e699aa5a6cf5a2144b72e49eaa76836e6433d99d Mon Sep 17 00:00:00 2001 From: Eric Dumazet Date: Mon, 8 Jun 2026 16:46:13 +0000 Subject: [PATCH 009/464] ip6_vti: fix incorrect tunnel matching in vti6_tnl_lookup() [ Upstream commit a5c0359f5cbc51a2e2b114d6041e0f3c73f903e9 ] In vti6_tnl_lookup(), when an exact match for a tunnel fails, the code falls back to searching for wildcard tunnels: - Tunnels matching the packet's local address, with any remote address wildcard remote). - Tunnels matching the packet's remote address, with any local address (wildcard local). However, vti6 stores all these different types of tunnels in the same hash table (ip6n->tnls_r_l) prone to hash collisions. The bug is that the fallback search loops in vti6_tnl_lookup() were missing checks to ensure that the candidate tunnel actually has a wildcard address. Fixes: fbe68ee87522 ("vti6: Add a lookup method for tunnels with wildcard endpoints.") Signed-off-by: Eric Dumazet Cc: Steffen Klassert Reviewed-by: Nicolas Dichtel Link: https://patch.msgid.link/20260608164613.933023-1-edumazet@google.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-53221 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/ipv6/ip6_vti.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c index df793c8bfffb0..8b2c59c8eb27e 100644 --- a/net/ipv6/ip6_vti.c +++ b/net/ipv6/ip6_vti.c @@ -106,6 +106,7 @@ vti6_tnl_lookup(struct net *net, const struct in6_addr *remote, hash = HASH(&any, local); for_each_vti6_tunnel_rcu(ip6n->tnls_r_l[hash]) { if (ipv6_addr_equal(local, &t->parms.laddr) && + ipv6_addr_any(&t->parms.raddr) && (t->dev->flags & IFF_UP)) return t; } @@ -113,6 +114,7 @@ vti6_tnl_lookup(struct net *net, const struct in6_addr *remote, hash = HASH(remote, &any); for_each_vti6_tunnel_rcu(ip6n->tnls_r_l[hash]) { if (ipv6_addr_equal(remote, &t->parms.raddr) && + ipv6_addr_any(&t->parms.laddr) && (t->dev->flags & IFF_UP)) return t; } From 0f24947e200bb5c1e318bab43b99ee77349a3c7a Mon Sep 17 00:00:00 2001 From: Til Kaiser Date: Sun, 7 Jun 2026 15:49:41 +0200 Subject: [PATCH 010/464] net: mvpp2: limit XDP frame size to the RX buffer [ Upstream commit f3c6aa078927e6fe8121c9c591ddee8716c5305a ] mvpp2 has short and long BM pools, and short pool buffers can be smaller than PAGE_SIZE. The XDP path nevertheless initializes every xdp_buff with PAGE_SIZE as frame size. XDP helpers use frame_sz to validate tail growth and to derive the hard end of the data area. Advertising PAGE_SIZE for short buffers can let bpf_xdp_adjust_tail() grow a packet past the real allocation, corrupting memory or later tripping skb tailroom checks. Initialize the XDP buffer with bm_pool->frag_size so XDP tailroom matches the actual buffer backing the packet. Fixes: 07dd0a7aae7f ("mvpp2: add basic XDP support") Signed-off-by: Til Kaiser Link: https://patch.msgid.link/20260607134943.21996-3-mail@tk154.de Signed-off-by: Paolo Abeni Signed-off-by: Sasha Levin CVE-2026-53216 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c index f442b874bb593..a0f563e13720b 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c @@ -3978,7 +3978,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, else xdp_rxq = &rxq->xdp_rxq_long; - xdp_init_buff(&xdp, PAGE_SIZE, xdp_rxq); + xdp_init_buff(&xdp, bm_pool->frag_size, xdp_rxq); xdp_prepare_buff(&xdp, data, MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM, rx_bytes, true); From cbfc192fdc0d2a50bc162ec56764417649c4f9a6 Mon Sep 17 00:00:00 2001 From: Til Kaiser Date: Sun, 7 Jun 2026 15:49:42 +0200 Subject: [PATCH 011/464] net: mvpp2: refill RX buffers before XDP or skb use [ Upstream commit 5e8e2a9624df72fca7c736b2966b2cbf6c9c3ff6 ] The RX error path returns the current descriptor buffer to the hardware BM pool. That is only valid while the driver still owns the buffer. mvpp2_rx_refill() can fail after the current buffer has been handed to XDP or attached to an skb. In those cases mvpp2_run_xdp() may have recycled, redirected, or queued the page for XDP_TX, and an skb free also retires the data buffer. Returning such a buffer to BM lets hardware DMA into memory that is no longer owned by the RX ring. Refill the BM pool before handing the current buffer to XDP or to the skb. If the allocation fails there, drop the packet and return the still-owned current buffer to BM, preserving the pool depth. Once the refill succeeds, later local drops retire/free the current buffer instead of returning it to BM. Fixes: 07dd0a7aae7f ("mvpp2: add basic XDP support") Fixes: d6526926de73 ("net: mvpp2: fix memory leak in mvpp2_rx") Signed-off-by: Til Kaiser Link: https://patch.msgid.link/20260607134943.21996-4-mail@tk154.de Signed-off-by: Paolo Abeni Signed-off-by: Sasha Levin CVE-2026-53215 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 43 +++++++++++-------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c index a0f563e13720b..dcabeee91e7de 100644 --- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c +++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c @@ -3970,6 +3970,12 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, else frag_size = bm_pool->frag_size; + err = mvpp2_rx_refill(port, bm_pool, pp, pool); + if (err) { + netdev_err(port->dev, "failed to refill BM pools\n"); + goto err_drop_frame; + } + if (xdp_prog) { struct xdp_rxq_info *xdp_rxq; @@ -3987,12 +3993,6 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, if (ret) { xdp_ret |= ret; - err = mvpp2_rx_refill(port, bm_pool, pp, pool); - if (err) { - netdev_err(port->dev, "failed to refill BM pools\n"); - goto err_drop_frame; - } - ps.rx_packets++; ps.rx_bytes += rx_bytes; continue; @@ -4007,8 +4007,21 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, skb = slab_build_skb(data); if (!skb) { netdev_warn(port->dev, "skb build failed\n"); - goto err_drop_frame; + if (pp) { + page_pool_put_page(pp, virt_to_head_page(data), + rx_bytes + MVPP2_MH_SIZE, + true); + } else { + dma_unmap_single_attrs(dev->dev.parent, dma_addr, + bm_pool->buf_size, + DMA_FROM_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC); + mvpp2_frag_free(bm_pool, pp, data); + } + goto err_drop_frame_retired; } + if (pp) + skb_mark_for_recycle(skb); /* If we have RX hardware timestamping enabled, grab the * timestamp from the queue and convert. @@ -4019,16 +4032,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, skb_hwtstamps(skb)); } - err = mvpp2_rx_refill(port, bm_pool, pp, pool); - if (err) { - netdev_err(port->dev, "failed to refill BM pools\n"); - dev_kfree_skb_any(skb); - goto err_drop_frame; - } - - if (pp) - skb_mark_for_recycle(skb); - else + if (!pp) dma_unmap_single_attrs(dev->dev.parent, dma_addr, bm_pool->buf_size, DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC); @@ -4047,13 +4051,14 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi, continue; err_drop_frame: - dev->stats.rx_errors++; - mvpp2_rx_error(port, rx_desc); /* Return the buffer to the pool */ if (rx_status & MVPP2_RXD_BUF_HDR) mvpp2_buff_hdr_pool_put(port, rx_desc, pool, rx_status); else mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr); +err_drop_frame_retired: + dev->stats.rx_errors++; + mvpp2_rx_error(port, rx_desc); } if (xdp_ret & MVPP2_XDP_REDIR) From 965f15461e3c90ca1109a9eab4cba2e12b8b0015 Mon Sep 17 00:00:00 2001 From: Michael Bommarito Date: Tue, 2 Jun 2026 18:04:57 -0400 Subject: [PATCH 012/464] RDMA/srp: bound SRP_RSP sense copy by the received length commit 13e91fd076306f5d0cdfa14f53d69e37274723c4 upstream. srp_process_rsp() copies sense data from rsp->data + resp_data_len, where resp_data_len is the full 32-bit value supplied by the SRP target and is never checked against the number of bytes actually received (wc->byte_len). The copy length is bounded to SCSI_SENSE_BUFFERSIZE, so at most 96 bytes are copied, but the source offset is not bounded. A malicious or compromised SRP target on the InfiniBand/RoCE fabric that the initiator has logged into can return an SRP_RSP with SRP_RSP_FLAG_SNSVALID set and a large resp_data_len. The receive buffer is allocated at the target-chosen max_ti_iu_len, so the source of the sense copy lands past the bytes actually received; with resp_data_len near 0xFFFFFFFF it is gigabytes past the buffer and the read faults. Copy the sense data only if it has not been truncated, that is, only if the response header, the response data, and the sense region fit within the bytes actually received; otherwise drop the sense and log. The in-tree iSER and NVMe-RDMA receive paths already bound their parse by wc->byte_len; this brings ib_srp into line with them. Fixes: aef9ec39c47f ("IB: Add SCSI RDMA Protocol (SRP) initiator") Link: https://patch.msgid.link/r/20260602220457.2542840-1-michael.bommarito@gmail.com Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Michael Bommarito Reviewed-by: Bart Van Assche Signed-off-by: Jason Gunthorpe Signed-off-by: Greg Kroah-Hartman CVE-2026-53186 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- drivers/infiniband/ulp/srp/ib_srp.c | 30 +++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index b58868e1cf11c..acbd787de265b 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -1932,7 +1932,8 @@ static int srp_post_recv(struct srp_rdma_ch *ch, struct srp_iu *iu) return ib_post_recv(ch->qp, &wr, NULL); } -static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp) +static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp, + u32 byte_len) { struct srp_target_port *target = ch->target; struct srp_request *req; @@ -1973,10 +1974,27 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp) scmnd->result = rsp->status; if (rsp->flags & SRP_RSP_FLAG_SNSVALID) { - memcpy(scmnd->sense_buffer, rsp->data + - be32_to_cpu(rsp->resp_data_len), - min_t(int, be32_to_cpu(rsp->sense_data_len), - SCSI_SENSE_BUFFERSIZE)); + u32 resp_len = be32_to_cpu(rsp->resp_data_len); + u32 sense_len = be32_to_cpu(rsp->sense_data_len); + + /* + * The sense data starts resp_data_len bytes past the + * response data area; both lengths come from the + * target-controlled response. Copy the sense data + * only if it has not been truncated, that is, only if + * the full sense region fits within the bytes actually + * received. Otherwise the copy source would run past + * the receive buffer (sized to the target-chosen + * max_ti_iu_len), reading out of bounds. + */ + if (sizeof(*rsp) + (u64)resp_len + sense_len <= byte_len) + memcpy(scmnd->sense_buffer, + rsp->data + resp_len, + min(sense_len, SCSI_SENSE_BUFFERSIZE)); + else + shost_printk(KERN_ERR, target->scsi_host, + "dropping truncated sense data (resp_data_len %u sense_data_len %u, %u bytes received)\n", + resp_len, sense_len, byte_len); } if (unlikely(rsp->flags & SRP_RSP_FLAG_DIUNDER)) @@ -2086,7 +2104,7 @@ static void srp_recv_done(struct ib_cq *cq, struct ib_wc *wc) switch (opcode) { case SRP_RSP: - srp_process_rsp(ch, iu->buf); + srp_process_rsp(ch, iu->buf, wc->byte_len); break; case SRP_CRED_REQ: From 3e639ca7ba436a6e158f1947e4eb42e08d912b3c Mon Sep 17 00:00:00 2001 From: Michael Bommarito Date: Tue, 2 Jun 2026 15:46:42 -0400 Subject: [PATCH 013/464] IB/isert: Reject login PDUs shorter than ISER_HEADERS_LEN commit 29e7b925ae6df64894e82ab6419994dc25580a8a upstream. In drivers/infiniband/ulp/isert/ib_isert.c, isert_login_recv_done() computes the login request payload length as wc->byte_len minus ISER_HEADERS_LEN with no lower bound, and login_req_len is a signed int. A remote iSER initiator can post a login Send work request carrying fewer than ISER_HEADERS_LEN (76) bytes, so the subtraction underflows and login_req_len becomes negative. isert_rx_login_req() then reads that negative length back into a signed int, takes size = min(rx_buflen, MAX_KEY_VALUE_PAIRS), and because the min() is signed it keeps the negative value; the value is then passed as the memcpy() length and sign-extended to a multi-gigabyte size_t. The copy into the 8192-byte login->req_buf runs far out of bounds and faults, crashing the target node. The login phase precedes iSCSI authentication, so no credentials are required to reach this path. Reject any login PDU shorter than ISER_HEADERS_LEN before the subtraction, mirroring the existing early return on a failed work completion, so login_req_len can never go negative. The upper bound was already safe: a posted login buffer cannot deliver more than ISER_RX_PAYLOAD_SIZE, so the difference stays at or below MAX_KEY_VALUE_PAIRS and the existing min() clamps it; only the missing lower bound needs to be added. Fixes: b8d26b3be8b3 ("iser-target: Add iSCSI Extensions for RDMA (iSER) target driver") Link: https://patch.msgid.link/r/20260602194642.2273217-1-michael.bommarito@gmail.com Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Michael Bommarito Signed-off-by: Jason Gunthorpe Signed-off-by: Greg Kroah-Hartman CVE-2026-53176 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- drivers/infiniband/ulp/isert/ib_isert.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 348005e71891c..1015a51f750af 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -1383,6 +1383,12 @@ isert_login_recv_done(struct ib_cq *cq, struct ib_wc *wc) ib_dma_sync_single_for_cpu(ib_dev, isert_conn->login_desc->dma_addr, ISER_RX_SIZE, DMA_FROM_DEVICE); + if (unlikely(wc->byte_len < ISER_HEADERS_LEN)) { + isert_dbg("login request length %u is too short\n", + wc->byte_len); + return; + } + isert_conn->login_req_len = wc->byte_len - ISER_HEADERS_LEN; if (isert_conn->conn) { From 741498f10ac00211570471107b49887fe61bd773 Mon Sep 17 00:00:00 2001 From: Hyunwoo Kim Date: Tue, 2 Jun 2026 19:21:05 +0900 Subject: [PATCH 014/464] inet: frags: fix use-after-free caused by the fqdir_pre_exit() flush commit 32594b09854970d7ba83eb2dc8c69a2edd158c8e upstream. On netns teardown, fqdir_pre_exit() walks the fqdir rhashtable and flushes every fragment queue that is not yet complete using inet_frag_queue_flush(). That helper frees all the skbs queued on the fragment queue but does not set INET_FRAG_COMPLETE, and leaves q->fragments_tail and q->last_run_head pointing at the freed skbs. The queue itself stays in the rhashtable. fqdir_pre_exit() first lowers high_thresh to 0 to stop new queue lookups, but it cannot stop a fragment that already obtained the queue through inet_frag_find() earlier and stalled just before taking the queue lock. Once that fragment resumes after the flush and takes the queue lock, it passes the INET_FRAG_COMPLETE check and then dereferences the freed fragments_tail. inet_frag_queue_insert() reads FRAG_CB() and ->len of that pointer and, on the append path, writes ->next_frag, causing a slab use-after-free. IPv6, nf_conntrack_reasm6 and 6lowpan reassembly share the same flush path and are affected as well. Reset rb_fragments, fragments_tail and last_run_head in inet_frag_queue_flush() so a flushed queue no longer points at the freed skbs. A fragment that resumes after the flush and takes the queue lock then finds an empty queue and starts a new run instead of dereferencing the freed fragments_tail. ip_frag_reinit() already performed this reset after its own flush, so drop the now duplicate code there. Cc: stable@vger.kernel.org Fixes: 006a5035b495 ("inet: frags: flush pending skbs in fqdir_pre_exit()") Suggested-by: Eric Dumazet Signed-off-by: Hyunwoo Kim Link: https://patch.msgid.link/ah6ukYq5G98LshdA@v4bel Signed-off-by: Jakub Kicinski Signed-off-by: Greg Kroah-Hartman CVE-2026-53175 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/ipv4/inet_fragment.c | 3 +++ net/ipv4/ip_fragment.c | 3 --- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/net/ipv4/inet_fragment.c b/net/ipv4/inet_fragment.c index 393770920abd7..1127519b8416c 100644 --- a/net/ipv4/inet_fragment.c +++ b/net/ipv4/inet_fragment.c @@ -328,6 +328,9 @@ void inet_frag_queue_flush(struct inet_frag_queue *q, reason = reason ?: SKB_DROP_REASON_FRAG_REASM_TIMEOUT; sum = inet_frag_rbtree_purge(&q->rb_fragments, reason); sub_frag_mem_limit(q->fqdir, sum); + q->rb_fragments = RB_ROOT; + q->fragments_tail = NULL; + q->last_run_head = NULL; } EXPORT_SYMBOL(inet_frag_queue_flush); diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c index 56b0f738d2f27..c790d2f494870 100644 --- a/net/ipv4/ip_fragment.c +++ b/net/ipv4/ip_fragment.c @@ -250,9 +250,6 @@ static int ip_frag_reinit(struct ipq *qp) qp->q.flags = 0; qp->q.len = 0; qp->q.meat = 0; - qp->q.rb_fragments = RB_ROOT; - qp->q.fragments_tail = NULL; - qp->q.last_run_head = NULL; qp->iif = 0; qp->ecn = 0; From 494bd950842561e8ba08d49e384eda77e2d7a025 Mon Sep 17 00:00:00 2001 From: David Howells Date: Thu, 4 Jun 2026 12:46:00 +0100 Subject: [PATCH 015/464] rxrpc: Fix the ACK parser to extract the SACK table for parsing commit 333b6d5bb9f87827ac2639c737bf9613dbae7253 upstream. Fix modification of the received skbuff in rxrpc_input_soft_acks() and a potential incorrect access of the buffer in a fragmented UDP packet (the packet would probably have to be deliberately pre-generated as fragmented) when AF_RXRPC tries to extract the contents of the SACK table by copying out the contents of the SACK table into a buffer before attempting to parse AF_RXRPC assumes that it can just call skb_condense() and then validly access the SACK table from skb->data and that it will be a flat buffer - but skb_condense() can silently fail to do anything under some circumstances. Note that whilst rxrpc_input_soft_acks() should be able to parse extended ACKs, the rest of AF_RXRPC doesn't currently support that. Further, there's then no need to call skb_condense() in rxrpc_input_ack(), so don't. Fixes: d57a3a151660 ("rxrpc: Save last ACK's SACK table rather than marking txbufs") Reported-by: Michael Bommarito Link: https://lore.kernel.org/r/20260513180907.2061972-1-michael.bommarito@gmail.com Signed-off-by: David Howells cc: Marc Dionne cc: Jeffrey Altman cc: Eric Dumazet cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Simon Horman cc: linux-afs@lists.infradead.org cc: netdev@vger.kernel.org cc: stable@kernel.org Link: https://patch.msgid.link/105362.1780573560@warthog.procyon.org.uk Signed-off-by: Paolo Abeni Signed-off-by: Greg Kroah-Hartman CVE-2026-53151 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/rxrpc/input.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c index 24aceb183c2c3..ce761466b02d6 100644 --- a/net/rxrpc/input.c +++ b/net/rxrpc/input.c @@ -963,23 +963,34 @@ static void rxrpc_input_soft_acks(struct rxrpc_call *call, struct rxrpc_skb_priv *sp = rxrpc_skb(skb); struct rxrpc_txqueue *tq = call->tx_queue; unsigned long extracted = ~0UL; - unsigned int nr = 0; + unsigned int nr = 0, nsack; rxrpc_seq_t seq = call->acks_hard_ack + 1; rxrpc_seq_t lowest_nak = seq + sp->ack.nr_acks; - u8 *acks = skb->data + sizeof(struct rxrpc_wire_header) + sizeof(struct rxrpc_ackpacket); + u8 sack[256] __aligned(sizeof(unsigned long)); + u8 *acks = sack; _enter("%x,%x,%u", tq->qbase, seq, sp->ack.nr_acks); while (after(seq, tq->qbase + RXRPC_NR_TXQUEUE - 1)) tq = tq->next; + /* Extract an individual SACK table. A normal SACK table is up to 255 + * bytes with 1 ACK flag per byte, but an extended SACK table can be up + * to 256 bytes with up to 8 ACK/NACK flags per byte. The ACK flags go + * across all bit 0's then all bit 1's, then all bit 2's, ... + */ + memset(sack, 0, sizeof(sack)); + nsack = umin(sp->ack.nr_acks, 256); + if (skb_copy_bits(skb, + sizeof(struct rxrpc_wire_header) + sizeof(struct rxrpc_ackpacket), + sack, nsack) < 0) + return; + for (unsigned int i = 0; i < sp->ack.nr_acks; i++) { /* Decant ACKs until we hit a txqueue boundary. */ + if ((i & 255) == 0) + acks = sack; shiftr_adv_rotr(acks, extracted); - if (i == 256) { - acks -= i; - i = 0; - } seq++; nr++; if ((seq & RXRPC_TXQ_MASK) != 0) @@ -1117,9 +1128,6 @@ static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb) skb_copy_bits(skb, ioffset, &trailer, sizeof(trailer)) < 0) return rxrpc_proto_abort(call, 0, rxrpc_badmsg_short_ack_trailer); - if (nr_acks > 0) - skb_condense(skb); - call->acks_latest_ts = ktime_get_real(); call->acks_hard_ack = hard_ack; call->acks_prev_seq = prev_pkt; From a1973ce9d232a9d39fa0ecfa08aadffa69589d3b Mon Sep 17 00:00:00 2001 From: Zhengchuan Liang Date: Sat, 4 Apr 2026 17:39:48 +0800 Subject: [PATCH 016/464] netfilter: require Ethernet MAC header before using eth_hdr() [ Upstream commit 62443dc21114c0bbc476fa62973db89743f2f137 ] `ip6t_eui64`, `xt_mac`, the `bitmap:ip,mac`, `hash:ip,mac`, and `hash:mac` ipset types, and `nf_log_syslog` access `eth_hdr(skb)` after either assuming that the skb is associated with an Ethernet device or checking only that the `ETH_HLEN` bytes at `skb_mac_header(skb)` lie between `skb->head` and `skb->data`. Make these paths first verify that the skb is associated with an Ethernet device, that the MAC header was set, and that it spans at least a full Ethernet header before accessing `eth_hdr(skb)`. Suggested-by: Florian Westphal Tested-by: Ren Wei Signed-off-by: Zhengchuan Liang Signed-off-by: Ren Wei Signed-off-by: Florian Westphal Signed-off-by: Sasha Levin CVE-2026-53131 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/ipv6/netfilter/ip6t_eui64.c | 7 +++++-- net/netfilter/ipset/ip_set_bitmap_ipmac.c | 5 +++-- net/netfilter/ipset/ip_set_hash_ipmac.c | 9 +++++---- net/netfilter/ipset/ip_set_hash_mac.c | 5 +++-- net/netfilter/nf_log_syslog.c | 8 +++++++- net/netfilter/xt_mac.c | 4 +--- 6 files changed, 24 insertions(+), 14 deletions(-) diff --git a/net/ipv6/netfilter/ip6t_eui64.c b/net/ipv6/netfilter/ip6t_eui64.c index da69a27e8332c..bbb684f9964c0 100644 --- a/net/ipv6/netfilter/ip6t_eui64.c +++ b/net/ipv6/netfilter/ip6t_eui64.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -21,8 +22,10 @@ eui64_mt6(const struct sk_buff *skb, struct xt_action_param *par) { unsigned char eui64[8]; - if (!(skb_mac_header(skb) >= skb->head && - skb_mac_header(skb) + ETH_HLEN <= skb->data)) { + if (!skb->dev || skb->dev->type != ARPHRD_ETHER) + return false; + + if (!skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) { par->hotdrop = true; return false; } diff --git a/net/netfilter/ipset/ip_set_bitmap_ipmac.c b/net/netfilter/ipset/ip_set_bitmap_ipmac.c index 2c625e0f49ec0..752f59ef87442 100644 --- a/net/netfilter/ipset/ip_set_bitmap_ipmac.c +++ b/net/netfilter/ipset/ip_set_bitmap_ipmac.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -220,8 +221,8 @@ bitmap_ipmac_kadt(struct ip_set *set, const struct sk_buff *skb, return -IPSET_ERR_BITMAP_RANGE; /* Backward compatibility: we don't check the second flag */ - if (skb_mac_header(skb) < skb->head || - (skb_mac_header(skb) + ETH_HLEN) > skb->data) + if (!skb->dev || skb->dev->type != ARPHRD_ETHER || + !skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) return -EINVAL; e.id = ip_to_id(map, ip); diff --git a/net/netfilter/ipset/ip_set_hash_ipmac.c b/net/netfilter/ipset/ip_set_hash_ipmac.c index 467c59a83c0ab..b9a2681e24888 100644 --- a/net/netfilter/ipset/ip_set_hash_ipmac.c +++ b/net/netfilter/ipset/ip_set_hash_ipmac.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -89,8 +90,8 @@ hash_ipmac4_kadt(struct ip_set *set, const struct sk_buff *skb, struct hash_ipmac4_elem e = { .ip = 0, { .foo[0] = 0, .foo[1] = 0 } }; struct ip_set_ext ext = IP_SET_INIT_KEXT(skb, opt, set); - if (skb_mac_header(skb) < skb->head || - (skb_mac_header(skb) + ETH_HLEN) > skb->data) + if (!skb->dev || skb->dev->type != ARPHRD_ETHER || + !skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) return -EINVAL; if (opt->flags & IPSET_DIM_TWO_SRC) @@ -205,8 +206,8 @@ hash_ipmac6_kadt(struct ip_set *set, const struct sk_buff *skb, }; struct ip_set_ext ext = IP_SET_INIT_KEXT(skb, opt, set); - if (skb_mac_header(skb) < skb->head || - (skb_mac_header(skb) + ETH_HLEN) > skb->data) + if (!skb->dev || skb->dev->type != ARPHRD_ETHER || + !skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) return -EINVAL; if (opt->flags & IPSET_DIM_TWO_SRC) diff --git a/net/netfilter/ipset/ip_set_hash_mac.c b/net/netfilter/ipset/ip_set_hash_mac.c index 718814730acf6..41a122591fe24 100644 --- a/net/netfilter/ipset/ip_set_hash_mac.c +++ b/net/netfilter/ipset/ip_set_hash_mac.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -77,8 +78,8 @@ hash_mac4_kadt(struct ip_set *set, const struct sk_buff *skb, struct hash_mac4_elem e = { { .foo[0] = 0, .foo[1] = 0 } }; struct ip_set_ext ext = IP_SET_INIT_KEXT(skb, opt, set); - if (skb_mac_header(skb) < skb->head || - (skb_mac_header(skb) + ETH_HLEN) > skb->data) + if (!skb->dev || skb->dev->type != ARPHRD_ETHER || + !skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) return -EINVAL; if (opt->flags & IPSET_DIM_ONE_SRC) diff --git a/net/netfilter/nf_log_syslog.c b/net/netfilter/nf_log_syslog.c index 41503847d9d7f..98d2b9db16efe 100644 --- a/net/netfilter/nf_log_syslog.c +++ b/net/netfilter/nf_log_syslog.c @@ -78,7 +78,10 @@ dump_arp_packet(struct nf_log_buf *m, else logflags = NF_LOG_DEFAULT_MASK; - if (logflags & NF_LOG_MACDECODE) { + if ((logflags & NF_LOG_MACDECODE) && + skb->dev && skb->dev->type == ARPHRD_ETHER && + skb_mac_header_was_set(skb) && + skb_mac_header_len(skb) >= ETH_HLEN) { nf_log_buf_add(m, "MACSRC=%pM MACDST=%pM ", eth_hdr(skb)->h_source, eth_hdr(skb)->h_dest); nf_log_dump_vlan(m, skb); @@ -789,6 +792,9 @@ static void dump_mac_header(struct nf_log_buf *m, switch (dev->type) { case ARPHRD_ETHER: + if (!skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) + return; + nf_log_buf_add(m, "MACSRC=%pM MACDST=%pM ", eth_hdr(skb)->h_source, eth_hdr(skb)->h_dest); nf_log_dump_vlan(m, skb); diff --git a/net/netfilter/xt_mac.c b/net/netfilter/xt_mac.c index bd2354760895d..7fc5156825e49 100644 --- a/net/netfilter/xt_mac.c +++ b/net/netfilter/xt_mac.c @@ -29,9 +29,7 @@ static bool mac_mt(const struct sk_buff *skb, struct xt_action_param *par) if (skb->dev == NULL || skb->dev->type != ARPHRD_ETHER) return false; - if (skb_mac_header(skb) < skb->head) - return false; - if (skb_mac_header(skb) + ETH_HLEN > skb->data) + if (!skb_mac_header_was_set(skb) || skb_mac_header_len(skb) < ETH_HLEN) return false; ret = ether_addr_equal(eth_hdr(skb)->h_source, info->srcaddr); ret ^= info->invert; From 75b0ff44587f840b2e5cf059e0e0778a2ac3082b Mon Sep 17 00:00:00 2001 From: Paolo Bonzini Date: Fri, 26 Jun 2026 13:23:15 +0200 Subject: [PATCH 017/464] KVM: x86: Fix shadow paging use-after-free due to unexpected role commit 81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb upstream. Commit 0cb2af2ea66ad ("KVM: x86: Fix shadow paging use-after-free due to unexpected GFN") fixed a shadow paging mismatch between stored and computed GFNs; the bug could be triggered by changing a PDE mapping from outside the guest, and then deleting a memslot. The rmap_remove() call would miss entries created after the PDE change because the GFN of the leaf SPTE does not match the GFN of the struct kvm_mmu_page. A similar hole however remains if the modified PDE points to a non-leaf page. In this case the gfn can be made to match, but the role does not match: the original large 2MB page creates a kvm_mmu_page with direct=1, while the new 4KB needs a kvm_mmu_page with direct=0. However, kvm_mmu_get_child_sp() does not compare the role, and therefore reuses the page. The next step is installing a leaf (4KB) SPTE on the new path which records an rmap entry under the gfn resolved by the walk. But when that child is zapped its parent kvm_mmu_page has direct=1 and kvm_mmu_page_get_gfn() computes the gfn for the 4KB page as sp->gfn + index instead of using sp->shadowed_translation[] (or sp->gfns[] in older kernels). It therefore fails to remove the recorded entry. When the memslot is dropped the shadow page is freed but the rmap entry survives, as in the scenario that was already fixed. Code that later walks that gfn (dirty logging, MMU notifier invalidation, and so on) dereferences an sptep that lies in the freed page, causing the use-after-free. Fixes: 2032a93d66fa ("KVM: MMU: Don't allocate gfns page for direct mmu pages") Reported-by: Hyunwoo Kim Signed-off-by: Paolo Bonzini Signed-off-by: Sasha Levin CVE-2026-53359 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- arch/x86/kvm/mmu/mmu.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 729240bc00a26..3e1218abbbb75 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2453,13 +2453,15 @@ static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, bool direct, unsigned int access) { - union kvm_mmu_page_role role; + union kvm_mmu_page_role role = kvm_mmu_child_role(sptep, direct, access); - if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep) && - spte_to_child_sp(*sptep) && spte_to_child_sp(*sptep)->gfn == gfn) + if (is_shadow_present_pte(*sptep) && + !is_large_pte(*sptep) && + spte_to_child_sp(*sptep) && + spte_to_child_sp(*sptep)->gfn == gfn && + spte_to_child_sp(*sptep)->role.word == role.word) return ERR_PTR(-EEXIST); - role = kvm_mmu_child_role(sptep, direct, access); return kvm_mmu_get_shadow_page(vcpu, gfn, role); } From 2727d0fd4080162cf70ec1bb6b377cdf8ec8b0eb Mon Sep 17 00:00:00 2001 From: Tristan Madani Date: Wed, 27 May 2026 13:57:50 +0000 Subject: [PATCH 018/464] netfilter: nft_tunnel: fix use-after-free on object destroy commit c32b26aaa2f9216520a38b3f4bfeec846eb3eb8a upstream. nft_tunnel_obj_destroy() calls metadata_dst_free() which directly kfree()s the metadata_dst, ignoring the dst_entry refcount. Packets that took a reference via dst_hold() in nft_tunnel_obj_eval() and are still queued (e.g. in a netem qdisc) are left with a dangling pointer. When these packets are eventually dequeued, dst_release() operates on freed memory. Replace metadata_dst_free() with dst_release() so the metadata_dst is freed only after all references are dropped. The dst subsystem already handles metadata_dst cleanup in dst_destroy() when DST_METADATA is set. Fixes: af308b94a2a4 ("netfilter: nf_tables: add tunnel support") Cc: stable@vger.kernel.org Signed-off-by: Tristan Madani Reviewed-by: Fernando Fernandez Mancera Signed-off-by: Florian Westphal Signed-off-by: Pablo Neira Ayuso Signed-off-by: Greg Kroah-Hartman CVE-2026-53212 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- net/netfilter/nft_tunnel.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/netfilter/nft_tunnel.c b/net/netfilter/nft_tunnel.c index a12486ae089d6..db183e43941ac 100644 --- a/net/netfilter/nft_tunnel.c +++ b/net/netfilter/nft_tunnel.c @@ -702,7 +702,7 @@ static void nft_tunnel_obj_destroy(const struct nft_ctx *ctx, { struct nft_tunnel_obj *priv = nft_obj_data(obj); - metadata_dst_free(priv->md); + dst_release(&priv->md->dst); } static struct nft_object_type nft_tunnel_obj_type; From 9f141c5f082cd109559dec9785e88634fe74a9f5 Mon Sep 17 00:00:00 2001 From: Rajat Gupta Date: Sun, 31 May 2026 08:32:21 -0400 Subject: [PATCH 019/464] net/sched: fix pedit partial COW leading to page cache corruption MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [ Upstream commit 899ee91156e57784090c5565e4f31bd7dbffbc5a ] tcf_pedit_act() computes the COW range for skb_ensure_writable() once before the key loop using tcfp_off_max_hint, but the hint does not account for the runtime header offset added by typed keys. This can leave part of the write region un-COW'd. Fix by moving skb_ensure_writable() inside the per-key loop where the actual write offset is known, and add overflow checking on the offset arithmetic. For negative offsets (e.g. Ethernet header edits at ingress), use skb_cow() to COW the headroom instead. Guard offset_valid() against INT_MIN, where negation is undefined. Fixes: 8b796475fd78 ("net/sched: act_pedit: really ensure the skb is writable") Reported-by: Yiming Qian Reported-by: Keenan Dong Reported-by: Han Guidong <2045gemini@gmail.com> Reported-by: Zhang Cen Reviewed-by: Han Guidong <2045gemini@gmail.com> Tested-by: Han Guidong <2045gemini@gmail.com> Reviewed-by: Davide Caratti Tested-by: Davide Caratti Reviewed-by: Toke Høiland-Jørgensen Tested-by: Toke Høiland-Jørgensen Reviewed-by: Victor Nogueira Tested-by: Victor Nogueira Acked-by: Jamal Hadi Salim Signed-off-by: Rajat Gupta Link: https://patch.msgid.link/20260531123221.48732-1-jhs@mojatatu.com Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin CVE-2026-46331 Signed-off-by: Alice C. Munduruca Signed-off-by: Stefan Bader --- include/net/tc_act/tc_pedit.h | 1 - net/sched/act_pedit.c | 77 +++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 37 deletions(-) diff --git a/include/net/tc_act/tc_pedit.h b/include/net/tc_act/tc_pedit.h index f58ee15cd858c..cb7b82f2cbc7f 100644 --- a/include/net/tc_act/tc_pedit.h +++ b/include/net/tc_act/tc_pedit.h @@ -15,7 +15,6 @@ struct tcf_pedit_parms { struct tc_pedit_key *tcfp_keys; struct tcf_pedit_key_ex *tcfp_keys_ex; int action; - u32 tcfp_off_max_hint; unsigned char tcfp_nkeys; unsigned char tcfp_flags; struct rcu_head rcu; diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c index bc20f08a27890..bd3b1da3cd63b 100644 --- a/net/sched/act_pedit.c +++ b/net/sched/act_pedit.c @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include #include #include @@ -242,7 +244,6 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla, goto out_free_ex; } - nparms->tcfp_off_max_hint = 0; nparms->tcfp_flags = parm->flags; nparms->tcfp_nkeys = parm->nkeys; @@ -268,14 +269,6 @@ static int tcf_pedit_init(struct net *net, struct nlattr *nla, BITS_PER_TYPE(int) - 1, nparms->tcfp_keys[i].shift); - /* The AT option can read a single byte, we can bound the actual - * value with uchar max. - */ - cur += (0xff & offmask) >> nparms->tcfp_keys[i].shift; - - /* Each key touches 4 bytes starting from the computed offset */ - nparms->tcfp_off_max_hint = - max(nparms->tcfp_off_max_hint, cur + 4); } p = to_pedit(*a); @@ -318,15 +311,12 @@ static void tcf_pedit_cleanup(struct tc_action *a) call_rcu(&parms->rcu, tcf_pedit_cleanup_rcu); } -static bool offset_valid(struct sk_buff *skb, int offset) +static bool offset_valid(struct sk_buff *skb, int offset, int len) { - if (offset > 0 && offset > skb->len) - return false; - - if (offset < 0 && -offset > skb_headroom(skb)) + if (offset < -(int)skb_headroom(skb)) return false; - return true; + return offset <= (int)skb->len - len; } static int pedit_l4_skb_offset(struct sk_buff *skb, int *hoffset, const int header_type) @@ -393,18 +383,10 @@ TC_INDIRECT_SCOPE int tcf_pedit_act(struct sk_buff *skb, struct tcf_pedit_key_ex *tkey_ex; struct tcf_pedit_parms *parms; struct tc_pedit_key *tkey; - u32 max_offset; int i; parms = rcu_dereference_bh(p->parms); - max_offset = (skb_transport_header_was_set(skb) ? - skb_transport_offset(skb) : - skb_network_offset(skb)) + - parms->tcfp_off_max_hint; - if (skb_ensure_writable(skb, min(skb->len, max_offset))) - goto done; - tcf_lastuse_update(&p->tcf_tm); tcf_action_update_bstats(&p->common, skb); @@ -412,10 +394,11 @@ TC_INDIRECT_SCOPE int tcf_pedit_act(struct sk_buff *skb, tkey_ex = parms->tcfp_keys_ex; for (i = parms->tcfp_nkeys; i > 0; i--, tkey++) { + int write_offset, write_len; int offset = tkey->off; int hoffset = 0; - u32 *ptr, hdata; - u32 val; + u32 cur_val, val; + u32 *ptr; int rc; if (tkey_ex) { @@ -433,13 +416,15 @@ TC_INDIRECT_SCOPE int tcf_pedit_act(struct sk_buff *skb, if (tkey->offmask) { u8 *d, _d; + int at_offset; - if (!offset_valid(skb, hoffset + tkey->at)) { + if (check_add_overflow(hoffset, (int)tkey->at, &at_offset) || + !offset_valid(skb, at_offset, sizeof(_d))) { pr_info_ratelimited("tc action pedit 'at' offset %d out of bounds\n", hoffset + tkey->at); goto bad; } - d = skb_header_pointer(skb, hoffset + tkey->at, + d = skb_header_pointer(skb, at_offset, sizeof(_d), &_d); if (!d) goto bad; @@ -451,31 +436,51 @@ TC_INDIRECT_SCOPE int tcf_pedit_act(struct sk_buff *skb, } } - if (!offset_valid(skb, hoffset + offset)) { - pr_info_ratelimited("tc action pedit offset %d out of bounds\n", hoffset + offset); + if (check_add_overflow(hoffset, offset, &write_offset)) { + pr_info_ratelimited("tc action pedit offset overflow\n"); goto bad; } - ptr = skb_header_pointer(skb, hoffset + offset, - sizeof(hdata), &hdata); - if (!ptr) + if (!offset_valid(skb, write_offset, sizeof(*ptr))) { + pr_info_ratelimited("tc action pedit offset %d out of bounds\n", + write_offset); goto bad; + } + + if (write_offset < 0) { + if (skb_cow(skb, -write_offset)) + goto bad; + if (write_offset + (int)sizeof(*ptr) > 0) { + if (skb_ensure_writable(skb, + min_t(int, skb->len, + write_offset + (int)sizeof(*ptr)))) + goto bad; + } + } else { + if (check_add_overflow(write_offset, (int)sizeof(*ptr), + &write_len)) + goto bad; + if (skb_ensure_writable(skb, min_t(int, skb->len, + write_len))) + goto bad; + } + + ptr = (u32 *)(skb->data + write_offset); + cur_val = get_unaligned(ptr); /* just do it, baby */ switch (cmd) { case TCA_PEDIT_KEY_EX_CMD_SET: val = tkey->val; break; case TCA_PEDIT_KEY_EX_CMD_ADD: - val = (*ptr + tkey->val) & ~tkey->mask; + val = (cur_val + tkey->val) & ~tkey->mask; break; default: pr_info_ratelimited("tc action pedit bad command (%d)\n", cmd); goto bad; } - *ptr = ((*ptr & tkey->mask) ^ val); - if (ptr == &hdata) - skb_store_bits(skb, hoffset + offset, ptr, 4); + put_unaligned((cur_val & tkey->mask) ^ val, ptr); } goto done; From 97ba1ff1fae3a4060928747137ba15901abdd562 Mon Sep 17 00:00:00 2001 From: Manuel Diewald Date: Fri, 17 Jul 2026 15:41:45 +0200 Subject: [PATCH 020/464] UBUNTU: [Packaging] update annotations scripts BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Manuel Diewald --- debian/scripts/misc/kconfig/annotations.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/debian/scripts/misc/kconfig/annotations.py b/debian/scripts/misc/kconfig/annotations.py index 5f34410d4ef87..ba9db44964708 100644 --- a/debian/scripts/misc/kconfig/annotations.py +++ b/debian/scripts/misc/kconfig/annotations.py @@ -80,7 +80,7 @@ def _parse_body(self, data: str, parent=True): if not line: continue - # Catpure flavors of included files + # Capture flavors of included files if line.startswith("# FLAVOUR: "): self.include_flavour += line.split(" ")[2:] continue @@ -211,7 +211,7 @@ def _json_parse(self, data, is_included=False): self.include = data["attributes"]["include"] self.include_flavour = [] else: - # We are procesing an imported annotations, so merge all the + # We are processing an imported annotations, so merge all the # configs and attributes. try: self.config = data["config"] | self.config From f77190c8615c45af97cdf53af3da8867b95fea0b Mon Sep 17 00:00:00 2001 From: Manuel Diewald Date: Fri, 17 Jul 2026 15:41:58 +0200 Subject: [PATCH 021/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Manuel Diewald --- debian.master/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.master/changelog b/debian.master/changelog index 37104d35ea2cc..ae5130d0c57f5 100644 --- a/debian.master/changelog +++ b/debian.master/changelog @@ -1,3 +1,11 @@ +linux (7.0.0-29.29) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Manuel Diewald Fri, 17 Jul 2026 15:41:57 +0200 + linux (7.0.0-28.28) resolute; urgency=medium * resolute/linux: 7.0.0-28.28 -proposed tracker (LP: #2157520) From ead0b96678fc2f04c0fc6ab17af20ab0987f679f Mon Sep 17 00:00:00 2001 From: Manuel Diewald Date: Fri, 17 Jul 2026 15:45:36 +0200 Subject: [PATCH 022/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2160781 Properties: no-test-build Signed-off-by: Manuel Diewald --- debian.master/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.master/tracking-bug b/debian.master/tracking-bug index d424626a3a249..946847fa44098 100644 --- a/debian.master/tracking-bug +++ b/debian.master/tracking-bug @@ -1 +1 @@ -2157520 2026.06.22-2 +2160781 s2026.06.22-1 From 6e350aadbd942c604174b9fd79d535121ef80e1c Mon Sep 17 00:00:00 2001 From: Manuel Diewald Date: Fri, 17 Jul 2026 18:57:53 +0200 Subject: [PATCH 023/464] UBUNTU: Ubuntu-7.0.0-29.29 Signed-off-by: Manuel Diewald --- debian.master/changelog | 68 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 63 insertions(+), 5 deletions(-) diff --git a/debian.master/changelog b/debian.master/changelog index ae5130d0c57f5..75da03b364103 100644 --- a/debian.master/changelog +++ b/debian.master/changelog @@ -1,10 +1,68 @@ -linux (7.0.0-29.29) UNRELEASED; urgency=medium +linux (7.0.0-29.29) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux: 7.0.0-29.29 -proposed tracker (LP: #2160781) - -- Manuel Diewald Fri, 17 Jul 2026 15:41:57 +0200 + * Packaging resync (LP: #1786013) + - [Packaging] update annotations scripts + + * CVE-2026-46331 + - net/sched: fix pedit partial COW leading to page cache corruption + + * CVE-2026-53212 + - netfilter: nft_tunnel: fix use-after-free on object destroy + + * CVE-2026-53359 + - KVM: x86: Fix shadow paging use-after-free due to unexpected role + + * CVE-2026-53131 + - netfilter: require Ethernet MAC header before using eth_hdr() + + * CVE-2026-53151 + - rxrpc: Fix the ACK parser to extract the SACK table for parsing + + * CVE-2026-53175 + - inet: frags: fix use-after-free caused by the fqdir_pre_exit() flush + + * CVE-2026-53176 + - IB/isert: Reject login PDUs shorter than ISER_HEADERS_LEN + + * CVE-2026-53186 + - RDMA/srp: bound SRP_RSP sense copy by the received length + + * CVE-2026-53215 + - net: mvpp2: refill RX buffers before XDP or skb use + + * CVE-2026-53216 + - net: mvpp2: limit XDP frame size to the RX buffer + + * CVE-2026-53221 + - ip6_vti: fix incorrect tunnel matching in vti6_tnl_lookup() + + * CVE-2026-53224 + - sctp: validate embedded INIT chunk and address list lengths in cookie + + * CVE-2026-53225 + - sctp: fix uninit-value in __sctp_rcv_asconf_lookup() + + * CVE-2026-53228 + - ipv6: sit: reload inner IPv6 header after GSO offloads + + * CVE-2026-52924 + - sctp: purge outqueue on stale COOKIE-ECHO handling + + * CVE-2026-53246 + - sctp: validate cached peer INIT chunk length in COOKIE_ECHO processing + + * CVE-2026-53247 + - net: ethernet: mtk_eth_soc: Fix use-after-free in metadata dst teardown + + * CVE-2026-53260 + - tcp: Add preempt_{disable,enable}_nested() in reqsk_queue_hash_req(). + + * Performance regression causes SDXL inference slowdown (~42x) (LP: #2158267) + - drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages + + -- Manuel Diewald Fri, 17 Jul 2026 18:57:53 +0200 linux (7.0.0-28.28) resolute; urgency=medium From f9f60a504b3d484397fc41bba52fa87976327f1e Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Fri, 27 Feb 2026 14:50:31 -0600 Subject: [PATCH 024/464] UBUNTU: [Packaging] Initialize resolute/linux-nvidia-bos Ignore: yes Signed-off-by: Jacob Martin --- Ubuntu.md | 6 +- debian.nvidia-bos/changelog | 77 ++++++++ debian.nvidia-bos/config/README.rst | 185 ++++++++++++++++++ debian.nvidia-bos/config/annotations | 14 ++ .../control.d/flavour-control.stub | 87 ++++++++ .../control.d/flavour-signed-control.stub | 38 ++++ debian.nvidia-bos/control.d/vars.nvidia | 5 + debian.nvidia-bos/control.d/vars.nvidia-64k | 5 + debian.nvidia-bos/control.stub.in | 99 ++++++++++ debian.nvidia-bos/dkms-versions | 8 + debian.nvidia-bos/etc/update.conf | 7 + debian.nvidia-bos/modprobe.d/common.conf | 3 + debian.nvidia-bos/reconstruct | 34 ++++ debian.nvidia-bos/rules.d/amd64.mk | 20 ++ debian.nvidia-bos/rules.d/arm64.mk | 20 ++ debian.nvidia-bos/tracking-bug | 1 + debian.nvidia-bos/upstream-stable | 3 + debian.nvidia-bos/variants | 4 + debian/debian.env | 2 +- 19 files changed, 614 insertions(+), 4 deletions(-) create mode 100644 debian.nvidia-bos/changelog create mode 100644 debian.nvidia-bos/config/README.rst create mode 100644 debian.nvidia-bos/config/annotations create mode 100644 debian.nvidia-bos/control.d/flavour-control.stub create mode 100644 debian.nvidia-bos/control.d/flavour-signed-control.stub create mode 100644 debian.nvidia-bos/control.d/vars.nvidia create mode 100644 debian.nvidia-bos/control.d/vars.nvidia-64k create mode 100644 debian.nvidia-bos/control.stub.in create mode 100644 debian.nvidia-bos/dkms-versions create mode 100644 debian.nvidia-bos/etc/update.conf create mode 100644 debian.nvidia-bos/modprobe.d/common.conf create mode 100644 debian.nvidia-bos/reconstruct create mode 100644 debian.nvidia-bos/rules.d/amd64.mk create mode 100644 debian.nvidia-bos/rules.d/arm64.mk create mode 100644 debian.nvidia-bos/tracking-bug create mode 100644 debian.nvidia-bos/upstream-stable create mode 100644 debian.nvidia-bos/variants diff --git a/Ubuntu.md b/Ubuntu.md index a49309c1d8c9d..e6dab098bdf6a 100644 --- a/Ubuntu.md +++ b/Ubuntu.md @@ -1,8 +1,8 @@ -Name: linux +Name: linux-nvidia-bos Version: 7.0.0 Series: 26.04 (resolute) Description: - This is the source code for the Ubuntu linux kernel for the 26.04 series. This - source tree is used to produce the flavours: generic, generic-64k. + This is the source code for the NVIDIA BOS linux kernel for the 26.04 series. This + source tree is used to produce the flavours: nvidia, nvidia-64k. This kernel is configured to support the widest range of desktop, laptop and server configurations. diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog new file mode 100644 index 0000000000000..5e34f09477c96 --- /dev/null +++ b/debian.nvidia-bos/changelog @@ -0,0 +1,77 @@ +linux-nvidia-bos (6.19.0-2000.0) resolute; urgency=medium + + * Initial changelog entry. + + -- Jacob Martin Fri, 27 Feb 2026 14:49:42 -0600 + +linux (6.19.0-6.6) resolute; urgency=medium + + * resolute/linux: 6.19.0-6.6 -proposed tracker (LP: #2142114) + + * Resolute update: v6.19.2 upstream stable release (LP: #2142112) + - Revert "driver core: enforce device_lock for driver_match_device()" + - Linux 6.19.2 + + * Resolute update: v6.19.1 upstream stable release (LP: #2142111) + - io_uring/io-wq: add exit-on-idle state + - io_uring: allow io-wq workers to exit when unused + - smb: client: split cached_fid bitfields to avoid shared-byte RMW races + - ksmbd: fix infinite loop caused by next_smb2_rcv_hdr_off reset in error + paths + - ksmbd: add chann_lock to protect ksmbd_chann_list xarray + - smb: server: fix leak of active_num_conn in ksmbd_tcp_new_connection() + - smb: smbdirect: introduce smbdirect_socket.recv_io.credits.available + - smb: smbdirect: introduce smbdirect_socket.send_io.bcredits.* + - smb: server: make use of smbdirect_socket.recv_io.credits.available + - smb: server: let recv_done() queue a refill when the peer is low on + credits + - smb: server: make use of smbdirect_socket.send_io.bcredits + - smb: server: fix last send credit problem causing disconnects + - smb: server: let send_done handle a completion without IB_SEND_SIGNALED + - smb: client: make use of smbdirect_socket.recv_io.credits.available + - smb: client: let recv_done() queue a refill when the peer is low on + credits + - smb: client: let smbd_post_send() make use of request->wr + - smb: client: remove pointless sc->recv_io.credits.count rollback + - smb: client: remove pointless sc->send_io.pending handling in + smbd_post_send_iter() + - smb: client: port and use the wait_for_credits logic used by server + - smb: client: split out smbd_ib_post_send() + - smb: client: introduce and use smbd_{alloc, free}_send_io() + - smb: client: use smbdirect_send_batch processing + - smb: client: make use of smbdirect_socket.send_io.bcredits + - smb: client: fix last send credit problem causing disconnects + - smb: client: let smbd_post_send_negotiate_req() use smbd_post_send() + - smb: client: let send_done handle a completion without IB_SEND_SIGNALED + - driver core: enforce device_lock for driver_match_device() + - Bluetooth: btusb: Add USB ID 7392:e611 for Edimax EW-7611UXB + - ALSA: hda/conexant: Add quirk for HP ZBook Studio G4 + - crypto: iaa - Fix out-of-bounds index in find_empty_iaa_compression_mode + - crypto: octeontx - Fix length check to avoid truncation in + ucode_load_store + - crypto: omap - Allocate OMAP_CRYPTO_FORCE_COPY scatterlists correctly + - crypto: virtio - Add spinlock protection with virtqueue notification + - crypto: virtio - Remove duplicated virtqueue_kick in + virtio_crypto_skcipher_crypt_req + - nilfs2: Fix potential block overflow that cause system hang + - hfs: ensure sb->s_fs_info is always cleaned up + - wifi: rtw88: Fix alignment fault in rtw_core_enable_beacon() + - scsi: qla2xxx: Validate sp before freeing associated memory + - scsi: qla2xxx: Allow recovery for tape devices + - scsi: qla2xxx: Delay module unload while fabric scan in progress + - scsi: qla2xxx: Free sp in error path to fix system crash + - scsi: qla2xxx: Query FW again before proceeding with login + - sched/mmcid: Don't assume CID is CPU owned on mode switch + - bus: fsl-mc: fix use-after-free in driver_override_show() + - erofs: fix UAF issue for file-backed mounts w/ directio option + - xfs: fix UAF in xchk_btree_check_block_owner + - drm/exynos: vidi: use ctx->lock to protect struct vidi_context member + variables related to memory alloc/free + - PCI: endpoint: Avoid creating sub-groups asynchronously + - wifi: rtl8xxxu: fix slab-out-of-bounds in rtl8xxxu_sta_add + - Linux 6.19.1 + + * AppArmor blocks write(2) to network sockets with Linux 6.19 (LP: #2141298) + - SAUCE: apparmor: fix aa_label_sk_perm to check for RULE_MEDIATES_NET + + -- Timo Aaltonen Wed, 18 Feb 2026 14:31:48 +0200 diff --git a/debian.nvidia-bos/config/README.rst b/debian.nvidia-bos/config/README.rst new file mode 100644 index 0000000000000..751ce7f3b284d --- /dev/null +++ b/debian.nvidia-bos/config/README.rst @@ -0,0 +1,185 @@ +================== +Config Annotations +================== + +:Author: Andrea Righi + +Overview +======== + +Each Ubuntu kernel needs to maintain its own .config for each supported +architecture and each flavour. + +Every time a new patch is applied or a kernel is rebased on top of a new +one, we need to update the .config's accordingly (config options can be +added, removed and also renamed). + +So, we need to make sure that some critical config options are always +matching the desired value in order to have a functional kernel. + +State of the art +================ + +At the moment configs are maintained as a set of Kconfig chunks (inside +`debian./config/`): a global one, plus per-arch / per-flavour +chunks. + +In addition to that, we need to maintain also a file called +'annotations'; the purpose of this file is to make sure that some +critical config options are not silently removed or changed when the +real .config is re-generated (for example after a rebase or after +applying a new set of patches). + +The main problem with this approach is that, often, we have duplicate +information that is stored both in the Kconfig chunks *and* in the +annotations files and, at the same time, the whole .config's information +is distributed between Kconfig chunks and annotations, making it hard to +maintain, review and manage in general. + +Proposed solution +================= + +The proposed solution is to store all the config information into the +"annotations" format and get rid of the config chunks (basically the +real .config's can be produced "compiling" annotations). + +Implementation +============== + +To help the management of the annotations an helper script is provided +(`debian/scripts/misc/annotations`): + +``` +usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR] [--config CONFIG] + (--query | --export | --import FILE | --update FILE | --check FILE) + +Manage Ubuntu kernel .config and annotations + +options: + -h, --help show this help message and exit + --version, -v show program's version number and exit + --file FILE, -f FILE Pass annotations or .config file to be parsed + --arch ARCH, -a ARCH Select architecture + --flavour FLAVOUR, -l FLAVOUR + Select flavour (default is "generic") + --config CONFIG, -c CONFIG + Select a specific config option + +Action: + --query, -q Query annotations + --export, -e Convert annotations to .config format + --import FILE, -i FILE + Import a full .config for a specific arch and flavour into annotations + --update FILE, -u FILE + Import a partial .config into annotations (only resync configs specified in FILE) + --check FILE, -k FILE + Validate kernel .config with annotations +``` + +This script allows to query config settings (per arch/flavour/config), +export them into the Kconfig format (generating the real .config files) +and check if the final .config matches the rules defined in the +annotations. + +Examples (annotations is defined as an alias to `debian/scripts/annotations`): + + - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across all the + supported architectures and flavours: + +``` +$ annotations --query --config CONFIG_DEBUG_INFO_BTF +{ + "policy": { + "amd64": "y", + "arm64": "y", + "armhf": "n", + "ppc64el": "y", + "riscv64": "y", + "s390x": "y" + }, + "note": "'Needs newer pahole for armhf'" +} +``` + + - Dump kernel .config for arm64 and flavour generic-64k: + +``` +$ annotations --arch arm64 --flavour generic-64k --export +CONFIG_DEBUG_FS=y +CONFIG_DEBUG_KERNEL=y +CONFIG_COMPAT=y +... +``` + + - Update annotations file with a new kernel .config for amd64 flavour + generic: + +``` +$ annotations --arch amd64 --flavour generic --import build/.config +``` + +Moreover, an additional kernelconfig commands are provided +(via debian/rules targets): + - `migrateconfigs`: automatically merge all the previous configs into + annotations (local changes still need to be committed) + +Annotations headers +=================== + +The main annotations file should contain a header to define the architectures +and flavours that are supported. + +Here is the format of the header for the generic kernel: +``` +# Menu: HEADER +# FORMAT: 4 +# ARCH: amd64 arm64 armhf ppc64el riscv64 s390x +# FLAVOUR: amd64-generic arm64-generic arm64-generic-64k armhf-generic armhf-generic-lpae ppc64el-generic riscv64-generic s390x-generic + +``` + +Example header of a derivative (linux-aws): +``` +# Menu: HEADER +# FORMAT: 4 +# ARCH: amd64 arm64 +# FLAVOUR: amd64-aws arm64-aws +# FLAVOUR_DEP: {'amd64-aws': 'amd64-generic', 'arm64-aws': 'arm64-generic'} + +include "../../debian.master/config/annotations" + +# Below you can define only the specific linux-aws configs that differ from linux generic + +``` + +Pros and Cons +============= + + Pros: + - avoid duplicate information in .config's and annotations + - allow to easily define groups of config settings (for a specific + environment or feature, such as annotations.clouds, annotations.ubuntu, + annotations.snapd, etc.) + - config options are more accessible, easy to change and review + - we can easily document how config options are managed (and external + contributors won't be discouraged anymore when they need to to change a + config option) + + Cons: + - potential regressions: the new tool/scripts can have potential bugs, + so we could experience regressions due to some missed config changes + - kernel team need to understand the new process (even if everything + is transparent, kernel cranking process is the same, there might be + corner cases that need to be addressed and resolved manually) + +TODO +==== + + - Migrate all flavour and arch definitions into annotations (rather + than having this information defined in multiple places inside + debian/scripts); right now this information is "partially" migrated, + meaning that we need to define arches and flavours in the headers + section of annotations (so that the annotations tool can figure out + the list of supported arches and flavours), but arches and flavours + are still defined elsewhere, ideally we would like to have arches and + flavours defined only in one place: annotations. diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations new file mode 100644 index 0000000000000..66f95c1d5d6bd --- /dev/null +++ b/debian.nvidia-bos/config/annotations @@ -0,0 +1,14 @@ +# Menu: HEADER +# FORMAT: 4 +# ARCH: amd64 arm64 +# FLAVOUR: amd64-nvidia arm64-nvidia arm64-nvidia-64k +# FLAVOUR_DEP: {'amd64-nvidia': 'amd64-generic', 'arm64-nvidia': 'arm64-generic', 'arm64-nvidia-64k': 'arm64-generic-64k'} + +include "../../debian.master/config/annotations" + + +# ---- Annotations without notes ---- + +CONFIG_AS_VERSION policy<{'amd64': '24600', 'arm64': '24600'}> +CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-14ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-13ubuntu3) 15.2.0"'}> +CONFIG_LD_VERSION policy<{'amd64': '24600', 'arm64': '24600'}> diff --git a/debian.nvidia-bos/control.d/flavour-control.stub b/debian.nvidia-bos/control.d/flavour-control.stub new file mode 100644 index 0000000000000..4055c769adb64 --- /dev/null +++ b/debian.nvidia-bos/control.d/flavour-control.stub @@ -0,0 +1,87 @@ +# Items that get replaced: +# FLAVOUR +# ARCH +# SUPPORTED +# TARGET +# BOOTLOADER +# =PROVIDES= +# +# Items marked with =FOO= are optional +# +# This file describes the template for packages that are created for each flavour +# in debian/control.d/vars.* +# +# This file gets edited in a couple of places. See the debian/control.stub rule in +# debian/rules. PGGVER, ABINUM, and SRCPKGNAME are all converted in the +# process of creating debian/control. +# +# The flavour specific strings (ARCH, etc) are converted using values from the various +# flavour files in debian/control.d/vars.* +# +# XXX: Leave the blank line before the first package!! + +Package: linux-modules-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: ARCH +Section: kernel +Priority: optional +Depends: ${misc:Depends}, ${shlibs:Depends}, wireless-regdb +Built-Using: ${linux:BuiltUsing} +Description: Linux kernel modules for version PKGVER + Contains the corresponding System.map file, the modules built by the + packager, and scripts that try to ensure that the system is not left in an + unbootable state after an update. + . + Supports SUPPORTED processors. + . + TARGET + . + You likely do not want to install this package directly. Instead, install + the linux-FLAVOUR meta-package, which will ensure that upgrades work + correctly, and that supporting packages are also installed. + +Package: linux-headers-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: ARCH +Section: devel +Priority: optional +Depends: ${misc:Depends}, SRCPKGNAME-headers-PKGVER-ABINUM, ${shlibs:Depends} +Provides: linux-headers, linux-headers-3.0 +Description: Linux kernel headers for version PKGVER + This package provides kernel header files for version PKGVER. + . + This is for sites that want the latest kernel headers. Please read + /usr/share/doc/linux-headers-PKGVER-ABINUM/debian.README.gz for details. + +Package: linux-lib-rust-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: amd64 +Multi-Arch: foreign +Section: devel +Priority: optional +Depends: ${misc:Depends}, coreutils +Description: Rust library files related to Linux kernel version PKGVER + This package provides kernel library files for version PKGVER, that allow to + compile out-of-tree kernel modules written in Rust. + +Package: linux-tools-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: ARCH +Section: devel +Priority: optional +Depends: ${misc:Depends}, SRCPKGNAME-tools-PKGVER-ABINUM +Description: Linux kernel version specific tools for version PKGVER-ABINUM + This package provides the architecture dependant parts for kernel + version locked tools (such as x86_energy_perf_policy) for + version PKGVER-ABINUM. + +Package: linux-cloud-tools-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: ARCH +Section: devel +Priority: optional +Depends: ${misc:Depends}, SRCPKGNAME-cloud-tools-PKGVER-ABINUM +Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM + This package provides the architecture dependant parts for kernel + version locked tools for cloud for version PKGVER-ABINUM. + diff --git a/debian.nvidia-bos/control.d/flavour-signed-control.stub b/debian.nvidia-bos/control.d/flavour-signed-control.stub new file mode 100644 index 0000000000000..b8551a52e743f --- /dev/null +++ b/debian.nvidia-bos/control.d/flavour-signed-control.stub @@ -0,0 +1,38 @@ +Package: linux-image=SIGN-ME-PKG=-PKGVER-ABINUM-FLAVOUR +Build-Profiles: +Architecture: ARCH +Section: kernel +Priority: optional +Provides: linux-image, fuse-module, =PROVIDES=${linux:rprovides} +Depends: ${misc:Depends}, ${shlibs:Depends}, kmod, linux-base (>= 4.5ubuntu1~16.04.1), linux-modules-PKGVER-ABINUM-FLAVOUR +Recommends: BOOTLOADER, initramfs-tools | linux-initramfs-tool +Breaks: flash-kernel (<< 3.90ubuntu2) [arm64 armhf], s390-tools (<< 2.3.0-0ubuntu3) [s390x] +Conflicts: linux-image=SIGN-PEER-PKG=-PKGVER-ABINUM-FLAVOUR +Suggests: bpftool, linux-perf, SRCPKGNAME-tools, linux-headers-PKGVER-ABINUM-FLAVOUR +Description: Linux kernel image for version PKGVER + This package contains the=SIGN-ME-TXT= Linux kernel image for version PKGVER. + . + Supports SUPPORTED processors. + . + TARGET + . + You likely do not want to install this package directly. Instead, install + the linux-FLAVOUR meta-package, which will ensure that upgrades work + correctly, and that supporting packages are also installed. + +Package: linux-image=SIGN-ME-PKG=-PKGVER-ABINUM-FLAVOUR-dbgsym +Build-Profiles: +Architecture: ARCH +Section: devel +Priority: optional +Depends: ${misc:Depends} +Provides: linux-debug +Description: Linux kernel debug image for version PKGVER + This package provides the=SIGN-ME-TXT= kernel debug image for version PKGVER. + . + This is for sites that wish to debug the kernel. + . + The kernel image contained in this package is NOT meant to boot from. It + is uncompressed, and unstripped. This package also includes the + unstripped modules. + diff --git a/debian.nvidia-bos/control.d/vars.nvidia b/debian.nvidia-bos/control.d/vars.nvidia new file mode 100644 index 0000000000000..1444699fc2bf2 --- /dev/null +++ b/debian.nvidia-bos/control.d/vars.nvidia @@ -0,0 +1,5 @@ +arch="amd64 arm64" +supported="NVIDIA" +target="Intended for NVIDIA platforms" +bootloader="grub-pc [amd64] | grub-efi-amd64 [amd64] | grub-efi-ia32 [amd64] | grub [amd64] | lilo [amd64] | flash-kernel [armhf arm64] | grub-efi-arm64 [arm64] | grub-efi-arm [armhf] | grub-ieee1275 [ppc64el]" +provides="kvm-api-4, redhat-cluster-modules, ivtv-modules, virtualbox-guest-modules [amd64]" diff --git a/debian.nvidia-bos/control.d/vars.nvidia-64k b/debian.nvidia-bos/control.d/vars.nvidia-64k new file mode 100644 index 0000000000000..072ec63f21074 --- /dev/null +++ b/debian.nvidia-bos/control.d/vars.nvidia-64k @@ -0,0 +1,5 @@ +arch="arm64" +supported="NVIDIA 64K pages" +target="Intended for NVIDIA systems" +bootloader="grub-efi-arm64 [arm64] | flash-kernel [arm64]" +provides="kvm-api-4, redhat-cluster-modules, ivtv-modules" diff --git a/debian.nvidia-bos/control.stub.in b/debian.nvidia-bos/control.stub.in new file mode 100644 index 0000000000000..9c20f84ca0721 --- /dev/null +++ b/debian.nvidia-bos/control.stub.in @@ -0,0 +1,99 @@ +Source: SRCPKGNAME +Section: devel +Priority: optional +Maintainer: Ubuntu Kernel Team +Rules-Requires-Root: no +Standards-Version: 3.9.4.0 +Build-Depends: + autoconf , + automake , + bc , + bindgen:native [amd64 arm64], + bison , + clang-21:native [amd64 arm64], + cpio, + curl , + debhelper-compat (= 10), + default-jdk-headless:native , + dkms , + flex , + gawk , + java-common , + kmod , + libaudit-dev , + libcap-dev , + libdebuginfod-dev [amd64 arm64] , + libdw-dev , + libelf-dev , + libiberty-dev , + liblzma-dev , + libnewt-dev , + libnuma-dev [amd64 arm64] , + libpci-dev , + libssl-dev , + libstdc++-dev, + libtool , + libtraceevent-dev [amd64 arm64] , + libtracefs-dev [amd64 arm64] , + libudev-dev , + libunwind8-dev [amd64 arm64] , + llvm-21-dev, + makedumpfile:native [amd64] , + openssl , + pahole (>= 1.29-2ubuntu2) [amd64 arm64] | dwarves (>= 1.21) [amd64 arm64] , + pkg-config , + python3:native , + python3-dev:native , + libpython3-dev , + python3-setuptools, + rsync [!i386] , + rust-src:native [amd64 arm64], + rustc:native (>= 1.82) [amd64 arm64], + rustfmt:native [amd64 arm64], + uuid-dev , + zstd , + bpftool:native [amd64 arm64] , +Build-Depends-Indep: + asciidoc , + bzip2 , + python3-docutils , + sharutils , + xmlto , +Vcs-Git: git://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-nvidia/+git/=SERIES= -b nvidia-bos-next +XS-Testsuite: autopkgtest +#XS-Testsuite-Depends: gcc-4.7 binutils + +Package: SRCPKGNAME-headers-PKGVER-ABINUM +Build-Profiles: +Architecture: all +Multi-Arch: foreign +Section: devel +Priority: optional +Depends: ${misc:Depends}, coreutils +Description: Header files related to Linux kernel version PKGVER + This package provides kernel header files for version PKGVER, for sites + that want the latest kernel headers. Please read + /usr/share/doc/SRCPKGNAME-headers-PKGVER-ABINUM/debian.README.gz for details + +Package: SRCPKGNAME-tools-PKGVER-ABINUM +Build-Profiles: +Architecture: amd64 arm64 +Section: devel +Priority: optional +Depends: ${misc:Depends}, ${shlibs:Depends}, linux-tools-common +Description: Linux kernel version specific tools for version PKGVER-ABINUM + This package provides the architecture dependant parts for kernel + version locked tools (such as perf and x86_energy_perf_policy) for + version PKGVER-ABINUM. + You probably want to install linux-tools-PKGVER-ABINUM-. + +Package: SRCPKGNAME-cloud-tools-PKGVER-ABINUM +Build-Profiles: +Architecture: amd64 +Section: devel +Priority: optional +Depends: ${misc:Depends}, ${shlibs:Depends}, linux-cloud-tools-common +Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM + This package provides the architecture dependant parts for kernel + version locked tools for cloud tools for version PKGVER-ABINUM. + You probably want to install linux-cloud-tools-PKGVER-ABINUM-. diff --git a/debian.nvidia-bos/dkms-versions b/debian.nvidia-bos/dkms-versions new file mode 100644 index 0000000000000..5ccbdb3d3e79c --- /dev/null +++ b/debian.nvidia-bos/dkms-versions @@ -0,0 +1,8 @@ +zfs-linux 2.4.0-1ubuntu3 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms +evdi 1.14.12+dfsg-1ubuntu1 modulename=evdi debpath=pool/universe/e/%package%/evdi-dkms_%version%_all.deb rprovides=evdi-modules rprovides=evdi-dkms type=standalone +ipu6-drivers 0~git202511120800.9766e218-0ubuntu2 modulename=ipu6 debpath=pool/universe/i/%package%/intel-ipu6-dkms_%version%_amd64.deb arch=amd64 rprovides=ipu6-modules rprovides=intel-ipu6-dkms type=standalone +ipu7-drivers 0~git202511120800.fc335577-0ubuntu1 modulename=ipu7 debpath=pool/universe/i/%package%/intel-ipu7-dkms_%version%_amd64.deb arch=amd64 rprovides=ipu7-modules rprovides=intel-ipu7-dkms type=standalone +backport-iwlwifi-dkms 1:0~96.13623-gitd16e74cc-0ubuntu2 modulename=iwlwifi debpath=pool/universe/b/%package%/backport-iwlwifi-dkms_%version%_all.deb arch=amd64 rprovides=iwlwifi-modules rprovides=backport-iwlwifi-dkms type=standalone +v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms +usbio-drivers 0~git202510282139.ee221eca-0ubuntu1 modulename=usbio debpath=pool/universe/u/%package%/intel-usbio-dkms_%version%_amd64.deb arch=amd64 rprovides=usbio-modules rprovides=intel-usbio-dkms type=standalone +vision-drivers 0~git202511121832.a8d772f2-0ubuntu1 modulename=vision debpath=pool/universe/v/%package%/intel-vision-dkms_%version%_amd64.deb arch=amd64 rprovides=vision-modules rprovides=intel-vision-dkms type=standalone diff --git a/debian.nvidia-bos/etc/update.conf b/debian.nvidia-bos/etc/update.conf new file mode 100644 index 0000000000000..3917a390a39f5 --- /dev/null +++ b/debian.nvidia-bos/etc/update.conf @@ -0,0 +1,7 @@ +# WARNING: we do not create update.conf when we are not a +# derivative. Various cranky components make use of this. +# If we start unconditionally creating update.conf we need +# to fix at least cranky close and cranky rebase. +RELEASE_REPO=git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/resolute +SOURCE_RELEASE_BRANCH=master-next +DEBIAN_MASTER=debian.master diff --git a/debian.nvidia-bos/modprobe.d/common.conf b/debian.nvidia-bos/modprobe.d/common.conf new file mode 100644 index 0000000000000..e0fbbd6e060d4 --- /dev/null +++ b/debian.nvidia-bos/modprobe.d/common.conf @@ -0,0 +1,3 @@ +# LP:1434842 -- disable OSS drivers by default to allow pulseaudio to emulate +blacklist snd-mixer-oss +blacklist snd-pcm-oss diff --git a/debian.nvidia-bos/reconstruct b/debian.nvidia-bos/reconstruct new file mode 100644 index 0000000000000..16e52ee71b8a0 --- /dev/null +++ b/debian.nvidia-bos/reconstruct @@ -0,0 +1,34 @@ +# Recreate any symlinks created since the orig. +[ ! -L 'ubuntu/igh-ecat/master/rtdm-ioctl.c' ] && ln -sf 'ioctl.c' 'ubuntu/igh-ecat/master/rtdm-ioctl.c' +chmod +x 'debian/cloud-tools/hv_get_dhcp_info' +chmod +x 'debian/cloud-tools/hv_get_dns_info' +chmod +x 'debian/cloud-tools/hv_set_ifconfig' +chmod +x 'debian/rules' +chmod +x 'debian/scripts/checks/final-checks' +chmod +x 'debian/scripts/checks/module-signature-check' +chmod +x 'debian/scripts/control-create' +chmod +x 'debian/scripts/dkms-build' +chmod +x 'debian/scripts/dkms-build--nvidia-N' +chmod +x 'debian/scripts/dkms-build-configure--zfs' +chmod +x 'debian/scripts/file-downloader' +chmod +x 'debian/scripts/link-headers' +chmod +x 'debian/scripts/link-lib-rust' +chmod +x 'debian/scripts/misc/annotations' +chmod +x 'debian/scripts/misc/find-missing-sauce.sh' +chmod +x 'debian/scripts/misc/gen-auto-reconstruct' +chmod +x 'debian/scripts/misc/git-ubuntu-log' +chmod +x 'debian/scripts/misc/insert-changes' +chmod +x 'debian/scripts/misc/insert-ubuntu-changes' +chmod +x 'debian/scripts/misc/kernelconfig' +chmod +x 'debian/scripts/sign-module' +chmod +x 'debian/templates/extra.postinst.in' +chmod +x 'debian/templates/extra.postrm.in' +chmod +x 'debian/templates/headers.postinst.in' +chmod +x 'debian/templates/image.postinst.in' +chmod +x 'debian/templates/image.postrm.in' +chmod +x 'debian/templates/image.preinst.in' +chmod +x 'debian/templates/image.prerm.in' +chmod +x 'debian/tests/rebuild' +chmod +x 'debian/tests/ubuntu-regression-suite' +# Remove any files deleted from the orig. +exit 0 diff --git a/debian.nvidia-bos/rules.d/amd64.mk b/debian.nvidia-bos/rules.d/amd64.mk new file mode 100644 index 0000000000000..8aa96b3e758cc --- /dev/null +++ b/debian.nvidia-bos/rules.d/amd64.mk @@ -0,0 +1,20 @@ +build_arch = x86 +defconfig = defconfig +flavours = nvidia +build_image = bzImage +kernel_file = arch/$(build_arch)/boot/bzImage +install_file = vmlinuz +vdso = vdso_install +no_dumpfile = true +uefi_signed = true +do_tools_usbip = true +do_tools_cpupower = true +do_tools_perf = true +do_tools_perf_jvmti = true +do_tools_perf_python = true +do_tools_bpftool = true +do_tools_x86 = true +do_tools_hyperv = false +do_tools_rtla = true +do_tools_acpidbg = true +do_lib_rust = false diff --git a/debian.nvidia-bos/rules.d/arm64.mk b/debian.nvidia-bos/rules.d/arm64.mk new file mode 100644 index 0000000000000..f086214eb37ad --- /dev/null +++ b/debian.nvidia-bos/rules.d/arm64.mk @@ -0,0 +1,20 @@ +build_arch = arm64 +defconfig = defconfig +flavours = nvidia nvidia-64k +build_image = vmlinuz.efi +kernel_file = arch/$(build_arch)/boot/vmlinuz.efi +install_file = vmlinuz +no_dumpfile = true +uefi_signed = true + +vdso = vdso_install + +do_tools_usbip = true +do_tools_cpupower = true +do_tools_perf = true +do_tools_perf_jvmti = true +do_tools_perf_python = true +do_tools_bpftool = true +do_tools_rtla = true + +do_dtbs = true diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug new file mode 100644 index 0000000000000..eaf24103d1343 --- /dev/null +++ b/debian.nvidia-bos/tracking-bug @@ -0,0 +1 @@ +2142114 d2026.02.16-1 diff --git a/debian.nvidia-bos/upstream-stable b/debian.nvidia-bos/upstream-stable new file mode 100644 index 0000000000000..30e10948e88f9 --- /dev/null +++ b/debian.nvidia-bos/upstream-stable @@ -0,0 +1,3 @@ +# The following upstream stable releases have been ported: +[upstream-stable] + linux-6.17.y = v6.17.1 diff --git a/debian.nvidia-bos/variants b/debian.nvidia-bos/variants new file mode 100644 index 0000000000000..6606318691bcd --- /dev/null +++ b/debian.nvidia-bos/variants @@ -0,0 +1,4 @@ +-6.19 +-- +-hwe-24.04 +-hwe-24.04-edge diff --git a/debian/debian.env b/debian/debian.env index be31a0c270197..60e59e55632e7 100644 --- a/debian/debian.env +++ b/debian/debian.env @@ -1 +1 @@ -DEBIAN=debian.master +DEBIAN=debian.nvidia-bos From 4be5fb23dfa941c51c45f6d837057e2c116a8a48 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 8 Apr 2025 09:00:03 -0500 Subject: [PATCH 025/464] UBUNTU: [Config] nvidia-6.14: import misc configs from noble:linux-nvidia Ignore: yes Signed-off-by: Jacob Martin (cherry picked from commit 1a32c7f18ea7bf2799f3413ecdae10e9de21da74 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 0d1a2deae660bfb5061640b96fe7a67402e63750 noble:linux-nvidia-6.17) [jacobmartin: dropped uses of CONFIG_PREEMPT_NONE / CONFIG_PREEMPT_VOLUNTARY, these have been disabled upstream for arm64 and amd64 arches by commit 7dadeaa6e851 ("sched: Further restrict the preemption modes") in favor of CONFIG_PREEMPT_LAZY, which is default in the parent kernel.] Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 31 +++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 66f95c1d5d6bd..2fb4abe2cc9af 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -6,9 +6,34 @@ include "../../debian.master/config/annotations" +CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND policy<{'arm64': 'n'}> +CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND note<'required for NVIDIA workloads'> + +CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE policy<{'amd64': 'n', 'arm64': 'y'}> +CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'required for NVIDIA workloads'> + +CONFIG_MTD policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_MTD note<'Essential for boot on ARM64'> + +CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> +CONFIG_NR_CPUS note<'LP: #1864198'> + +CONFIG_SPI_TEGRA210_QUAD policy<{'arm64': 'y'}> +CONFIG_SPI_TEGRA210_QUAD note<'Ensures the TPM is available before the IMA driver initializes'> + +CONFIG_TCG_TIS_SPI policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_TCG_TIS_SPI note<'Ensures the TPM is available before the IMA driver initializes'> + +CONFIG_UBUNTU_ODM_DRIVERS policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM drivers'> + # ---- Annotations without notes ---- -CONFIG_AS_VERSION policy<{'amd64': '24600', 'arm64': '24600'}> -CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-14ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-13ubuntu3) 15.2.0"'}> -CONFIG_LD_VERSION policy<{'amd64': '24600', 'arm64': '24600'}> +CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"'}> +CONFIG_GPIO_AAEON policy<{'amd64': '-'}> +CONFIG_LEDS_AAEON policy<{'amd64': '-'}> +CONFIG_MFD_AAEON policy<{'amd64': '-'}> +CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_SENSORS_AAEON policy<{'amd64': '-'}> From 5fbfd70de930d48663d3b25c3702d3caedc4c108 Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Fri, 5 Jan 2024 08:18:39 -0800 Subject: [PATCH 026/464] UBUNTU: [Packaging] dkms-versions standalone provides support Add support for exposing rprovides data for standalone modules too. Switch to exposing provides as a shared debian/substvar file and use that in the templates. Ignore: yes Signed-off-by: Brad Figg Signed-off-by: Ian May (cherry picked from commit afacdda832a97ab283d8e88d64fae1d7ce5b7060 noble:linux-nvidia/main-next) Signed-off-by: Jacob Martin (cherry picked from commit 52ba185348888cbe7f723260f751a552dfb3a78a) (cherry picked from commit 52ba18534888 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 8f0710a888eff2b25f67a6599377625f3fc54d7a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian/control.d/flavour-module.stub | 1 + debian/rules | 4 ++++ debian/rules.d/2-binary-arch.mk | 3 +-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/debian/control.d/flavour-module.stub b/debian/control.d/flavour-module.stub index 2810f83bb361f..4aa9ddbe76b95 100644 --- a/debian/control.d/flavour-module.stub +++ b/debian/control.d/flavour-module.stub @@ -4,6 +4,7 @@ Build-Profiles: Architecture: ARCH Section: kernel Priority: optional +Provides: ${MODULE:rprovides} Depends: ${misc:Depends}, linux-image-PKGVER-ABINUM-FLAVOUR | linux-image-unsigned-PKGVER-ABINUM-FLAVOUR, diff --git a/debian/rules b/debian/rules index 4f87ba2dcf796..7ab8e690e038a 100755 --- a/debian/rules +++ b/debian/rules @@ -147,6 +147,10 @@ clean: debian/control debian/canonical-certs.pem debian/canonical-revoked-certs. rm -f debian/scripts/fix-filenames + # SUBSTVARS: rprovides for all DKMS packages + echo "linux:rprovides=$(foreach dkms,$(all_built-in_dkms_modules),$(foreach provides,$(dkms_$(dkms)_rprovides),$(provides)$(comma)))" >"debian/substvars" + echo "$(foreach dkms,$(all_standalone_dkms_modules),$(dkms):rprovides=$(foreach provides,$(dkms_$(dkms)_rprovides),$(provides)$(comma))=NL=)" | sed -e "s/=NL= */\n/g" >>"debian/substvars" + .PHONY: distclean distclean: clean rm -rf debian/control debian/changelog diff --git a/debian/rules.d/2-binary-arch.mk b/debian/rules.d/2-binary-arch.mk index a249f782bca8e..7a8bb73afd78b 100644 --- a/debian/rules.d/2-binary-arch.mk +++ b/debian/rules.d/2-binary-arch.mk @@ -499,7 +499,7 @@ define dh_all dh_shlibdeps -p$(1) $(shlibdeps_opts) dh_installdeb -p$(1) dh_installdebconf -p$(1) - $(lockme) dh_gencontrol -p$(1) -- -Vlinux:rprovides='$(rprovides)' $(2) + $(lockme) dh_gencontrol -p$(1) -- -Tdebian/substvars $(2) dh_md5sums -p$(1) dh_builddeb -p$(1) endef @@ -531,7 +531,6 @@ binary-%: pkgcloud = $(cloud_flavour_pkg_name)-$* $(foreach _m,$(all_dkms_modules), \ $(eval binary-%: enable_$(_m) = $$(filter true,$$(call custom_override,do_$(_m),$$*))) \ ) -binary-%: rprovides = $(foreach _m,$(all_built-in_dkms_modules),$(if $(enable_$(_m)),$(foreach _r,$(dkms_$(_m)_rprovides),$(_r)$(comma) ))) binary-%: $(stampdir)/stamp-install-% @echo Debug: $@ dh_testdir From 25b4b6578a7685beb462a74749e3a94384cf712b Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 11 Mar 2026 11:38:23 -0500 Subject: [PATCH 027/464] Revert "UBUNTU: [Packaging] remove stale debian/dkms-versions scripting" This reverts commit 7a51fffb97c621f80da06bd0c8442359f78d735d. This stale debian/dkms-versions scripting is still used for derivatives of linux without a linux-main-modules package to parse the main package's dkms-versions file for out-of-tree module builds. Ignore: yes Signed-off-by: Jacob Martin --- debian/rules.d/0-common-vars.mk | 47 +++++++++++++++++++++++++++++++++ debian/scripts/control-create | 27 +++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/debian/rules.d/0-common-vars.mk b/debian/rules.d/0-common-vars.mk index 1ea787d668dcc..e68d6eb170198 100644 --- a/debian/rules.d/0-common-vars.mk +++ b/debian/rules.d/0-common-vars.mk @@ -219,3 +219,50 @@ custom_override = $(or $($(1)_$(2)),$($(1))) # selftests that Ubuntu cares about ubuntu_selftests = breakpoints cpu-hotplug efivarfs memfd memory-hotplug mount net ptrace seccomp timers powerpc user ftrace + +# DKMS +all_dkms_modules = + +subst_paired = $(subst $(firstword $(subst =, ,$(1))),$(lastword $(subst =, ,$(1))),$(2)) +recursive_call = $(if $(2),$(call recursive_call,$(1),$(wordlist 2,$(words $(2)),$(2)),$(call $(1),$(firstword $(2)),$(3))),$(3)) + +$(foreach _line,$(shell gawk '{ OFS = "!"; $$1 = $$1; print }' $(DEBIAN)/dkms-versions), \ + $(eval _params = $(subst !, ,$(_line))) \ + $(eval _deb_pkgname = $(firstword $(_params))) \ + $(eval _deb_version = $(word 2,$(_params))) \ + $(if $(filter modulename=%,$(_params)), \ + $(eval _m = $(word 2,$(subst =, ,$(filter modulename=%,$(_params))))) \ + , \ + $(info modulename for $(_deb_pkgname) not specified in dkms-versions. Assume $(_deb_pkgname).) \ + $(eval _m = $(_deb_pkgname)) \ + ) \ + $(eval all_dkms_modules += $(_m)) \ + $(eval dkms_$(_m)_version = $(_deb_version)) \ + $(foreach _p,$(patsubst debpath=%,%,$(filter debpath=%,$(_params))), \ + $(eval dkms_$(_m)_debpath += $(strip \ + $(call recursive_call,subst_paired, \ + %module%=$(_m) \ + %package%=$(_deb_pkgname) \ + %version%=$(lastword $(subst :, ,$(_deb_version))) \ + , \ + $(_p) \ + ) \ + )) \ + ) \ + $(if $(dkms_$(_m)_debpath),,$(error debpath for $(_deb_pkgname) not specified.)) \ + $(if $(filter arch=%,$(_params)), \ + $(eval dkms_$(_m)_archs = $(patsubst arch=%,%,$(filter arch=%,$(_params)))) \ + , \ + $(eval dkms_$(_m)_archs = any) \ + ) \ + $(eval dkms_$(_m)_rprovides = $(patsubst rprovides=%,%,$(filter rprovides=%,$(_params)))) \ + $(eval dkms_$(_m)_type = $(word 1,$(patsubst type=%,%,$(filter type=%,$(_params))) built-in)) \ + $(eval all_$(dkms_$(_m)_type)_dkms_modules += $(_m)) \ + $(if $(filter standalone,$(dkms_$(_m)_type)), \ + $(eval dkms_$(_m)_pkg_name = linux-modules-$(_m)-$(abi_release)) \ + $(eval dkms_$(_m)_subdir = ubuntu) \ + , \ + $(eval dkms_$(_m)_pkg_name = $(mods_pkg_name)) \ + $(eval dkms_$(_m)_subdir = kernel) \ + ) \ +) diff --git a/debian/scripts/control-create b/debian/scripts/control-create index f96da1ec1168c..acdaa36ec5144 100755 --- a/debian/scripts/control-create +++ b/debian/scripts/control-create @@ -100,6 +100,33 @@ gen_per_flavour () { sed "${sed_common_patterns[@]}" \ -e "s/ARCH/${arch}/g" \ "debian/control.d/flavour-buildinfo.stub" + + while read -r package version extras + do + module="$package" + module_type= + + # Module arch parameters are skipped here, so a package section will + # be generated for each flavour, and its Architecture will be set to + # all architectures with that flavour. Even that is being generated, + # it doesn't follow all of them will be built. That's to work-around + # dkms_exclude/dkms_include that manipulates supported architectures + # in $(DEBIAN)/rules.d/$(arch).mk. + for param in $extras; do + case "$param" in + modulename=*) module="${param#modulename=}" ;; + type=*) module_type="${param#type=}" ;; + *) continue ;; + esac + done + + [ "$module_type" = "standalone" ] || continue + + sed "${sed_common_patterns[@]}" \ + -e "s/ARCH/${arch}/g" \ + -e "s/MODULE/${module}/g" \ + debian/control.d/flavour-module.stub + done < "${DEBIAN}/dkms-versions" } gen_common From fde515bacbd33d020d42ce295b448da8c4ed4fe5 Mon Sep 17 00:00:00 2001 From: Ian May Date: Thu, 21 Mar 2024 17:12:26 -0500 Subject: [PATCH 028/464] UBUNTU: [Packaging] add versioning to dkms standalone rprovides When nvidia-fs-dkms is available as a dkms package, we want to default to using the signed modules if possible. Adding a version number for the nvidia-fs modules package enables the inbox modules to be selected over an equivalent dkms version. Ignore: yes Signed-off-by: Ian May (cherry picked from commit 607379d81d95894ef1f2575008242c50ba7c5d72 noble:linux-nvidia/main-next) Signed-off-by: Jacob Martin (cherry picked from commit f6927df081a74a0f84b3ca49b9e699c7bef361a6) (cherry picked from commit f6927df081a7 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 750ba56ba02182b042ce337385880f8e33e0fa11 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian/rules | 2 +- debian/rules.d/0-common-vars.mk | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/debian/rules b/debian/rules index 7ab8e690e038a..ec843ccde5ef5 100755 --- a/debian/rules +++ b/debian/rules @@ -149,7 +149,7 @@ clean: debian/control debian/canonical-certs.pem debian/canonical-revoked-certs. # SUBSTVARS: rprovides for all DKMS packages echo "linux:rprovides=$(foreach dkms,$(all_built-in_dkms_modules),$(foreach provides,$(dkms_$(dkms)_rprovides),$(provides)$(comma)))" >"debian/substvars" - echo "$(foreach dkms,$(all_standalone_dkms_modules),$(dkms):rprovides=$(foreach provides,$(dkms_$(dkms)_rprovides),$(provides)$(comma))=NL=)" | sed -e "s/=NL= */\n/g" >>"debian/substvars" + echo "$(foreach dkms,$(all_standalone_dkms_modules),$(dkms):rprovides=$(strip $(foreach provides,$(dkms_$(dkms)_rprovides),$(provides)$(comma)))=NL=)" | sed -e 's/~(/ (/g' -e 's/, (/ (/g' -e 's/=NL= */\n/g' >>"debian/substvars" .PHONY: distclean distclean: clean diff --git a/debian/rules.d/0-common-vars.mk b/debian/rules.d/0-common-vars.mk index e68d6eb170198..17af44bc5239f 100644 --- a/debian/rules.d/0-common-vars.mk +++ b/debian/rules.d/0-common-vars.mk @@ -255,7 +255,8 @@ $(foreach _line,$(shell gawk '{ OFS = "!"; $$1 = $$1; print }' $(DEBIAN)/dkms-ve , \ $(eval dkms_$(_m)_archs = any) \ ) \ - $(eval dkms_$(_m)_rprovides = $(patsubst rprovides=%,%,$(filter rprovides=%,$(_params)))) \ + $(eval _rprovides_raw = $(filter rprovides=%,$(_params))) \ + $(eval dkms_$(_m)_rprovides = $(patsubst rprovides=%,%,$(_rprovides_raw))) \ $(eval dkms_$(_m)_type = $(word 1,$(patsubst type=%,%,$(filter type=%,$(_params))) built-in)) \ $(eval all_$(dkms_$(_m)_type)_dkms_modules += $(_m)) \ $(if $(filter standalone,$(dkms_$(_m)_type)), \ From 4c1c88e2b98f4b1b462002f2b4156d71c67cc667 Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Thu, 4 Apr 2024 11:22:16 -0700 Subject: [PATCH 029/464] NVIDIA: [Config]: Disable the NOUVEAU driver which is not used with -nvidia kernels BugLink: https://bugs.launchpad.net/bugs/2060327 Signed-off-by: Brad Figg Acked-by: Brad Figg Signed-off-by: Ian May [jacobmartin: Add note to changed configs] Signed-off-by: Jacob Martin (cherry picked from commit 9b2615a63f73 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e3b5061c441ebca291c15b99260894d7eeb9f034 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 30 ++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 2fb4abe2cc9af..743a202d78fdd 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -12,9 +12,39 @@ CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND note<'required for NVIDIA worklo CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE policy<{'amd64': 'n', 'arm64': 'y'}> CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'required for NVIDIA workloads'> +CONFIG_DRM_NOUVEAU policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_DRM_NOUVEAU note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_DRM_NOUVEAU_BACKLIGHT policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_DRM_NOUVEAU_BACKLIGHT note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_DRM_NOUVEAU_CH7006 policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_DRM_NOUVEAU_CH7006 note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_DRM_NOUVEAU_SIL164 policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_DRM_NOUVEAU_SIL164 note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_DRM_NOUVEAU_SVM policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_DRM_NOUVEAU_SVM note<'Disable nouveau for NVIDIA kernels'> + CONFIG_MTD policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_MTD note<'Essential for boot on ARM64'> +CONFIG_NOUVEAU_DEBUG policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_NOUVEAU_DEBUG note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_NOUVEAU_DEBUG_DEFAULT policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_NOUVEAU_DEBUG_DEFAULT note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_NOUVEAU_DEBUG_MMU policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_NOUVEAU_DEBUG_MMU note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_NOUVEAU_DEBUG_PUSH policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_NOUVEAU_DEBUG_PUSH note<'Disable nouveau for NVIDIA kernels'> + +CONFIG_NOUVEAU_PLATFORM_DRIVER policy<{'arm64': '-'}> +CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA kernels'> + CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> From ebef7cbd833b687e783d94eaf6fb6c728dcc897d Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Fri, 5 Apr 2024 11:57:09 -0700 Subject: [PATCH 030/464] NVIDIA: [Config]: Adding CORESIGHT and ARM64_ERRATUM configs to annotations BugLink: https://bugs.launchpad.net/bugs/2060327 Signed-off-by: Brad Figg Acked-by: Brad Figg Signed-off-by: Ian May [jacobmartin: Add annotations note for changed configs] Signed-off-by: Jacob Martin (cherry picked from commit 3d31ea05380f noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 6de4075392d6b80387919934a10ac91c7fdfebb4 noble:linux-nvidia-6.17) [jacobmartin: set new CoreSight configs: - CONFIG_CORESIGHT_TNOC=m - CONFIG_CORESIGHT_CTCU=m] Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 93 ++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 743a202d78fdd..07524eed867db 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -6,6 +6,87 @@ include "../../debian.master/config/annotations" +CONFIG_ARM64_ERRATUM_1902691 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_1902691 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2038923 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2038923 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2064142 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2064142 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2119858 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2119858 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2139208 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2139208 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2224489 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2224489 note<'Required for Grace enablement'> + +CONFIG_ARM64_ERRATUM_2253138 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_2253138 note<'Required for Grace enablement'> + +CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE policy<{'arm64': 'y'}> +CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enablement'> + +CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE policy<{'arm64': 'y'}> +CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablement'> + +CONFIG_CORESIGHT policy<{'arm64': 'm'}> +CONFIG_CORESIGHT note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CATU policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_CATU note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CPU_DEBUG policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_CPU_DEBUG note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CPU_DEBUG_DEFAULT_ON policy<{'arm64': 'n'}> +CONFIG_CORESIGHT_CPU_DEBUG_DEFAULT_ON note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CTCU policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_CTCU note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CTI policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_CTI note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_CTI_INTEGRATION_REGS policy<{'arm64': 'n'}> +CONFIG_CORESIGHT_CTI_INTEGRATION_REGS note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_DUMMY policy<{'arm64': 'n'}> +CONFIG_CORESIGHT_DUMMY note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_LINKS_AND_SINKS policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_LINKS_AND_SINKS note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_LINK_AND_SINK_TMC policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_LINK_AND_SINK_TMC note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_SINK_ETBV10 policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_SINK_ETBV10 note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_SINK_TPIU policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_SINK_TPIU note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_SOURCE_ETM4X policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_SOURCE_ETM4X note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_STM policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_STM note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_TNOC policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_TNOC note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_TPDA policy<{'arm64': 'n'}> +CONFIG_CORESIGHT_TPDA note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_TPDM policy<{'arm64': 'n'}> +CONFIG_CORESIGHT_TPDM note<'Required for Grace enablement'> + +CONFIG_CORESIGHT_TRBE policy<{'arm64': 'm'}> +CONFIG_CORESIGHT_TRBE note<'Required for Grace enablement'> + CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND policy<{'arm64': 'n'}> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND note<'required for NVIDIA workloads'> @@ -27,6 +108,9 @@ CONFIG_DRM_NOUVEAU_SIL164 note<'Disable nouveau for NVIDIA CONFIG_DRM_NOUVEAU_SVM policy<{'amd64': '-', 'arm64': '-'}> CONFIG_DRM_NOUVEAU_SVM note<'Disable nouveau for NVIDIA kernels'> +CONFIG_ETM4X_IMPDEF_FEATURE policy<{'arm64': 'n'}> +CONFIG_ETM4X_IMPDEF_FEATURE note<'Required for Grace enablement'> + CONFIG_MTD policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_MTD note<'Essential for boot on ARM64'> @@ -48,6 +132,12 @@ CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> +CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> +CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> + +CONFIG_SAMPLE_CORESIGHT_SYSCFG policy<{'arm64': 'n'}> +CONFIG_SAMPLE_CORESIGHT_SYSCFG note<'Required for Grace enablement'> + CONFIG_SPI_TEGRA210_QUAD policy<{'arm64': 'y'}> CONFIG_SPI_TEGRA210_QUAD note<'Ensures the TPM is available before the IMA driver initializes'> @@ -57,6 +147,9 @@ CONFIG_TCG_TIS_SPI note<'Ensures the TPM is availab CONFIG_UBUNTU_ODM_DRIVERS policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM drivers'> +CONFIG_ULTRASOC_SMB policy<{'arm64': 'n'}> +CONFIG_ULTRASOC_SMB note<'Required for Grace enablement'> + # ---- Annotations without notes ---- From 3fcb784380e16b846171a62f05d78a3baa2ac4d6 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 17 Oct 2024 15:01:53 -0500 Subject: [PATCH 031/464] UBUNTU: [Config] Disable Ubuntu ODM drivers for NVIDIA kernels Ignore: yes Signed-off-by: Jacob Martin (cherry picked from commit 448ddcb3bec6206e2e86f653574c0e7a1ce30fac) (cherry picked from commit 448ddcb3bec6 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit b4e9b917c8c779f30c4873f9a0bc90f2f66a5e1e noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 07524eed867db..2b9471014acf1 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -111,6 +111,15 @@ CONFIG_DRM_NOUVEAU_SVM note<'Disable nouveau for NVIDIA CONFIG_ETM4X_IMPDEF_FEATURE policy<{'arm64': 'n'}> CONFIG_ETM4X_IMPDEF_FEATURE note<'Required for Grace enablement'> +CONFIG_GPIO_AAEON policy<{'amd64': '-'}> +CONFIG_GPIO_AAEON note<'Disable all Ubuntu ODM drivers'> + +CONFIG_LEDS_AAEON policy<{'amd64': '-'}> +CONFIG_LEDS_AAEON note<'Disable all Ubuntu ODM drivers'> + +CONFIG_MFD_AAEON policy<{'amd64': '-'}> +CONFIG_MFD_AAEON note<'Disable all Ubuntu ODM drivers'> + CONFIG_MTD policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_MTD note<'Essential for boot on ARM64'> @@ -138,6 +147,9 @@ CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablem CONFIG_SAMPLE_CORESIGHT_SYSCFG policy<{'arm64': 'n'}> CONFIG_SAMPLE_CORESIGHT_SYSCFG note<'Required for Grace enablement'> +CONFIG_SENSORS_AAEON policy<{'amd64': '-'}> +CONFIG_SENSORS_AAEON note<'Disable all Ubuntu ODM drivers'> + CONFIG_SPI_TEGRA210_QUAD policy<{'arm64': 'y'}> CONFIG_SPI_TEGRA210_QUAD note<'Ensures the TPM is available before the IMA driver initializes'> @@ -155,8 +167,4 @@ CONFIG_ULTRASOC_SMB note<'Required for Grace enablem CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"'}> -CONFIG_GPIO_AAEON policy<{'amd64': '-'}> -CONFIG_LEDS_AAEON policy<{'amd64': '-'}> -CONFIG_MFD_AAEON policy<{'amd64': '-'}> CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> -CONFIG_SENSORS_AAEON policy<{'amd64': '-'}> From cf7c999b6ec9f3b51a5b380149bb010c5ae20ad1 Mon Sep 17 00:00:00 2001 From: Ian May Date: Wed, 24 Apr 2024 22:45:17 -0500 Subject: [PATCH 032/464] UBUNTU: [Packaging] blacklist coresight_etm4x BugLink: https://bugs.launchpad.net/bugs/2061930 BugLink: https://bugs.launchpad.net/bugs/2067106 There are systems in production that don't have firmware that supports coresight_etm4x. Instead of removing completely, blacklist coresight_etm4x so systems with the correct firmware can use the module. Signed-off-by: Ian May Signed-off-by: Jamie Nguyen Acked-by: Brad Figg Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg Signed-off-by: Jacob Martin (backported from commit 217d1ae2aad8b33ff247bdab358f9134b90f6d4e noble:linux-nvidia-6.14) [maskedarray: adjusted context] Signed-off-by: Abdur Rahman (cherry picked from commit 3f7d9007f3b47889a2ce7eecf773bf99bf78f1d5 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/modprobe.d/common.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/debian.nvidia-bos/modprobe.d/common.conf b/debian.nvidia-bos/modprobe.d/common.conf index e0fbbd6e060d4..619c9a23fe210 100644 --- a/debian.nvidia-bos/modprobe.d/common.conf +++ b/debian.nvidia-bos/modprobe.d/common.conf @@ -1,3 +1,4 @@ # LP:1434842 -- disable OSS drivers by default to allow pulseaudio to emulate blacklist snd-mixer-oss blacklist snd-pcm-oss +blacklist coresight_etm4x From bf03871135819beb7ee21b3839e6969cb8fed56a Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Wed, 7 Aug 2024 11:13:22 -0700 Subject: [PATCH 033/464] NVIDIA: [Config] EFI: set CAPSULE_LOADER=y for arm64 BugLink: https://bugs.launchpad.net/bugs/2067111 Nvidia provide a way to flash the UEFI via capsule loader in arm64. CAPSULE_LOADER is also built-in in L4T kernel so for the easy use, need to make CAPSULE_LOADER as built-in in arm64. Nvidia-BugLink: https://nvbugspro.nvidia.com/bug/4601764 Signed-off-by: Brad Figg Acked-by: Jacob Martin Acked-by: Noah Wager (cherry picked from commit efbc80a791437c6bd3f477c656d6d84970f826d8 noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit 58d6077a21f6914b3b274dd1c29757343a29e32b) (cherry picked from commit 58d6077a21f6 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 812ae1ef0031de4d63257648e6d2a4c6708174df noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 2b9471014acf1..bbc3a8faf0d49 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -108,6 +108,9 @@ CONFIG_DRM_NOUVEAU_SIL164 note<'Disable nouveau for NVIDIA CONFIG_DRM_NOUVEAU_SVM policy<{'amd64': '-', 'arm64': '-'}> CONFIG_DRM_NOUVEAU_SVM note<'Disable nouveau for NVIDIA kernels'> +CONFIG_EFI_CAPSULE_LOADER policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_EFI_CAPSULE_LOADER note<'LP: #2067111'> + CONFIG_ETM4X_IMPDEF_FEATURE policy<{'arm64': 'n'}> CONFIG_ETM4X_IMPDEF_FEATURE note<'Required for Grace enablement'> From 334e3efa7912d70f4d2798be2106206d3f72560b Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Wed, 15 Nov 2023 10:27:43 +0000 Subject: [PATCH 034/464] NVIDIA: SAUCE: WAR: iommufd/pages: Bypass PFNMAP BugLink: https://bugs.launchpad.net/bugs/2095028 This is used for GPU memory mapping. The solution is a WAR while waiting for the upstream solution that would use dmabuf to map the entire range in a single sequence. Related topics: https://lore.kernel.org/kvm/20240624065552.1572580-1-vivek.kasireddy@intel.com/ https://lore.kernel.org/kvm/cover.1719909395.git.leon@kernel.org/ Signed-off-by: Ankit Agrawal (cherry picked from commit d3d7b64f1a3274e5df04dee1a8062f54a3fa1116 nvidia/kstable/dev/nic/iommufd_vsmmu-12122024) Signed-off-by: Koba Ko Acked-by: Matt Ochs Acked-by: Brad Figg Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 15e066a3cc7484e59f1e1c26d651947c98cf42dd noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit 8fcaed8d5824 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit ef306c832155a7f6555dcc428e995f54b5bab4a1 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/iommu/iommufd/pages.c | 81 +++++++++++++++++++++++++++++++++-- 1 file changed, 77 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/iommufd/pages.c b/drivers/iommu/iommufd/pages.c index 9b49f0c5b4599..27a1628e92a08 100644 --- a/drivers/iommu/iommufd/pages.c +++ b/drivers/iommu/iommufd/pages.c @@ -711,9 +711,10 @@ static void batch_unpin(struct pfn_batch *batch, struct iopt_pages *pages, size_t to_unpin = min_t(size_t, npages, batch->npfns[cur] - first_page_off); - unpin_user_page_range_dirty_lock( - pfn_to_page(batch->pfns[cur] + first_page_off), - to_unpin, pages->writable); + if (pfn_valid(batch->pfns[cur] + first_page_off)) + unpin_user_page_range_dirty_lock( + pfn_to_page(batch->pfns[cur] + first_page_off), + to_unpin, pages->writable); iopt_pages_sub_npinned(pages, to_unpin); cur++; first_page_off = 0; @@ -873,6 +874,41 @@ static long pin_memfd_pages(struct pfn_reader_user *user, unsigned long start, return npages_out; } +static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, + unsigned long vaddr, unsigned long *pfn, + bool write_fault) +{ + struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; + int ret; + + ret = follow_pfnmap_start(&args); + if (ret) { + bool unlocked = false; + + ret = fixup_user_fault(mm, vaddr, + FAULT_FLAG_REMOTE | + (write_fault ? FAULT_FLAG_WRITE : 0), + &unlocked); + if (unlocked) + return -EAGAIN; + + if (ret) + return ret; + + ret = follow_pfnmap_start(&args); + if (ret) + return ret; + } + + if (write_fault && !args.writable) + ret = -EFAULT; + else + *pfn = args.pfn; + + follow_pfnmap_end(&args); + return ret; +} + static int pfn_reader_user_pin(struct pfn_reader_user *user, struct iopt_pages *pages, unsigned long start_index, @@ -941,6 +977,42 @@ static int pfn_reader_user_pin(struct pfn_reader_user *user, user->gup_flags, user->upages, &user->locked); } + + if (rc < 0) { + struct vm_area_struct *vma; + unsigned long vaddr; + unsigned long pfn; + int pinned = 0; + + /* fast path above doesn't hold the lock */ + if (!user->locked) + mmap_read_lock(pages->source_mm); + vaddr = untagged_addr_remote(pages->source_mm, uptr); +retry: + vma = vma_lookup(pages->source_mm, vaddr); + if (vma && vma->vm_flags & VM_PFNMAP) { + do { + rc = follow_fault_pfn(vma, pages->source_mm, vaddr, + &pfn, pages->writable); + if (rc == -EAGAIN) + goto retry; + if (!rc) { + if (!pfn_valid(pfn)) { + user->upages[pinned] = pfn_to_page(pfn); + pinned += 1; + vaddr += PAGE_SIZE; + } else { + rc = -EFAULT; + } + } + } while (pinned < npages && vaddr < vma->vm_end && !rc); + } + if (pinned) + rc = pinned; + if (!user->locked) + mmap_read_unlock(pages->source_mm); + } + if (rc <= 0) { if (WARN_ON(!rc)) return -EFAULT; @@ -1313,7 +1385,8 @@ static void pfn_reader_release_pins(struct pfn_reader *pfns) user->upages_start; if (!user->file) { - unpin_user_pages(user->upages + start_index, npages); + if (pfn_valid(page_to_pfn(user->upages[0]))) + unpin_user_pages(user->upages + start_index, npages); } else { long n = user->ufolios_len / sizeof(*user->ufolios); From 118b0477c9a0fbf5240d0490aadf9dc7f2dec190 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Tue, 16 Jul 2024 01:47:44 +0000 Subject: [PATCH 035/464] NVIDIA: SAUCE: [Config] nvidia: Update annotations for Grace I/O virtualization BugLink: https://bugs.launchpad.net/bugs/2095028 This adds the following config options to annotations: CONFIG_ARM_SMMU_V3_IOMMUFD=y CONFIG_IOMMUFD_DRIVER_CORE=y CONFIG_IOMMUFD_VFIO_CONTAINER=y CONFIG_NVGRACE_GPU_VFIO_PCI=m CONFIG_VFIO_CONTAINER=n CONFIG_VFIO_IOMMU_TYPE1=- For CMA size requirements, the 64K kernel configuration needs 640MB in the worst-case scenario, while the 4K kernel configuration requires 40MB. Due to the current CMA alignment requirement of 512MB on 64k kernel and 128MB on 4k kernel, use each as default For 64k kernel, CONFIG_CMA_SIZE_MBYTES=1024 For 4k kernel, CONFIG_CMA_SIZE_MBYTES=128 These config options has been defined in debian.master CONFIG_IOMMUFD=m CONFIG_IOMMU_IOPF=y Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (backported from commit 35a55f343e80627c03640759886aa5d1c732acdf 24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs Acked-by: Brad Figg Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (backported from commit 1314cf03bfb0510f83cb861c7345a65c8c2e25a9 noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit d09b7e27c860 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (backported from commit 3660ee50e5984e96c615133e96ea90b11651193c noble:linux-nvidia-6.17) [mochs: Removed CONFIG_TEGRA241_CMDQV=n; we want it =y from debian.master] Signed-off-by: Matthew R. Ochs --- debian.nvidia-bos/config/annotations | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index bbc3a8faf0d49..7bfa5bcee00ba 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -33,6 +33,12 @@ CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enable CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablement'> +CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> +CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> + +CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '32', 'arm64-nvidia': '128', 'arm64-nvidia-64k': '1024'}> +CONFIG_CMA_SIZE_MBYTES note<'LP: #2095028'> + CONFIG_CORESIGHT policy<{'arm64': 'm'}> CONFIG_CORESIGHT note<'Required for Grace enablement'> @@ -117,6 +123,9 @@ CONFIG_ETM4X_IMPDEF_FEATURE note<'Required for Grace enablem CONFIG_GPIO_AAEON policy<{'amd64': '-'}> CONFIG_GPIO_AAEON note<'Disable all Ubuntu ODM drivers'> +CONFIG_IOMMUFD_VFIO_CONTAINER policy<{'arm64': 'y'}> +CONFIG_IOMMUFD_VFIO_CONTAINER note<'LP: #2095028'> + CONFIG_LEDS_AAEON policy<{'amd64': '-'}> CONFIG_LEDS_AAEON note<'Disable all Ubuntu ODM drivers'> @@ -165,6 +174,12 @@ CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM dri CONFIG_ULTRASOC_SMB policy<{'arm64': 'n'}> CONFIG_ULTRASOC_SMB note<'Required for Grace enablement'> +CONFIG_VFIO_CONTAINER policy<{'amd64': 'y', 'arm64': 'n'}> +CONFIG_VFIO_CONTAINER note<'LP: #2095028'> + +CONFIG_VFIO_IOMMU_TYPE1 policy<{'amd64': 'm', 'arm64': '-'}> +CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> + # ---- Annotations without notes ---- From 1f65fad693914fec6964e3b46f68d279e0e083f9 Mon Sep 17 00:00:00 2001 From: Yenchia Chen Date: Tue, 11 Feb 2025 10:36:31 +0800 Subject: [PATCH 036/464] NVIDIA: SAUCE: serial: 8250_mtk: Add ACPI support BugLink: https://bugs.launchpad.net/bugs/2096888 Add ACPI support to 8250_mtk driver. This makes it possible to use UART on ARM-based desktops with EDK2 UEFI firmware. Acked-by: Brad Figg Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 4647186b002bde9bf50ec26db8a776c3f22b6196 noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit d73760e5ac8d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 072848cc6957a7dea5078840c0b84473b2b14861 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/tty/serial/8250/8250_mtk.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/drivers/tty/serial/8250/8250_mtk.c b/drivers/tty/serial/8250/8250_mtk.c index 5875a7b9b4b10..39e8268cd4b9a 100644 --- a/drivers/tty/serial/8250/8250_mtk.c +++ b/drivers/tty/serial/8250/8250_mtk.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "8250.h" @@ -521,6 +522,7 @@ static int mtk8250_probe(struct platform_device *pdev) struct mtk8250_data *data; struct resource *regs; int irq, err; + struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev); irq = platform_get_irq(pdev, 0); if (irq < 0) @@ -543,12 +545,13 @@ static int mtk8250_probe(struct platform_device *pdev) data->clk_count = 0; - if (pdev->dev.of_node) { + if (is_of_node(fwnode)) { err = mtk8250_probe_of(pdev, &uart.port, data); if (err) return err; - } else + } else if (!fwnode) { return -ENODEV; + } spin_lock_init(&uart.port.lock); uart.port.mapbase = regs->start; @@ -564,14 +567,18 @@ static int mtk8250_probe(struct platform_device *pdev) uart.port.startup = mtk8250_startup; uart.port.set_termios = mtk8250_set_termios; uart.port.uartclk = clk_get_rate(data->uart_clk); + if (!uart.port.uartclk) + uart.port.uartclk = 26 * HZ_PER_MHZ; #ifdef CONFIG_SERIAL_8250_DMA if (data->dma) uart.dma = data->dma; #endif - /* Disable Rate Fix function */ - writel(0x0, uart.port.membase + + if (is_of_node(fwnode)) { + /* Disable Rate Fix function */ + writel(0x0, uart.port.membase + (MTK_UART_RATE_FIX << uart.port.regshift)); + } platform_set_drvdata(pdev, data); @@ -649,11 +656,18 @@ static const struct of_device_id mtk8250_of_match[] = { }; MODULE_DEVICE_TABLE(of, mtk8250_of_match); +static const struct acpi_device_id mtk8250_acpi_match[] = { + { "MTKI0511" }, + {} +}; +MODULE_DEVICE_TABLE(acpi, mtk8250_acpi_match); + static struct platform_driver mtk8250_platform_driver = { .driver = { .name = "mt6577-uart", .pm = &mtk8250_pm_ops, .of_match_table = mtk8250_of_match, + .acpi_match_table = mtk8250_acpi_match, }, .probe = mtk8250_probe, .remove = mtk8250_remove, From 0cce0fdad57927bdcd23eafd394ae8603b1d74c8 Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Tue, 11 Feb 2025 11:49:29 -0800 Subject: [PATCH 037/464] NVIDIA: SAUCE: Adds MT7925 BT devices BugLink: https://bugs.launchpad.net/bugs/2096882 Acked-by: Brad Figg Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (backported from commit a99eb0fc56550d106e276c1847db19996af18336 noble:linux-nvidia-6.11) [jacobmartin: Drop addition of 13d3:3604 already added by upstream commit f9685f315fd ("Bluetooth: btusb: Add MediaTek MT7925-B22M support ID 0x13d3:0x3604"). Drop driver_info flag "BTUSB_VALID_LE_STATES" as it was inverted by upstream commit 0fec656d08a ("Bluetooth: btusb: Invert LE State flag to set invalid rather then valid")] Signed-off-by: Jacob Martin (backported from commit a1d77cd8297c2bff09fd739a1ac3c0ed95e9e2d4 noble:linux-nvidia-6.14) [maskedarray: adjusted context] Signed-off-by: Abdur Rahman (cherry picked from commit f79eaa905f72fe2eff8774d787c1a84946c7a7fe noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/bluetooth/btusb.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c index 9c89a82046a7d..40dfce72536a1 100644 --- a/drivers/bluetooth/btusb.c +++ b/drivers/bluetooth/btusb.c @@ -754,6 +754,8 @@ static const struct usb_device_id quirks_table[] = { BTUSB_WIDEBAND_SPEECH }, { USB_DEVICE(0x13d3, 0x3608), .driver_info = BTUSB_MEDIATEK | BTUSB_WIDEBAND_SPEECH }, + { USB_DEVICE(0x13d3, 0x3609), .driver_info = BTUSB_MEDIATEK | + BTUSB_WIDEBAND_SPEECH }, { USB_DEVICE(0x13d3, 0x3613), .driver_info = BTUSB_MEDIATEK | BTUSB_WIDEBAND_SPEECH }, { USB_DEVICE(0x13d3, 0x3627), .driver_info = BTUSB_MEDIATEK | From f88470d92a8d0a0dcd4fcc2e539539b50b56c069 Mon Sep 17 00:00:00 2001 From: Us Chien Date: Sat, 12 Apr 2025 15:25:35 +0800 Subject: [PATCH 038/464] NVIDIA: SAUCE: MEDIATEK: usb: host: xhci-plat: support usb3 bulks stream low power BugLink: https://bugs.launchpad.net/bugs/2107509 Add a quirk to avoid U1 and U2 low power state operations during bulk stream transfers. Change-Id: Iaff484625eca6708713d0c2acaeddfc1103ac7d2 Signed-off-by: Us Chien Signed-off-by: Yenchia Chen Signed-off-by: Terje Bergstrom Acked-by: Brad Figg Acked-by: Matt Ochs Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 07399e87163635f78a35627866097a5cf6b6494a noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (backported from commit e521e80f789101cbf5041ee8d5606b9262b8250f) [maskedarray: changed the XHCI_NVIDIA_MT8901_HOST quirk bit value to 51] Signed-off-by: Abdur Rahman (cherry picked from commit 08ca4af323903e5341d555fc8c947491263eec85 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/usb/host/xhci-plat.c | 3 +++ drivers/usb/host/xhci.c | 5 +++++ drivers/usb/host/xhci.h | 1 + 3 files changed, 9 insertions(+) diff --git a/drivers/usb/host/xhci-plat.c b/drivers/usb/host/xhci-plat.c index 074d9c731639f..771836ab20ef0 100644 --- a/drivers/usb/host/xhci-plat.c +++ b/drivers/usb/host/xhci-plat.c @@ -277,6 +277,9 @@ int xhci_plat_probe(struct platform_device *pdev, struct device *sysdev, const s if (device_property_read_bool(tmpdev, "xhci-skip-phy-init-quirk")) xhci->quirks |= XHCI_SKIP_PHY_INIT; + if (device_property_read_bool(tmpdev, "xhci-nvidia-mediatek-host")) + xhci->quirks |= XHCI_NVIDIA_MT8901_HOST; + device_property_read_u32(tmpdev, "imod-interval-ns", &xhci->imod_interval); device_property_read_u16(tmpdev, "num-hc-interrupters", diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c index 8d8f0865fc121..35bd19073b685 100644 --- a/drivers/usb/host/xhci.c +++ b/drivers/usb/host/xhci.c @@ -3731,6 +3731,11 @@ static int xhci_alloc_streams(struct usb_hcd *hcd, struct usb_device *udev, if (ret < 0) goto cleanup; + if (xhci->quirks & XHCI_NVIDIA_MT8901_HOST) { + xhci_hub_control(hcd, SetPortFeature, USB_PORT_FEAT_U1_TIMEOUT, 0, NULL, 0); + xhci_hub_control(hcd, SetPortFeature, USB_PORT_FEAT_U2_TIMEOUT, 0, NULL, 0); + } + spin_lock_irqsave(&xhci->lock, flags); for (i = 0; i < num_eps; i++) { ep_index = xhci_get_endpoint_index(&eps[i]->desc); diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h index 2b0796f6d00ea..7421a806a6eae 100644 --- a/drivers/usb/host/xhci.h +++ b/drivers/usb/host/xhci.h @@ -1644,6 +1644,7 @@ struct xhci_hcd { #define XHCI_CDNS_SCTX_QUIRK BIT_ULL(48) #define XHCI_ETRON_HOST BIT_ULL(49) #define XHCI_LIMIT_ENDPOINT_INTERVAL_9 BIT_ULL(50) +#define XHCI_NVIDIA_MT8901_HOST BIT_ULL(51) unsigned int num_active_eps; unsigned int limit_active_eps; From 7cd0924dd4ff7edf42335e9a1fb06e69035ad258 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Mon, 28 Apr 2025 15:41:03 +0000 Subject: [PATCH 039/464] NVIDIA: SAUCE: r8127: Add Realtek r8127 ethernet driver BugLink: https://bugs.launchpad.net/bugs/2109730 Realtek R8127 driver can be downloaded from https://www.realtek.com/Download/List?cate_id=584 Where it is maintained as out of tree module. This patch adds the extracted content of r8127-11.014.00.tar.bz2 in the folder drivers/net/ethernet/realtek/r8127. 4bd62fc87de32760fb1f3b9cd3ec14e933035623 r8127-11.014.00.tar.bz2 All the clean-up, makefile and Kconfig related changes will be done in the subsequent commits. The source code contains a GPL2 compatible license. All the license information and Realtek copyright notice will be maintained in each file and newly added files. Signed-off-by: Abhishek Sahu Acked-by: Matt Ochs Acked-by: Carol L Soto Acked-by: Ian May Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Ian May (cherry picked from commit 7faf7ac3fffa292c696fa762de3863ca1f969f59 noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit e45f1b764ad7 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 24068b28f3b4b0ac746d520ffeb8864a844ef72f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/Makefile | 59 + drivers/net/ethernet/realtek/r8127/README | 147 + drivers/net/ethernet/realtek/r8127/autorun.sh | 101 + .../net/ethernet/realtek/r8127/src/Makefile | 209 + .../realtek/r8127/src/Makefile_linux24x | 75 + .../net/ethernet/realtek/r8127/src/r8127.h | 3068 +++ .../ethernet/realtek/r8127/src/r8127_dash.h | 261 + .../realtek/r8127/src/r8127_firmware.c | 264 + .../realtek/r8127/src/r8127_firmware.h | 68 + .../net/ethernet/realtek/r8127/src/r8127_n.c | 17824 ++++++++++++++++ .../ethernet/realtek/r8127/src/r8127_ptp.c | 944 + .../ethernet/realtek/r8127/src/r8127_ptp.h | 202 + .../realtek/r8127/src/r8127_realwow.h | 118 + .../ethernet/realtek/r8127/src/r8127_rss.c | 583 + .../ethernet/realtek/r8127/src/r8127_rss.h | 76 + .../ethernet/realtek/r8127/src/rtl_eeprom.c | 285 + .../ethernet/realtek/r8127/src/rtl_eeprom.h | 58 + .../net/ethernet/realtek/r8127/src/rtltool.c | 270 + .../net/ethernet/realtek/r8127/src/rtltool.h | 86 + 19 files changed, 24698 insertions(+) create mode 100755 drivers/net/ethernet/realtek/r8127/Makefile create mode 100755 drivers/net/ethernet/realtek/r8127/README create mode 100755 drivers/net/ethernet/realtek/r8127/autorun.sh create mode 100755 drivers/net/ethernet/realtek/r8127/src/Makefile create mode 100755 drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_dash.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_n.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_rss.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/r8127_rss.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h create mode 100755 drivers/net/ethernet/realtek/r8127/src/rtltool.c create mode 100755 drivers/net/ethernet/realtek/r8127/src/rtltool.h diff --git a/drivers/net/ethernet/realtek/r8127/Makefile b/drivers/net/ethernet/realtek/r8127/Makefile new file mode 100755 index 0000000000000..39e846ad3fc9f --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/Makefile @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: GPL-2.0-only +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ + +################################################################################ +# This product is covered by one or more of the following patents: +# US6,570,884, US6,115,776, and US6,327,625. +################################################################################ + +KFLAG := 2$(shell uname -r | sed -ne 's/^2\.[4]\..*/4/p')x + +all: clean modules install + +modules: +ifeq ($(KFLAG),24x) + $(MAKE) -C src/ -f Makefile_linux24x modules +else + $(MAKE) -C src/ modules +endif + +clean: +ifeq ($(KFLAG),24x) + $(MAKE) -C src/ -f Makefile_linux24x clean +else + $(MAKE) -C src/ clean +endif + +install: +ifeq ($(KFLAG),24x) + $(MAKE) -C src/ -f Makefile_linux24x install +else + $(MAKE) -C src/ install +endif + + + diff --git a/drivers/net/ethernet/realtek/r8127/README b/drivers/net/ethernet/realtek/r8127/README new file mode 100755 index 0000000000000..a2d451d938cab --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/README @@ -0,0 +1,147 @@ + + + This is the Linux device driver released for Realtek 5 Gigabit Ethernet controllers with PCI-Express interface. + + + + - Kernel source tree (supported Linux kernel 2.6.x and 2.4.x) + - For linux kernel 2.4.x, this driver supports 2.4.20 and latter. + - Compiler/binutils for kernel compilation + + + Unpack the tarball : + # tar vjxf r8127-11.aaa.bb.tar.bz2 + + Change to the directory: + # cd r8127-11.aaa.bb + + If you are running the target kernel, then you should be able to do : + + # ./autorun.sh (as root or with sudo) + + You can check whether the driver is loaded by using following commands. + + # lsmod | grep r8127 + # ifconfig -a + + If there is a device name, ethX, shown on the monitor, the linux + driver is loaded. Then, you can use the following command to activate + the ethX. + + # ifconfig ethX up + + ,where X=0,1,2,... + + + 1. Set manually + a. Set the IP address of your machine. + + # ifconfig ethX "the IP address of your machine" + + b. Set the IP address of DNS. + + Insert the following configuration in /etc/resolv.conf. + + nameserver "the IP address of DNS" + + c. Set the IP address of gateway. + + # route add default gw "the IP address of gateway" + + 2. Set by doing configurations in /etc/sysconfig/network-scripts + /ifcfg-ethX for Redhat and Fedora, or /etc/sysconfig/network + /ifcfg-ethX for SuSE. There are two examples to set network + configurations. + + a. Fixed IP address: + DEVICE=eth0 + BOOTPROTO=static + ONBOOT=yes + TYPE=ethernet + NETMASK=255.255.255.0 + IPADDR=192.168.1.1 + GATEWAY=192.168.1.254 + BROADCAST=192.168.1.255 + + b. DHCP: + DEVICE=eth0 + BOOTPROTO=dhcp + ONBOOT=yes + + + There are two ways to modify the MAC address of the NIC. + 1. Use ifconfig: + + # ifconfig ethX hw ether YY:YY:YY:YY:YY:YY + + ,where X is the device number assigned by Linux kernel, and + YY:YY:YY:YY:YY:YY is the MAC address assigned by the user. + + 2. Use ip: + + # ip link set ethX address YY:YY:YY:YY:YY:YY + + ,where X is the device number assigned by Linux kernel, and + YY:YY:YY:YY:YY:YY is the MAC address assigned by the user. + + + + 1. Force the link status when insert the driver. + + If the user is in the path ~/r8127, the link status can be forced + to one of the 5 modes as following command. + + # insmod ./src/r8127.ko speed=SPEED_MODE duplex=DUPLEX_MODE autoneg=NWAY_OPTION + + ,where + SPEED_MODE = 1000 for 1000Mbps + = 100 for 100Mbps + = 10 for 10Mbps + DUPLEX_MODE = 0 for half-duplex + = 1 for full-duplex + NWAY_OPTION = 0 for auto-negotiation off (true force) + = 1 for auto-negotiation on (nway force) + For example: + + # insmod ./src/r8127.ko speed=100 duplex=0 autoneg=1 + + will force PHY to operate in 100Mpbs Half-duplex(nway force). + + 2. Force the link status by using ethtool. + a. Insert the driver first. + b. Make sure that ethtool exists in /sbin. + c. Force the link status as the following command. + + 2.5G before kernel v4.10 + # ethtool -s eth0 autoneg on advertise 0x802f + + 2.5G for kernel v4.10 and later + # ethtool -s eth0 autoneg on advertise 0x80000000002f + + 5G for kernel v4.10 and later (Couldn't be supported before kernel v4.10) + # ethtool -s eth0 autoneg on advertise 0x180000000002f + + # ethtool -s eth0 autoneg on advertise 0x1000 (10G) + # ethtool -s eth0 autoneg on advertise 0x002f (1G) + # ethtool -s eth0 autoneg on advertise 0x000f (100M full) + # ethtool -s eth0 autoneg on advertise 0x0003 (10M full) + + + Transmitting Jumbo Frames, whose packet size is bigger than 1500 bytes, please change mtu by the following command. + + # ifconfig ethX mtu MTU + + , where X=0,1,2,..., and MTU is configured by user. + + RTL8127 supports Jumbo Frame size up to 9 kBytes. + + + Get/Set device EEE status + + Get EEE device status + # ethtool --show-eee enp1s0 + + Set EEE device status + # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x0008 (100M full) + # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x0020 (1G) + # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x8000 (2.5G) diff --git a/drivers/net/ethernet/realtek/r8127/autorun.sh b/drivers/net/ethernet/realtek/r8127/autorun.sh new file mode 100755 index 0000000000000..fd87bced11583 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/autorun.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# invoke insmod with all arguments we got +# and use a pathname, as insmod doesn't look in . by default + +TARGET_PATH=$(find /lib/modules/$(uname -r)/kernel/drivers/net/ethernet -name realtek -type d) +if [ "$TARGET_PATH" = "" ]; then + TARGET_PATH=$(find /lib/modules/$(uname -r)/kernel/drivers/net -name realtek -type d) +fi +if [ "$TARGET_PATH" = "" ]; then + TARGET_PATH=/lib/modules/$(uname -r)/kernel/drivers/net +fi +echo +echo "Check old driver and unload it." +check=`lsmod | grep r8169` +if [ "$check" != "" ]; then + echo "rmmod r8169" + /sbin/rmmod r8169 +fi + +check=`lsmod | grep r8127` +if [ "$check" != "" ]; then + echo "rmmod r8127" + /sbin/rmmod r8127 +fi + +echo "Build the module and install" +echo "-------------------------------" >> log.txt +date 1>>log.txt +make $@ all 1>>log.txt || exit 1 +module=`ls src/*.ko` +module=${module#src/} +module=${module%.ko} + +if [ "$module" = "" ]; then + echo "No driver exists!!!" + exit 1 +elif [ "$module" != "r8169" ]; then + if test -e $TARGET_PATH/r8169.ko ; then + echo "Backup r8169.ko" + if test -e $TARGET_PATH/r8169.bak ; then + i=0 + while test -e $TARGET_PATH/r8169.bak$i + do + i=$(($i+1)) + done + echo "rename r8169.ko to r8169.bak$i" + mv $TARGET_PATH/r8169.ko $TARGET_PATH/r8169.bak$i + else + echo "rename r8169.ko to r8169.bak" + mv $TARGET_PATH/r8169.ko $TARGET_PATH/r8169.bak + fi + fi + if test -e $TARGET_PATH/r8169.ko.zst ; then + echo "Backup r8169.ko.zst" + if test -e $TARGET_PATH/r8169.zst.bak ; then + i=0 + while test -e $TARGET_PATH/r8169.zst.bak$i + do + i=$(($i+1)) + done + echo "rename r8169.ko.zst to r8169.zst.bak$i" + mv $TARGET_PATH/r8169.ko.zst $TARGET_PATH/r8169.zst.bak$i + else + echo "rename r8169.ko.zst to r8169.zst.bak" + mv $TARGET_PATH/r8169.ko.zst $TARGET_PATH/r8169.zst.bak + fi + fi +fi + +echo "DEPMOD $(uname -r)" +depmod `uname -r` +echo "load module $module" +modprobe $module + +is_update_initramfs=n +distrib_list="ubuntu debian" + +if [ -r /etc/debian_version ]; then + is_update_initramfs=y +elif [ -r /etc/lsb-release ]; then + for distrib in $distrib_list + do + /bin/grep -i "$distrib" /etc/lsb-release 2>&1 /dev/null && \ + is_update_initramfs=y && break + done +fi + +if [ "$is_update_initramfs" = "y" ]; then + if which update-initramfs >/dev/null ; then + echo "Updating initramfs. Please wait." + update-initramfs -u -k $(uname -r) + else + echo "update-initramfs: command not found" + exit 1 + fi +fi + +echo "Completed." +exit 0 diff --git a/drivers/net/ethernet/realtek/r8127/src/Makefile b/drivers/net/ethernet/realtek/r8127/src/Makefile new file mode 100755 index 0000000000000..d270904691bf3 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/Makefile @@ -0,0 +1,209 @@ +# SPDX-License-Identifier: GPL-2.0-only +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ + +################################################################################ +# This product is covered by one or more of the following patents: +# US6,570,884, US6,115,776, and US6,327,625. +################################################################################ + +CONFIG_SOC_LAN = y +ENABLE_REALWOW_SUPPORT = n +ENABLE_DASH_SUPPORT = n +ENABLE_DASH_PRINTER_SUPPORT = n +CONFIG_DOWN_SPEED_100 = n +CONFIG_ASPM = y +ENABLE_S5WOL = y +ENABLE_S5_KEEP_CURR_MAC = n +ENABLE_EEE = y +ENABLE_S0_MAGIC_PACKET = n +ENABLE_TX_NO_CLOSE = y +ENABLE_MULTIPLE_TX_QUEUE = n +ENABLE_PTP_SUPPORT = n +ENABLE_RSS_SUPPORT = n +ENABLE_LIB_SUPPORT = n +ENABLE_USE_FIRMWARE_FILE = n +DISABLE_WOL_SUPPORT = n +DISABLE_MULTI_MSIX_VECTOR = n +ENABLE_DOUBLE_VLAN = n +ENABLE_PAGE_REUSE = n +ENABLE_RX_PACKET_FRAGMENT = n +ENABLE_GIGA_LITE = y + +ifneq ($(KERNELRELEASE),) + obj-m := r8127.o + r8127-objs := r8127_n.o rtl_eeprom.o rtltool.o + ifeq ($(CONFIG_SOC_LAN), y) + EXTRA_CFLAGS += -DCONFIG_SOC_LAN + endif + ifeq ($(ENABLE_REALWOW_SUPPORT), y) + r8127-objs += r8127_realwow.o + EXTRA_CFLAGS += -DENABLE_REALWOW_SUPPORT + endif + ifeq ($(ENABLE_DASH_SUPPORT), y) + r8127-objs += r8127_dash.o + EXTRA_CFLAGS += -DENABLE_DASH_SUPPORT + endif + ifeq ($(ENABLE_DASH_PRINTER_SUPPORT), y) + r8127-objs += r8127_dash.o + EXTRA_CFLAGS += -DENABLE_DASH_SUPPORT -DENABLE_DASH_PRINTER_SUPPORT + endif + EXTRA_CFLAGS += -DCONFIG_R8127_NAPI + EXTRA_CFLAGS += -DCONFIG_R8127_VLAN + ifeq ($(CONFIG_DOWN_SPEED_100), y) + EXTRA_CFLAGS += -DCONFIG_DOWN_SPEED_100 + endif + ifeq ($(CONFIG_ASPM), y) + EXTRA_CFLAGS += -DCONFIG_ASPM + endif + ifeq ($(ENABLE_S5WOL), y) + EXTRA_CFLAGS += -DENABLE_S5WOL + endif + ifeq ($(ENABLE_S5_KEEP_CURR_MAC), y) + EXTRA_CFLAGS += -DENABLE_S5_KEEP_CURR_MAC + endif + ifeq ($(ENABLE_EEE), y) + EXTRA_CFLAGS += -DENABLE_EEE + endif + ifeq ($(ENABLE_S0_MAGIC_PACKET), y) + EXTRA_CFLAGS += -DENABLE_S0_MAGIC_PACKET + endif + ifeq ($(ENABLE_TX_NO_CLOSE), y) + EXTRA_CFLAGS += -DENABLE_TX_NO_CLOSE + endif + ifeq ($(ENABLE_MULTIPLE_TX_QUEUE), y) + EXTRA_CFLAGS += -DENABLE_MULTIPLE_TX_QUEUE + endif + ifeq ($(ENABLE_PTP_SUPPORT), y) + r8127-objs += r8127_ptp.o + EXTRA_CFLAGS += -DENABLE_PTP_SUPPORT + endif + ifeq ($(ENABLE_RSS_SUPPORT), y) + r8127-objs += r8127_rss.o + EXTRA_CFLAGS += -DENABLE_RSS_SUPPORT + endif + ifeq ($(ENABLE_LIB_SUPPORT), y) + r8127-objs += r8127_lib.o + EXTRA_CFLAGS += -DENABLE_LIB_SUPPORT + endif + ifeq ($(ENABLE_USE_FIRMWARE_FILE), y) + r8127-objs += r8127_firmware.o + EXTRA_CFLAGS += -DENABLE_USE_FIRMWARE_FILE + endif + ifeq ($(DISABLE_WOL_SUPPORT), y) + EXTRA_CFLAGS += -DDISABLE_WOL_SUPPORT + endif + ifeq ($(DISABLE_MULTI_MSIX_VECTOR), y) + EXTRA_CFLAGS += -DDISABLE_MULTI_MSIX_VECTOR + endif + ifeq ($(ENABLE_DOUBLE_VLAN), y) + EXTRA_CFLAGS += -DENABLE_DOUBLE_VLAN + endif + ifeq ($(ENABLE_PAGE_REUSE), y) + EXTRA_CFLAGS += -DENABLE_PAGE_REUSE + endif + ifeq ($(ENABLE_RX_PACKET_FRAGMENT), y) + EXTRA_CFLAGS += -DENABLE_RX_PACKET_FRAGMENT + endif + ifeq ($(ENABLE_GIGA_LITE), y) + EXTRA_CFLAGS += -DENABLE_GIGA_LITE + endif +else + BASEDIR := /lib/modules/$(shell uname -r) + KERNELDIR ?= $(BASEDIR)/build + PWD :=$(shell pwd) + DRIVERDIR := $(shell find $(BASEDIR)/kernel/drivers/net/ethernet -name realtek -type d) + ifeq ($(DRIVERDIR),) + DRIVERDIR := $(shell find $(BASEDIR)/kernel/drivers/net -name realtek -type d) + endif + ifeq ($(DRIVERDIR),) + DRIVERDIR := $(BASEDIR)/kernel/drivers/net + endif + RTKDIR := $(subst $(BASEDIR)/,,$(DRIVERDIR)) + + KERNEL_GCC_VERSION := $(shell cat /proc/version | sed -n 's/.*gcc version \([[:digit:]]\.[[:digit:]]\.[[:digit:]]\).*/\1/p') + CCVERSION = $(shell $(CC) -dumpversion) + + KVER = $(shell uname -r) + KMAJ = $(shell echo $(KVER) | \ + sed -e 's/^\([0-9][0-9]*\)\.[0-9][0-9]*\.[0-9][0-9]*.*/\1/') + KMIN = $(shell echo $(KVER) | \ + sed -e 's/^[0-9][0-9]*\.\([0-9][0-9]*\)\.[0-9][0-9]*.*/\1/') + KREV = $(shell echo $(KVER) | \ + sed -e 's/^[0-9][0-9]*\.[0-9][0-9]*\.\([0-9][0-9]*\).*/\1/') + + kver_ge = $(shell \ + echo test | awk '{if($(KMAJ) < $(1)) {print 0} else { \ + if($(KMAJ) > $(1)) {print 1} else { \ + if($(KMIN) < $(2)) {print 0} else { \ + if($(KMIN) > $(2)) {print 1} else { \ + if($(KREV) < $(3)) {print 0} else { print 1 } \ + }}}}}' \ + ) + +.PHONY: all +all: print_vars clean modules install + +print_vars: + @echo + @echo "CC: " $(CC) + @echo "CCVERSION: " $(CCVERSION) + @echo "KERNEL_GCC_VERSION: " $(KERNEL_GCC_VERSION) + @echo "KVER: " $(KVER) + @echo "KMAJ: " $(KMAJ) + @echo "KMIN: " $(KMIN) + @echo "KREV: " $(KREV) + @echo "BASEDIR: " $(BASEDIR) + @echo "DRIVERDIR: " $(DRIVERDIR) + @echo "PWD: " $(PWD) + @echo "RTKDIR: " $(RTKDIR) + @echo + +.PHONY:modules +modules: +#ifeq ($(call kver_ge,5,0,0),1) + $(MAKE) -C $(KERNELDIR) M=$(PWD) modules +#else +# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) modules +#endif + +.PHONY:clean +clean: +#ifeq ($(call kver_ge,5,0,0),1) + $(MAKE) -C $(KERNELDIR) M=$(PWD) clean +#else +# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) clean +#endif + +.PHONY:install +install: +#ifeq ($(call kver_ge,5,0,0),1) + $(MAKE) -C $(KERNELDIR) M=$(PWD) INSTALL_MOD_DIR=$(RTKDIR) modules_install +#else +# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) INSTALL_MOD_DIR=$(RTKDIR) modules_install +#endif + +endif diff --git a/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x b/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x new file mode 100755 index 0000000000000..7cb3d91a85a64 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: GPL-2.0-only +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ + +################################################################################ +# This product is covered by one or more of the following patents: +# US6,570,884, US6,115,776, and US6,327,625. +################################################################################ + +CC := gcc +LD := ld +ARCH := $(shell uname -m | sed 's/i.86/i386/') +KSRC := /lib/modules/$(shell uname -r)/build +CONFIG_FILE := $(KSRC)/include/linux/autoconf.h +KMISC := /lib/modules/$(shell uname -r)/kernel/drivers/net/ + + +ifeq ($(ARCH),x86_64) + MODCFLAGS += -mcmodel=kernel -mno-red-zone +endif + +#standard flags for module builds +MODCFLAGS += -DLINUX -D__KERNEL__ -DMODULE -O2 -pipe -Wall +MODCFLAGS += -I$(KSRC)/include -I. +MODCFLAGS += -DMODVERSIONS -DEXPORT_SYMTAB -include $(KSRC)/include/linux/modversions.h +SOURCE := r8127_n.c rtl_eeprom.c rtltool.c +OBJS := $(SOURCE:.c=.o) + + +SMP := $(shell $(CC) $(MODCFLAGS) -E -dM $(CONFIG_FILE) | \ + grep CONFIG_SMP | awk '{print $$3}') + +ifneq ($(SMP),1) + SMP := 0 +endif + +ifeq ($(SMP),1) + MODCFLAGS += -D__SMP__ +endif + +modules: $(OBJS) + $(LD) -r $^ -o r8127.o + strip --strip-debug r8127.o + +%.o: %.c + $(CC) $(MODCFLAGS) -c $< -o $@ + +clean: + rm *.o -f + +install: + install -m 744 -c r8127.o $(KMISC) diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127.h b/drivers/net/ethernet/realtek/r8127/src/r8127.h new file mode 100755 index 0000000000000..fccb974bc08c4 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127.h @@ -0,0 +1,3068 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef __R8127_H +#define __R8127_H + +//#include +#include +#include +#include +#include "r8127_dash.h" +#include "r8127_realwow.h" +#ifdef ENABLE_PTP_SUPPORT +#include "r8127_ptp.h" +#endif +#include "r8127_rss.h" +#ifdef ENABLE_LIB_SUPPORT +#include "r8127_lib.h" +#endif + +#ifndef fallthrough +#define fallthrough +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +#define netif_xmit_stopped netif_tx_queue_stopped +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,7,0) +#ifndef MDIO_AN_EEE_ADV_100TX +#define MDIO_AN_EEE_ADV_100TX 0x0002 /* Advertise 100TX EEE cap */ +#endif +#ifndef MDIO_AN_EEE_ADV_1000T +#define MDIO_AN_EEE_ADV_1000T 0x0004 /* Advertise 1000T EEE cap */ +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) +#define MDIO_EEE_100TX MDIO_AN_EEE_ADV_100TX /* 100TX EEE cap */ +#define MDIO_EEE_1000T MDIO_AN_EEE_ADV_1000T /* 1000T EEE cap */ +#define MDIO_EEE_10GT 0x0008 /* 10GT EEE cap */ +#define MDIO_EEE_1000KX 0x0010 /* 1000KX EEE cap */ +#define MDIO_EEE_10GKX4 0x0020 /* 10G KX4 EEE cap */ +#define MDIO_EEE_10GKR 0x0040 /* 10G KR EEE cap */ +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) */ + +static inline u32 mmd_eee_adv_to_ethtool_adv_t(u16 eee_adv) +{ + u32 adv = 0; + + if (eee_adv & MDIO_EEE_100TX) + adv |= ADVERTISED_100baseT_Full; + if (eee_adv & MDIO_EEE_1000T) + adv |= ADVERTISED_1000baseT_Full; + if (eee_adv & MDIO_EEE_10GT) + adv |= ADVERTISED_10000baseT_Full; + if (eee_adv & MDIO_EEE_1000KX) + adv |= ADVERTISED_1000baseKX_Full; + if (eee_adv & MDIO_EEE_10GKX4) + adv |= ADVERTISED_10000baseKX4_Full; + if (eee_adv & MDIO_EEE_10GKR) + adv |= ADVERTISED_10000baseKR_Full; + + return adv; +} + +static inline u16 ethtool_adv_to_mmd_eee_adv_t(u32 adv) +{ + u16 reg = 0; + + if (adv & ADVERTISED_100baseT_Full) + reg |= MDIO_EEE_100TX; + if (adv & ADVERTISED_1000baseT_Full) + reg |= MDIO_EEE_1000T; + if (adv & ADVERTISED_10000baseT_Full) + reg |= MDIO_EEE_10GT; + if (adv & ADVERTISED_1000baseKX_Full) + reg |= MDIO_EEE_1000KX; + if (adv & ADVERTISED_10000baseKX4_Full) + reg |= MDIO_EEE_10GKX4; + if (adv & ADVERTISED_10000baseKR_Full) + reg |= MDIO_EEE_10GKR; + + return reg; +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,7,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,9,0) +static inline bool skb_transport_header_was_set(const struct sk_buff *skb) +{ + return skb->transport_header != ~0U; +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,9,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,20,0) +static inline void linkmode_set_bit(int nr, volatile unsigned long *addr) +{ + __set_bit(nr, addr); +} + +static inline void linkmode_clear_bit(int nr, volatile unsigned long *addr) +{ + __clear_bit(nr, addr); +} + +static inline int linkmode_test_bit(int nr, volatile unsigned long *addr) +{ + return test_bit(nr, addr); +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(4,20,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,0,0) +static inline void linkmode_mod_bit(int nr, volatile unsigned long *addr, + int set) +{ + if (set) + linkmode_set_bit(nr, addr); + else + linkmode_clear_bit(nr, addr); +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,0,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,3,0) +static inline +ssize_t strscpy(char *dest, const char *src, size_t count) +{ + long res = 0; + + if (count == 0) + return -E2BIG; + + while (count) { + char c; + + c = src[res]; + dest[res] = c; + if (!c) + return res; + res++; + count--; + } + + /* Hit buffer length without finding a NUL; force NUL-termination. */ + if (res) + dest[res-1] = '\0'; + + return -E2BIG; +} +#endif + +#if (LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0)) +static inline unsigned char *skb_checksum_start(const struct sk_buff *skb) +{ +#if (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22)) + return skb->head + skb->csum_start; +#else /* < 2.6.22 */ + return skb_transport_header(skb); +#endif +} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +static inline void netdev_tx_sent_queue(struct netdev_queue *dev_queue, + unsigned int bytes) +{} +static inline void netdev_tx_completed_queue(struct netdev_queue *dev_queue, + unsigned int pkts, + unsigned int bytes) +{} +static inline void netdev_tx_reset_queue(struct netdev_queue *q) {} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,8,0) +static inline void fsleep(unsigned long usecs) +{ + if (usecs <= 10) + udelay(usecs); + else if (usecs <= 20000) + usleep_range(usecs, 2 * usecs); + else + msleep(DIV_ROUND_UP(usecs, 1000)); +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,8,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,2,0) +#define netdev_xmit_more() (0) +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,8,0) +#define netif_testing_on(dev) +#define netif_testing_off(dev) +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(6,2,0) +#define netdev_sw_irq_coalesce_default_on(dev) +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(6,2,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,32) +typedef int netdev_tx_t; +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,12,0) +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,1,9) +static inline bool page_is_pfmemalloc(struct page *page) +{ + /* + * Page index cannot be this large so this must be + * a pfmemalloc page. + */ + return page->index == -1UL; +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(4,1,9) */ +static inline bool dev_page_is_reusable(struct page *page) +{ + return likely(page_to_nid(page) == numa_mem_id() && + !page_is_pfmemalloc(page)); +} +#endif + +/* +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,12,0)&& !defined(ENABLE_LIB_SUPPORT) +#define RTL_USE_NEW_INTR_API +#endif +*/ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,10,0) +#define dma_map_page_attrs(dev, page, offset, size, dir, attrs) \ + dma_map_page(dev, page, offset, size, dir) +#define dma_unmap_page_attrs(dev, page, size, dir, attrs) \ + dma_unmap_page(dev, page, size, dir) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,10,0) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) +#define page_ref_inc(page) atomic_inc(&page->_count) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,4,216) +#define page_ref_count(page) atomic_read(&page->_count) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,4,216) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) +#define skb_transport_offset(skb) (skb->h.raw - skb->data) +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,26) +#define device_set_wakeup_enable(dev, val) do {} while (0) +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) +static inline void ether_addr_copy(u8 *dst, const u8 *src) +{ + u16 *a = (u16 *)dst; + const u16 *b = (const u16 *)src; + + a[0] = b[0]; + a[1] = b[1]; + a[2] = b[2]; +} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,15,0) +#define IS_ERR_OR_NULL(ptr) (!ptr) +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,13,0) +#define reinit_completion(x) ((x)->done = 0) +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,39) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,32) +#define pm_runtime_mark_last_busy(x) +#define pm_runtime_put_autosuspend(x) pm_runtime_put(x) +#define pm_runtime_put_sync_autosuspend(x) pm_runtime_put_sync(x) + +static inline bool pm_runtime_suspended(struct device *dev) +{ + return dev->power.runtime_status == RPM_SUSPENDED + && !dev->power.disable_depth; +} + +static inline bool pm_runtime_active(struct device *dev) +{ + return dev->power.runtime_status == RPM_ACTIVE + || dev->power.disable_depth; +} +#endif +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,36) +#define queue_delayed_work(long_wq, work, delay) schedule_delayed_work(work, delay) +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,34) +#define netif_printk(priv, type, level, netdev, fmt, args...) \ + do { \ + if (netif_msg_##type(priv)) \ + printk(level "%s: " fmt,(netdev)->name , ##args); \ + } while (0) + +#define netif_emerg(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_EMERG, netdev, fmt, ##args) +#define netif_alert(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_ALERT, netdev, fmt, ##args) +#define netif_crit(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_CRIT, netdev, fmt, ##args) +#define netif_err(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_ERR, netdev, fmt, ##args) +#define netif_warn(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_WARNING, netdev, fmt, ##args) +#define netif_notice(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_NOTICE, netdev, fmt, ##args) +#define netif_info(priv, type, netdev, fmt, args...) \ + netif_printk(priv, type, KERN_INFO, (netdev), fmt, ##args) +#endif +#endif +#endif +#endif +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,15) +#define setup_timer(_timer, _function, _data) \ +do { \ + (_timer)->function = _function; \ + (_timer)->data = _data; \ + init_timer(_timer); \ +} while (0) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,15) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0) +#if defined(skb_vlan_tag_present) && !defined(vlan_tx_tag_present) +#define vlan_tx_tag_present skb_vlan_tag_present +#endif +#if defined(skb_vlan_tag_get) && !defined(vlan_tx_tag_get) +#define vlan_tx_tag_get skb_vlan_tag_get +#endif +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0) + +#define RTL_ALLOC_SKB_INTR(napi, length) dev_alloc_skb(length) +#define R8127_USE_NAPI_ALLOC_SKB 0 +#ifdef CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,19,0) +#undef RTL_ALLOC_SKB_INTR +#define RTL_ALLOC_SKB_INTR(napi, length) napi_alloc_skb(napi, length) +#undef R8127_USE_NAPI_ALLOC_SKB +#define R8127_USE_NAPI_ALLOC_SKB 1 +#endif +#endif + +#define RTL_BUILD_SKB_INTR(data, frag_size) build_skb(data, frag_size) +#ifdef CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,12,0) +#undef RTL_BUILD_SKB_INTR +#define RTL_BUILD_SKB_INTR(data, frag_size) napi_build_skb(data, frag_size) +#endif +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) +#define eth_random_addr(addr) random_ether_addr(addr) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) +#define netdev_features_t u32 +#endif +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,5,0) +#define NETIF_F_ALL_CSUM NETIF_F_CSUM_MASK +#else +#ifndef NETIF_F_ALL_CSUM +#define NETIF_F_ALL_CSUM NETIF_F_CSUM_MASK +#endif +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,4,37) +#define ENABLE_R8127_PROCFS +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,11,0) +#define ENABLE_R8127_SYSFS +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) +#define NETIF_F_HW_VLAN_RX NETIF_F_HW_VLAN_CTAG_RX +#define NETIF_F_HW_VLAN_TX NETIF_F_HW_VLAN_CTAG_TX +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,8,0) +#define __devinit +#define __devexit +#define __devexit_p(func) func +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) +#define CHECKSUM_PARTIAL CHECKSUM_HW +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) +#define irqreturn_t void +#define IRQ_HANDLED 1 +#define IRQ_NONE 0 +#define IRQ_RETVAL(x) +#endif + +#ifndef NETIF_F_RXALL +#define NETIF_F_RXALL 0 +#endif + +#ifndef NETIF_F_RXFCS +#define NETIF_F_RXFCS 0 +#endif + +#if !defined(HAVE_FREE_NETDEV) && (LINUX_VERSION_CODE < KERNEL_VERSION(3,1,0)) +#define free_netdev(x) kfree(x) +#endif + +#ifndef SET_NETDEV_DEV +#define SET_NETDEV_DEV(net, pdev) +#endif + +#ifndef SET_MODULE_OWNER +#define SET_MODULE_OWNER(dev) +#endif + +#ifndef SA_SHIRQ +#define SA_SHIRQ IRQF_SHARED +#endif + +#ifndef NETIF_F_GSO +#define gso_size tso_size +#define gso_segs tso_segs +#endif + +#ifndef PCI_VENDOR_ID_DLINK +#define PCI_VENDOR_ID_DLINK 0x1186 +#endif + +#ifndef dma_mapping_error +#define dma_mapping_error(a,b) 0 +#endif + +#ifndef netif_err +#define netif_err(a,b,c,d) +#endif + +#ifndef AUTONEG_DISABLE +#define AUTONEG_DISABLE 0x00 +#endif + +#ifndef AUTONEG_ENABLE +#define AUTONEG_ENABLE 0x01 +#endif + +#ifndef BMCR_SPEED1000 +#define BMCR_SPEED1000 0x0040 +#endif + +#ifndef BMCR_SPEED100 +#define BMCR_SPEED100 0x2000 +#endif + +#ifndef BMCR_SPEED10 +#define BMCR_SPEED10 0x0000 +#endif + +#ifndef SPEED_UNKNOWN +#define SPEED_UNKNOWN -1 +#endif + +#ifndef DUPLEX_UNKNOWN +#define DUPLEX_UNKNOWN 0xff +#endif + +#ifndef SUPPORTED_Pause +#define SUPPORTED_Pause (1 << 13) +#endif + +#ifndef SUPPORTED_Asym_Pause +#define SUPPORTED_Asym_Pause (1 << 14) +#endif + +#ifndef MDIO_EEE_100TX +#define MDIO_EEE_100TX 0x0002 +#endif + +#ifndef MDIO_EEE_1000T +#define MDIO_EEE_1000T 0x0004 +#endif + +#ifndef MDIO_EEE_2_5GT +#define MDIO_EEE_2_5GT 0x0001 +#endif + +#ifndef MDIO_EEE_5GT +#define MDIO_EEE_5GT 0x0002 +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0) +#define ethtool_keee ethtool_eee +#define rtl8127_ethtool_adv_to_mmd_eee_adv_cap1_t ethtool_adv_to_mmd_eee_adv_t +static inline u32 rtl8127_ethtool_adv_to_mmd_eee_adv_cap2_t(u32 adv) +{ + u32 result = 0; + + if (adv & SUPPORTED_2500baseX_Full) + result |= MDIO_EEE_2_5GT; + + return result; +} +#else +#define rtl8127_ethtool_adv_to_mmd_eee_adv_cap1_t linkmode_to_mii_eee_cap1_t +#define rtl8127_ethtool_adv_to_mmd_eee_adv_cap2_t linkmode_to_mii_eee_cap2_t +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0) */ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) +#ifdef CONFIG_NET_POLL_CONTROLLER +#define RTL_NET_POLL_CONTROLLER dev->poll_controller=rtl8127_netpoll +#else +#define RTL_NET_POLL_CONTROLLER +#endif + +#ifdef CONFIG_R8127_VLAN +#define RTL_SET_VLAN dev->vlan_rx_register=rtl8127_vlan_rx_register +#else +#define RTL_SET_VLAN +#endif + +#define RTL_NET_DEVICE_OPS(ops) dev->open=rtl8127_open; \ + dev->hard_start_xmit=rtl8127_start_xmit; \ + dev->get_stats=rtl8127_get_stats; \ + dev->stop=rtl8127_close; \ + dev->tx_timeout=rtl8127_tx_timeout; \ + dev->set_multicast_list=rtl8127_set_rx_mode; \ + dev->change_mtu=rtl8127_change_mtu; \ + dev->set_mac_address=rtl8127_set_mac_address; \ + dev->do_ioctl=rtl8127_do_ioctl; \ + RTL_NET_POLL_CONTROLLER; \ + RTL_SET_VLAN; +#else +#define RTL_NET_DEVICE_OPS(ops) dev->netdev_ops=&ops +#endif + +#ifndef FALSE +#define FALSE 0 +#endif + +#ifndef TRUE +#define TRUE 1 +#endif + +#ifndef false +#define false 0 +#endif + +#ifndef true +#define true 1 +#endif + +//Hardware will continue interrupt 10 times after interrupt finished. +#define RTK_KEEP_INTERRUPT_COUNT (10) + +//the low 32 bit address of receive buffer must be 8-byte alignment. +#ifndef NET_IP_ALIGN +#define NET_IP_ALIGN 2 +#endif +#define R8127_RX_ALIGN NET_IP_ALIGN + +#ifdef CONFIG_R8127_NAPI +#define NAPI_SUFFIX "-NAPI" +#else +#define NAPI_SUFFIX "" +#endif +#if defined(ENABLE_DASH_PRINTER_SUPPORT) +#define DASH_SUFFIX "-PRINTER" +#elif defined(ENABLE_DASH_SUPPORT) +#define DASH_SUFFIX "-DASH" +#else +#define DASH_SUFFIX "" +#endif + +#if defined(ENABLE_REALWOW_SUPPORT) +#define REALWOW_SUFFIX "-REALWOW" +#else +#define REALWOW_SUFFIX "" +#endif + +#if defined(ENABLE_PTP_SUPPORT) +#define PTP_SUFFIX "-PTP" +#else +#define PTP_SUFFIX "" +#endif + +#if defined(ENABLE_RSS_SUPPORT) +#define RSS_SUFFIX "-RSS" +#else +#define RSS_SUFFIX "" +#endif + +#define RTL8127_VERSION "11.014.00" NAPI_SUFFIX DASH_SUFFIX REALWOW_SUFFIX PTP_SUFFIX RSS_SUFFIX +#define MODULENAME "r8127" +#define PFX MODULENAME ": " + +#define GPL_CLAIM "\ +r8127 Copyright (C) 2025 Realtek NIC software team \n \ +This program comes with ABSOLUTELY NO WARRANTY; for details, please see . \n \ +This is free software, and you are welcome to redistribute it under certain conditions; see . \n" + +#ifdef RTL8127_DEBUG +#define assert(expr) \ + if(!(expr)) { \ + printk("Assertion failed! %s,%s,%s,line=%d\n", \ + #expr,__FILE__,__FUNCTION__,__LINE__); \ + } +#define dprintk(fmt, args...) do { printk(PFX fmt, ## args); } while (0) +#else +#define assert(expr) do {} while (0) +#define dprintk(fmt, args...) do {} while (0) +#endif /* RTL8127_DEBUG */ + +#define R8127_MSG_DEFAULT \ + (NETIF_MSG_DRV | NETIF_MSG_PROBE | NETIF_MSG_IFUP | NETIF_MSG_IFDOWN) + +#ifdef CONFIG_R8127_NAPI +#define rtl8127_rx_hwaccel_skb vlan_hwaccel_receive_skb +#define rtl8127_rx_quota(count, quota) min(count, quota) +#else +#define rtl8127_rx_hwaccel_skb vlan_hwaccel_rx +#define rtl8127_rx_quota(count, quota) count +#endif + +/* MAC address length */ +#ifndef MAC_ADDR_LEN +#define MAC_ADDR_LEN 6 +#endif + +#ifndef MAC_PROTOCOL_LEN +#define MAC_PROTOCOL_LEN 2 +#endif + +#ifndef ETH_FCS_LEN +#define ETH_FCS_LEN 4 +#endif + +#ifndef NETIF_F_TSO6 +#define NETIF_F_TSO6 0 +#endif + +#define Reserved2_data 7 +#define RX_DMA_BURST_unlimited 7 /* Maximum PCI burst, '7' is unlimited */ +#define RX_DMA_BURST_512 5 +#define RX_DMA_BURST_256 4 +#define TX_DMA_BURST_unlimited 7 +#define TX_DMA_BURST_1024 6 +#define TX_DMA_BURST_512 5 +#define TX_DMA_BURST_256 4 +#define TX_DMA_BURST_128 3 +#define TX_DMA_BURST_64 2 +#define TX_DMA_BURST_32 1 +#define TX_DMA_BURST_16 0 +#define Reserved1_data 0x3F +#define RxPacketMaxSize 0x3FE8 /* 16K - 1 - ETH_HLEN - VLAN - CRC... */ +#define Jumbo_Frame_1k ETH_DATA_LEN +#define Jumbo_Frame_2k (2*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_3k (3*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_4k (4*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_5k (5*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_6k (6*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_7k (7*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_8k (8*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define Jumbo_Frame_9k (9*1024 - ETH_HLEN - VLAN_HLEN - ETH_FCS_LEN) +#define InterFrameGap 0x03 /* 3 means InterFrameGap = the shortest one */ +#define RxEarly_off_V1 (0x07 << 11) +#define RxEarly_off_V2 (1 << 11) +#define Rx_Single_fetch_V2 (1 << 14) +#define Rx_Close_Multiple (1 << 21) +#define Rx_Fetch_Number_8 (1 << 30) + +#define R8127_REGS_SIZE (256) +#define R8127_MAC_REGS_SIZE (256) +#define R8127_PHY_REGS_SIZE (16*2) +#define R8127_EPHY_REGS_SIZE (31*2) +#define R8127_ERI_REGS_SIZE (0x100) +#define R8127_REGS_DUMP_SIZE (0x400) +#define R8127_PCI_REGS_SIZE (0x100) +#define R8127_NAPI_WEIGHT 64 + +#define R8127_MAX_MSIX_VEC_8125A 4 +#define R8127_MAX_MSIX_VEC_8125B 32 +#define R8127_MAX_MSIX_VEC_8125D 32 +#define R8127_MIN_MSIX_VEC_8125B 22 +#define R8127_MIN_MSIX_VEC_8125BP 31 +#define R8127_MIN_MSIX_VEC_8125D 20 +#define R8127_MIN_MSIX_VEC_8127 30 +#define R8127_MAX_MSIX_VEC 32 +#define R8127_MAX_RX_QUEUES_VEC_V3 (16) +#define R8127_MAX_RX_QUEUES_VEC_V4 (8) + +#define RTL8127_TX_TIMEOUT (6 * HZ) +#define RTL8127_LINK_TIMEOUT (1 * HZ) +#define RTL8127_ESD_TIMEOUT (2 * HZ) + +#define rtl8127_rx_page_size(order) (PAGE_SIZE << order) + +#define MAX_NUM_TX_DESC 1024 /* Maximum number of Tx descriptor registers */ +#define MAX_NUM_RX_DESC 1024 /* Maximum number of Rx descriptor registers */ + +#define MIN_NUM_TX_DESC 256 /* Minimum number of Tx descriptor registers */ +#define MIN_NUM_RX_DESC 256 /* Minimum number of Rx descriptor registers */ + +#define NUM_TX_DESC MAX_NUM_TX_DESC /* Number of Tx descriptor registers */ +#define NUM_RX_DESC MAX_NUM_RX_DESC /* Number of Rx descriptor registers */ + +#ifdef ENABLE_DOUBLE_VLAN +#define RX_BUF_SIZE 0x05F6 /* 0x05F6(1526) = 1514 + 8(double vlan) + 4(crc) bytes */ +#define RT_VALN_HLEN 8 /* 8(double vlan) bytes */ +#else +#define RX_BUF_SIZE 0x05F2 /* 0x05F2(1522) = 1514 + 4(single vlan) + 4(crc) bytes */ +#define RT_VALN_HLEN 4 /* 4(single vlan) bytes */ +#endif + +#define R8127_MAX_TX_QUEUES (2) +#define R8127_MAX_RX_QUEUES_V2 (4) +#define R8127_MAX_RX_QUEUES_V3 (16) +#define R8127_MAX_RX_QUEUES R8127_MAX_RX_QUEUES_V3 +#define R8127_MAX_QUEUES R8127_MAX_RX_QUEUES + +#define OCP_STD_PHY_BASE 0xa400 + +//Channel Wait Count +#define R8127_CHANNEL_WAIT_COUNT (20000) +#define R8127_CHANNEL_WAIT_TIME (1) // 1us +#define R8127_CHANNEL_EXIT_DELAY_TIME (20) //20us + +#ifdef ENABLE_LIB_SUPPORT +#define R8127_MULTI_RX_Q(tp) 0 +#else +#define R8127_MULTI_RX_Q(tp) (tp->num_rx_rings > 1) +#endif + +#define NODE_ADDRESS_SIZE 6 + +#define SHORT_PACKET_PADDING_BUF_SIZE 256 + +#define RTK_MAGIC_DEBUG_VALUE 0x0badbeef + +/* write/read MMIO register */ +#define RTL_W8(tp, reg, val8) writeb((val8), tp->mmio_addr + (reg)) +#define RTL_W16(tp, reg, val16) writew((val16), tp->mmio_addr + (reg)) +#define RTL_W32(tp, reg, val32) writel((val32), tp->mmio_addr + (reg)) +#define RTL_R8(tp, reg) readb(tp->mmio_addr + (reg)) +#define RTL_R16(tp, reg) readw(tp->mmio_addr + (reg)) +#define RTL_R32(tp, reg) ((unsigned long) readl(tp->mmio_addr + (reg))) + +#ifndef DMA_64BIT_MASK +#define DMA_64BIT_MASK 0xffffffffffffffffULL +#endif + +#ifndef DMA_32BIT_MASK +#define DMA_32BIT_MASK 0x00000000ffffffffULL +#endif + +#ifndef NETDEV_TX_OK +#define NETDEV_TX_OK 0 /* driver took care of packet */ +#endif + +#ifndef NETDEV_TX_BUSY +#define NETDEV_TX_BUSY 1 /* driver tx path was busy*/ +#endif + +#ifndef NETDEV_TX_LOCKED +#define NETDEV_TX_LOCKED -1t /* driver tx lock was already taken */ +#endif + +#ifndef ADVERTISED_Pause +#define ADVERTISED_Pause (1 << 13) +#endif + +#ifndef ADVERTISED_Asym_Pause +#define ADVERTISED_Asym_Pause (1 << 14) +#endif + +#ifndef ADVERTISE_PAUSE_CAP +#define ADVERTISE_PAUSE_CAP 0x400 +#endif + +#ifndef ADVERTISE_PAUSE_ASYM +#define ADVERTISE_PAUSE_ASYM 0x800 +#endif + +#ifndef MII_CTRL1000 +#define MII_CTRL1000 0x09 +#endif + +#ifndef ADVERTISE_1000FULL +#define ADVERTISE_1000FULL 0x200 +#endif + +#ifndef ADVERTISE_1000HALF +#define ADVERTISE_1000HALF 0x100 +#endif + +#ifndef BIT_ULL +#define BIT_ULL(nr) (1ULL << (nr)) +#endif + +#ifndef ADVERTISED_2500baseX_Full +#define ADVERTISED_2500baseX_Full 0x8000 +#endif +#define RTK_ADVERTISED_5000baseX_Full BIT_ULL(48) +#define RTK_SUPPORTED_5000baseX_Full BIT_ULL(48) + +#define RTK_ADVERTISE_2500FULL 0x80 +#define RTK_ADVERTISE_5000FULL 0x100 +#define RTK_ADVERTISE_10000FULL 0x1000 +#define RTK_LPA_ADVERTISE_2500FULL 0x20 +#define RTK_LPA_ADVERTISE_5000FULL 0x40 +#define RTK_LPA_ADVERTISE_10000FULL 0x800 + +#define RTK_EEE_ADVERTISE_2500FULL BIT(0) +#define RTK_EEE_ADVERTISE_5000FULL BIT(1) +#define RTK_LPA_EEE_ADVERTISE_2500FULL BIT(0) +#define RTK_LPA_EEE_ADVERTISE_5000FULL BIT(1) + +/* Tx NO CLOSE */ +#define MAX_TX_NO_CLOSE_DESC_PTR_V2 0x10000 +#define MAX_TX_NO_CLOSE_DESC_PTR_MASK_V2 0xFFFF +#define MAX_TX_NO_CLOSE_DESC_PTR_V3 0x100000000 +#define MAX_TX_NO_CLOSE_DESC_PTR_MASK_V3 0xFFFFFFFF +#define MAX_TX_NO_CLOSE_DESC_PTR_V4 0x80000000 +#define MAX_TX_NO_CLOSE_DESC_PTR_MASK_V4 0x7FFFFFFF +#define TX_NO_CLOSE_SW_PTR_MASK_V2 0x1FFFF + +#ifndef ETH_MIN_MTU +#define ETH_MIN_MTU 68 +#endif + +#define D0_SPEED_UP_SPEED_DISABLE 0 +#define D0_SPEED_UP_SPEED_1000 1 +#define D0_SPEED_UP_SPEED_2500 2 +#define D0_SPEED_UP_SPEED_5000 3 +#define D0_SPEED_UP_SPEED_10000 4 + +#define RTL8127_MAC_MCU_PAGE_SIZE 256 //256 words + +#ifndef WRITE_ONCE +#define WRITE_ONCE(var, val) (*((volatile typeof(val) *)(&(var))) = (val)) +#endif +#ifndef READ_ONCE +#define READ_ONCE(var) (*((volatile typeof(var) *)(&(var)))) +#endif + +#ifndef SPEED_5000 +#define SPEED_5000 5000 +#endif + +#ifndef SPEED_10000 +#define SPEED_10000 10000 +#endif + +#define R8127_LINK_STATE_OFF 0 +#define R8127_LINK_STATE_ON 1 +#define R8127_LINK_STATE_UNKNOWN 2 + +/*****************************************************************************/ + +//#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,3) +#if ((LINUX_VERSION_CODE < KERNEL_VERSION(2,4,27)) || \ + ((LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0)) && \ + (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,3)))) +/* copied from linux kernel 2.6.20 include/linux/netdev.h */ +#define NETDEV_ALIGN 32 +#define NETDEV_ALIGN_CONST (NETDEV_ALIGN - 1) + +static inline void *netdev_priv(struct net_device *dev) +{ + return (char *)dev + ((sizeof(struct net_device) + + NETDEV_ALIGN_CONST) + & ~NETDEV_ALIGN_CONST); +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,3) + +/*****************************************************************************/ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) +#define RTLDEV tp +#else +#define RTLDEV dev +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) +/*****************************************************************************/ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) +typedef struct net_device *napi_ptr; +typedef int *napi_budget; + +#define napi dev +#define RTL_NAPI_CONFIG(ndev, priv, function, weig) ndev->poll=function; \ + ndev->weight=weig; +#define RTL_NAPI_QUOTA(budget, ndev) min(*budget, ndev->quota) +#define RTL_GET_PRIV(stuct_ptr, priv_struct) netdev_priv(stuct_ptr) +#define RTL_GET_NETDEV(priv_ptr) +#define RTL_RX_QUOTA(budget) *budget +#define RTL_NAPI_QUOTA_UPDATE(ndev, work_done, budget) *budget -= work_done; \ + ndev->quota -= work_done; +#define RTL_NETIF_RX_COMPLETE(dev, napi, work_done) netif_rx_complete(dev) +#define RTL_NETIF_RX_SCHEDULE_PREP(dev, napi) netif_rx_schedule_prep(dev) +#define __RTL_NETIF_RX_SCHEDULE(dev, napi) __netif_rx_schedule(dev) +#define RTL_NAPI_RETURN_VALUE work_done >= work_to_do +#define RTL_NAPI_ENABLE(dev, napi) netif_poll_enable(dev) +#define RTL_NAPI_DISABLE(dev, napi) netif_poll_disable(dev) +#define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1)) +#else +typedef struct napi_struct *napi_ptr; +typedef int napi_budget; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,1,0) +#define RTL_NAPI_CONFIG(ndev, priv, function, weight) netif_napi_add_weight(ndev, &priv->napi, function, weight) +#else +#define RTL_NAPI_CONFIG(ndev, priv, function, weight) netif_napi_add(ndev, &priv->napi, function, weight) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(6,1,0) +#define RTL_NAPI_QUOTA(budget, ndev) min(budget, budget) +#define RTL_GET_PRIV(stuct_ptr, priv_struct) container_of(stuct_ptr, priv_struct, stuct_ptr) +#define RTL_GET_NETDEV(priv_ptr) struct net_device *dev = priv_ptr->dev; +#define RTL_RX_QUOTA(budget) budget +#define RTL_NAPI_QUOTA_UPDATE(ndev, work_done, budget) +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) +#define RTL_NETIF_RX_COMPLETE(dev, napi, work_done) netif_rx_complete(dev, napi) +#define RTL_NETIF_RX_SCHEDULE_PREP(dev, napi) netif_rx_schedule_prep(dev, napi) +#define __RTL_NETIF_RX_SCHEDULE(dev, napi) __netif_rx_schedule(dev, napi) +#endif +#if LINUX_VERSION_CODE == KERNEL_VERSION(2,6,29) +#define RTL_NETIF_RX_COMPLETE(dev, napi, work_done) netif_rx_complete(napi) +#define RTL_NETIF_RX_SCHEDULE_PREP(dev, napi) netif_rx_schedule_prep(napi) +#define __RTL_NETIF_RX_SCHEDULE(dev, napi) __netif_rx_schedule(napi) +#endif +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,29) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,19,0) +#define RTL_NETIF_RX_COMPLETE(dev, napi, work_done) napi_complete_done(napi, work_done) +#else +#define RTL_NETIF_RX_COMPLETE(dev, napi, work_done) napi_complete(napi) +#endif +#define RTL_NETIF_RX_SCHEDULE_PREP(dev, napi) napi_schedule_prep(napi) +#define __RTL_NETIF_RX_SCHEDULE(dev, napi) __napi_schedule(napi) +#endif +#define RTL_NAPI_RETURN_VALUE work_done +#define RTL_NAPI_ENABLE(dev, napi) napi_enable(napi) +#define RTL_NAPI_DISABLE(dev, napi) napi_disable(napi) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,27) +#define RTL_NAPI_DEL(priv) +#else +#define RTL_NAPI_DEL(priv) netif_napi_del(&priv->napi) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,27) + +/*****************************************************************************/ +#ifdef CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,6,0) +#define RTL_NAPI_CONSUME_SKB_ANY(skb, budget) napi_consume_skb(skb, budget) +#elif LINUX_VERSION_CODE >= KERNEL_VERSION(3,14,0) +#define RTL_NAPI_CONSUME_SKB_ANY(skb, budget) dev_consume_skb_any(skb); +#else +#define RTL_NAPI_CONSUME_SKB_ANY(skb, budget) dev_kfree_skb_any(skb); +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(4,6,0) +#else //CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,14,0) +#define RTL_NAPI_CONSUME_SKB_ANY(skb, budget) dev_consume_skb_any(skb); +#else +#define RTL_NAPI_CONSUME_SKB_ANY(skb, budget) dev_kfree_skb_any(skb); +#endif +#endif //CONFIG_R8127_NAPI + +/*****************************************************************************/ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,9) +#ifdef __CHECKER__ +#define __iomem __attribute__((noderef, address_space(2))) +extern void __chk_io_ptr(void __iomem *); +#define __bitwise __attribute__((bitwise)) +#else +#define __iomem +#define __chk_io_ptr(x) (void)0 +#define __bitwise +#endif +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,9) + +/*****************************************************************************/ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,8) +#ifdef __CHECKER__ +#define __force __attribute__((force)) +#else +#define __force +#endif +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,8) + +#ifndef module_param +#define module_param(v,t,p) MODULE_PARM(v, "i"); +#endif + +#ifndef PCI_DEVICE +#define PCI_DEVICE(vend,dev) \ + .vendor = (vend), .device = (dev), \ + .subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID +#endif + +/*****************************************************************************/ +/* 2.5.28 => 2.4.23 */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,5,28)) + +static inline void _kc_synchronize_irq(void) +{ + synchronize_irq(); +} +#undef synchronize_irq +#define synchronize_irq(X) _kc_synchronize_irq() + +#include +#define work_struct tq_struct +#undef INIT_WORK +#define INIT_WORK(a,b,c) INIT_TQUEUE(a,(void (*)(void *))b,c) +#undef container_of +#define container_of list_entry +#define schedule_work schedule_task +#define flush_scheduled_work flush_scheduled_tasks +#endif /* 2.5.28 => 2.4.17 */ + +/*****************************************************************************/ +/* 2.6.4 => 2.6.0 */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4)) +#define MODULE_VERSION(_version) MODULE_INFO(version, _version) +#endif /* 2.6.4 => 2.6.0 */ +/*****************************************************************************/ +/* 2.6.0 => 2.5.28 */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)) +#define MODULE_INFO(version, _version) +#ifndef CONFIG_E1000_DISABLE_PACKET_SPLIT +#define CONFIG_E1000_DISABLE_PACKET_SPLIT 1 +#endif + +#define pci_set_consistent_dma_mask(dev,mask) 1 + +#undef dev_put +#define dev_put(dev) __dev_put(dev) + +#ifndef skb_fill_page_desc +#define skb_fill_page_desc _kc_skb_fill_page_desc +extern void _kc_skb_fill_page_desc(struct sk_buff *skb, int i, struct page *page, int off, int size); +#endif + +#ifndef pci_dma_mapping_error +#define pci_dma_mapping_error _kc_pci_dma_mapping_error +static inline int _kc_pci_dma_mapping_error(dma_addr_t dma_addr) +{ + return dma_addr == 0; +} +#endif + +#undef ALIGN +#define ALIGN(x,a) (((x)+(a)-1)&~((a)-1)) + +#endif /* 2.6.0 => 2.5.28 */ + +/*****************************************************************************/ +/* 2.4.22 => 2.4.17 */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,4,22)) +#define pci_name(x) ((x)->slot_name) +#endif /* 2.4.22 => 2.4.17 */ + +/*****************************************************************************/ +/* 2.6.5 => 2.6.0 */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,5)) +#define pci_dma_sync_single_for_cpu pci_dma_sync_single +#define pci_dma_sync_single_for_device pci_dma_sync_single_for_cpu +#endif /* 2.6.5 => 2.6.0 */ + +/*****************************************************************************/ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) +/* + * initialize a work-struct's func and data pointers: + */ +#define PREPARE_WORK(_work, _func, _data) \ + do { \ + (_work)->func = _func; \ + (_work)->data = _data; \ + } while (0) + +#endif +/*****************************************************************************/ +/* 2.6.4 => 2.6.0 */ +#if ((LINUX_VERSION_CODE < KERNEL_VERSION(2,4,25) && \ + LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22)) || \ + (LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) && \ + LINUX_VERSION_CODE < KERNEL_VERSION(2,6,4))) +#define ETHTOOL_OPS_COMPAT +#endif /* 2.6.4 => 2.6.0 */ + +/*****************************************************************************/ +/* Installations with ethtool version without eeprom, adapter id, or statistics + * support */ + +#ifndef ETH_GSTRING_LEN +#define ETH_GSTRING_LEN 32 +#endif + +#ifndef ETHTOOL_GSTATS +#define ETHTOOL_GSTATS 0x1d +#undef ethtool_drvinfo +#define ethtool_drvinfo k_ethtool_drvinfo +struct k_ethtool_drvinfo { + u32 cmd; + char driver[32]; + char version[32]; + char fw_version[32]; + char bus_info[32]; + char reserved1[32]; + char reserved2[16]; + u32 n_stats; + u32 testinfo_len; + u32 eedump_len; + u32 regdump_len; +}; + +struct ethtool_stats { + u32 cmd; + u32 n_stats; + u64 data[0]; +}; +#endif /* ETHTOOL_GSTATS */ + +#ifndef ETHTOOL_PHYS_ID +#define ETHTOOL_PHYS_ID 0x1c +#endif /* ETHTOOL_PHYS_ID */ + +#ifndef ETHTOOL_GSTRINGS +#define ETHTOOL_GSTRINGS 0x1b +enum ethtool_stringset { + ETH_SS_TEST = 0, + ETH_SS_STATS, +}; +struct ethtool_gstrings { + u32 cmd; /* ETHTOOL_GSTRINGS */ + u32 string_set; /* string set id e.c. ETH_SS_TEST, etc*/ + u32 len; /* number of strings in the string set */ + u8 data[0]; +}; +#endif /* ETHTOOL_GSTRINGS */ + +#ifndef ETHTOOL_TEST +#define ETHTOOL_TEST 0x1a +enum ethtool_test_flags { + ETH_TEST_FL_OFFLINE = (1 << 0), + ETH_TEST_FL_FAILED = (1 << 1), +}; +struct ethtool_test { + u32 cmd; + u32 flags; + u32 reserved; + u32 len; + u64 data[0]; +}; +#endif /* ETHTOOL_TEST */ + +#ifndef ETHTOOL_GEEPROM +#define ETHTOOL_GEEPROM 0xb +#undef ETHTOOL_GREGS +struct ethtool_eeprom { + u32 cmd; + u32 magic; + u32 offset; + u32 len; + u8 data[0]; +}; + +struct ethtool_value { + u32 cmd; + u32 data; +}; +#endif /* ETHTOOL_GEEPROM */ + +#ifndef ETHTOOL_GLINK +#define ETHTOOL_GLINK 0xa +#endif /* ETHTOOL_GLINK */ + +#ifndef ETHTOOL_GREGS +#define ETHTOOL_GREGS 0x00000004 /* Get NIC registers */ +#define ethtool_regs _kc_ethtool_regs +/* for passing big chunks of data */ +struct _kc_ethtool_regs { + u32 cmd; + u32 version; /* driver-specific, indicates different chips/revs */ + u32 len; /* bytes */ + u8 data[0]; +}; +#endif /* ETHTOOL_GREGS */ + +#ifndef ETHTOOL_GMSGLVL +#define ETHTOOL_GMSGLVL 0x00000007 /* Get driver message level */ +#endif +#ifndef ETHTOOL_SMSGLVL +#define ETHTOOL_SMSGLVL 0x00000008 /* Set driver msg level, priv. */ +#endif +#ifndef ETHTOOL_NWAY_RST +#define ETHTOOL_NWAY_RST 0x00000009 /* Restart autonegotiation, priv */ +#endif +#ifndef ETHTOOL_GLINK +#define ETHTOOL_GLINK 0x0000000a /* Get link status */ +#endif +#ifndef ETHTOOL_GEEPROM +#define ETHTOOL_GEEPROM 0x0000000b /* Get EEPROM data */ +#endif +#ifndef ETHTOOL_SEEPROM +#define ETHTOOL_SEEPROM 0x0000000c /* Set EEPROM data */ +#endif +#ifndef ETHTOOL_GCOALESCE +#define ETHTOOL_GCOALESCE 0x0000000e /* Get coalesce config */ +/* for configuring coalescing parameters of chip */ +#define ethtool_coalesce _kc_ethtool_coalesce +struct _kc_ethtool_coalesce { + u32 cmd; /* ETHTOOL_{G,S}COALESCE */ + + /* How many usecs to delay an RX interrupt after + * a packet arrives. If 0, only rx_max_coalesced_frames + * is used. + */ + u32 rx_coalesce_usecs; + + /* How many packets to delay an RX interrupt after + * a packet arrives. If 0, only rx_coalesce_usecs is + * used. It is illegal to set both usecs and max frames + * to zero as this would cause RX interrupts to never be + * generated. + */ + u32 rx_max_coalesced_frames; + + /* Same as above two parameters, except that these values + * apply while an IRQ is being serviced by the host. Not + * all cards support this feature and the values are ignored + * in that case. + */ + u32 rx_coalesce_usecs_irq; + u32 rx_max_coalesced_frames_irq; + + /* How many usecs to delay a TX interrupt after + * a packet is sent. If 0, only tx_max_coalesced_frames + * is used. + */ + u32 tx_coalesce_usecs; + + /* How many packets to delay a TX interrupt after + * a packet is sent. If 0, only tx_coalesce_usecs is + * used. It is illegal to set both usecs and max frames + * to zero as this would cause TX interrupts to never be + * generated. + */ + u32 tx_max_coalesced_frames; + + /* Same as above two parameters, except that these values + * apply while an IRQ is being serviced by the host. Not + * all cards support this feature and the values are ignored + * in that case. + */ + u32 tx_coalesce_usecs_irq; + u32 tx_max_coalesced_frames_irq; + + /* How many usecs to delay in-memory statistics + * block updates. Some drivers do not have an in-memory + * statistic block, and in such cases this value is ignored. + * This value must not be zero. + */ + u32 stats_block_coalesce_usecs; + + /* Adaptive RX/TX coalescing is an algorithm implemented by + * some drivers to improve latency under low packet rates and + * improve throughput under high packet rates. Some drivers + * only implement one of RX or TX adaptive coalescing. Anything + * not implemented by the driver causes these values to be + * silently ignored. + */ + u32 use_adaptive_rx_coalesce; + u32 use_adaptive_tx_coalesce; + + /* When the packet rate (measured in packets per second) + * is below pkt_rate_low, the {rx,tx}_*_low parameters are + * used. + */ + u32 pkt_rate_low; + u32 rx_coalesce_usecs_low; + u32 rx_max_coalesced_frames_low; + u32 tx_coalesce_usecs_low; + u32 tx_max_coalesced_frames_low; + + /* When the packet rate is below pkt_rate_high but above + * pkt_rate_low (both measured in packets per second) the + * normal {rx,tx}_* coalescing parameters are used. + */ + + /* When the packet rate is (measured in packets per second) + * is above pkt_rate_high, the {rx,tx}_*_high parameters are + * used. + */ + u32 pkt_rate_high; + u32 rx_coalesce_usecs_high; + u32 rx_max_coalesced_frames_high; + u32 tx_coalesce_usecs_high; + u32 tx_max_coalesced_frames_high; + + /* How often to do adaptive coalescing packet rate sampling, + * measured in seconds. Must not be zero. + */ + u32 rate_sample_interval; +}; +#endif /* ETHTOOL_GCOALESCE */ + +#ifndef ETHTOOL_SCOALESCE +#define ETHTOOL_SCOALESCE 0x0000000f /* Set coalesce config. */ +#endif +#ifndef ETHTOOL_GRINGPARAM +#define ETHTOOL_GRINGPARAM 0x00000010 /* Get ring parameters */ +/* for configuring RX/TX ring parameters */ +#define ethtool_ringparam _kc_ethtool_ringparam +struct _kc_ethtool_ringparam { + u32 cmd; /* ETHTOOL_{G,S}RINGPARAM */ + + /* Read only attributes. These indicate the maximum number + * of pending RX/TX ring entries the driver will allow the + * user to set. + */ + u32 rx_max_pending; + u32 rx_mini_max_pending; + u32 rx_jumbo_max_pending; + u32 tx_max_pending; + + /* Values changeable by the user. The valid values are + * in the range 1 to the "*_max_pending" counterpart above. + */ + u32 rx_pending; + u32 rx_mini_pending; + u32 rx_jumbo_pending; + u32 tx_pending; +}; +#endif /* ETHTOOL_GRINGPARAM */ + +#ifndef ETHTOOL_SRINGPARAM +#define ETHTOOL_SRINGPARAM 0x00000011 /* Set ring parameters, priv. */ +#endif +#ifndef ETHTOOL_GPAUSEPARAM +#define ETHTOOL_GPAUSEPARAM 0x00000012 /* Get pause parameters */ +/* for configuring link flow control parameters */ +#define ethtool_pauseparam _kc_ethtool_pauseparam +struct _kc_ethtool_pauseparam { + u32 cmd; /* ETHTOOL_{G,S}PAUSEPARAM */ + + /* If the link is being auto-negotiated (via ethtool_cmd.autoneg + * being true) the user may set 'autonet' here non-zero to have the + * pause parameters be auto-negotiated too. In such a case, the + * {rx,tx}_pause values below determine what capabilities are + * advertised. + * + * If 'autoneg' is zero or the link is not being auto-negotiated, + * then {rx,tx}_pause force the driver to use/not-use pause + * flow control. + */ + u32 autoneg; + u32 rx_pause; + u32 tx_pause; +}; +#endif /* ETHTOOL_GPAUSEPARAM */ + +#ifndef ETHTOOL_SPAUSEPARAM +#define ETHTOOL_SPAUSEPARAM 0x00000013 /* Set pause parameters. */ +#endif +#ifndef ETHTOOL_GRXCSUM +#define ETHTOOL_GRXCSUM 0x00000014 /* Get RX hw csum enable (ethtool_value) */ +#endif +#ifndef ETHTOOL_SRXCSUM +#define ETHTOOL_SRXCSUM 0x00000015 /* Set RX hw csum enable (ethtool_value) */ +#endif +#ifndef ETHTOOL_GTXCSUM +#define ETHTOOL_GTXCSUM 0x00000016 /* Get TX hw csum enable (ethtool_value) */ +#endif +#ifndef ETHTOOL_STXCSUM +#define ETHTOOL_STXCSUM 0x00000017 /* Set TX hw csum enable (ethtool_value) */ +#endif +#ifndef ETHTOOL_GSG +#define ETHTOOL_GSG 0x00000018 /* Get scatter-gather enable +* (ethtool_value) */ +#endif +#ifndef ETHTOOL_SSG +#define ETHTOOL_SSG 0x00000019 /* Set scatter-gather enable +* (ethtool_value). */ +#endif +#ifndef ETHTOOL_TEST +#define ETHTOOL_TEST 0x0000001a /* execute NIC self-test, priv. */ +#endif +#ifndef ETHTOOL_GSTRINGS +#define ETHTOOL_GSTRINGS 0x0000001b /* get specified string set */ +#endif +#ifndef ETHTOOL_PHYS_ID +#define ETHTOOL_PHYS_ID 0x0000001c /* identify the NIC */ +#endif +#ifndef ETHTOOL_GSTATS +#define ETHTOOL_GSTATS 0x0000001d /* get NIC-specific statistics */ +#endif +#ifndef ETHTOOL_GTSO +#define ETHTOOL_GTSO 0x0000001e /* Get TSO enable (ethtool_value) */ +#endif +#ifndef ETHTOOL_STSO +#define ETHTOOL_STSO 0x0000001f /* Set TSO enable (ethtool_value) */ +#endif + +#ifndef ETHTOOL_BUSINFO_LEN +#define ETHTOOL_BUSINFO_LEN 32 +#endif + +/*****************************************************************************/ + +enum RTL8127_registers { + MAC0 = 0x00, /* Ethernet hardware address. */ + MAC4 = 0x04, + MAR0 = 0x08, /* Multicast filter. */ + CounterAddrLow = 0x10, + CounterAddrHigh = 0x14, + CustomLED = 0x18, + TxDescStartAddrLow = 0x20, + TxDescStartAddrHigh = 0x24, + TxHDescStartAddrLow = 0x28, + TxHDescStartAddrHigh = 0x2c, + FLASH = 0x30, + INT_CFG0_8125 = 0x34, + ERSR = 0x36, + ChipCmd = 0x37, + TxPoll = 0x38, + IntrMask = 0x3C, + IntrStatus = 0x3E, + TxConfig = 0x40, + RxConfig = 0x44, + TCTR = 0x48, + Cfg9346 = 0x50, + Config0 = 0x51, + Config1 = 0x52, + Config2 = 0x53, + Config3 = 0x54, + Config4 = 0x55, + Config5 = 0x56, + TDFNR = 0x57, + TimeInt0 = 0x58, + TimeInt1 = 0x5C, + PHYAR = 0x60, + CSIDR = 0x64, + CSIAR = 0x68, + PHYstatus = 0x6C, + MACDBG = 0x6D, + GPIO = 0x6E, + PMCH = 0x6F, + ERIDR = 0x70, + ERIAR = 0x74, + INT_CFG1_8125 = 0x7A, + EPHY_RXER_NUM = 0x7C, + EPHYAR = 0x80, + TimeInt2 = 0x8C, + OCPDR = 0xB0, + MACOCP = 0xB0, + OCPAR = 0xB4, + SecMAC0 = 0xB4, + SecMAC4 = 0xB8, + PHYOCP = 0xB8, + DBG_reg = 0xD1, + TwiCmdReg = 0xD2, + MCUCmd_reg = 0xD3, + RxMaxSize = 0xDA, + EFUSEAR = 0xDC, + CPlusCmd = 0xE0, + IntrMitigate = 0xE2, + RxDescAddrLow = 0xE4, + RxDescAddrHigh = 0xE8, + MTPS = 0xEC, + FuncEvent = 0xF0, + PPSW = 0xF2, + FuncEventMask = 0xF4, + TimeInt3 = 0xF4, + FuncPresetState = 0xF8, + CMAC_IBCR0 = 0xF8, + CMAC_IBCR2 = 0xF9, + CMAC_IBIMR0 = 0xFA, + CMAC_IBISR0 = 0xFB, + FuncForceEvent = 0xFC, + //8125 + IMR0_8125 = 0x38, + ISR0_8125 = 0x3C, + TPPOLL_8125 = 0x90, + IMR1_8125 = 0x800, + ISR1_8125 = 0x802, + IMR2_8125 = 0x804, + ISR2_8125 = 0x806, + IMR3_8125 = 0x808, + ISR3_8125 = 0x80A, + BACKUP_ADDR0_8125 = 0x19E0, + BACKUP_ADDR1_8125 = 0X19E4, + TCTR0_8125 = 0x0048, + TCTR1_8125 = 0x004C, + TCTR2_8125 = 0x0088, + TCTR3_8125 = 0x001C, + TIMER_INT0_8125 = 0x0058, + TIMER_INT1_8125 = 0x005C, + TIMER_INT2_8125 = 0x008C, + TIMER_INT3_8125 = 0x00F4, + INT_MITI_V2_0_RX = 0x0A00, + INT_MITI_V2_0_TX = 0x0A02, + INT_MITI_V2_1_RX = 0x0A08, + INT_MITI_V2_1_TX = 0x0A0A, + IMR_V2_CLEAR_REG_8125 = 0x0D00, + ISR_V2_8125 = 0x0D04, + IMR_V2_SET_REG_8125 = 0x0D0C, + TDU_STA_8125 = 0x0D08, + RDU_STA_8125 = 0x0D0A, + IMR_V4_L2_CLEAR_REG_8125 = 0x0D10, + IMR_V4_L2_SET_REG_8125 = 0x0D18, + ISR_V4_L2_8125 = 0x0D14, + SW_TAIL_PTR0_8125BP = 0x0D30, + SW_TAIL_PTR1_8125BP = 0x0D38, + HW_CLO_PTR0_8125BP = 0x0D34, + HW_CLO_PTR1_8125BP = 0x0D3C, + DOUBLE_VLAN_CONFIG = 0x1000, + TX_NEW_CTRL = 0x203E, + TNPDS_Q1_LOW_8125 = 0x2100, + PLA_TXQ0_IDLE_CREDIT = 0x2500, + PLA_TXQ1_IDLE_CREDIT = 0x2504, + SW_TAIL_PTR0_8125 = 0x2800, + HW_CLO_PTR0_8125 = 0x2802, + SW_TAIL_PTR0_8126 = 0x2800, + HW_CLO_PTR0_8126 = 0x2800, + RDSAR_Q1_LOW_8125 = 0x4000, + RSS_CTRL_8125 = 0x4500, + Q_NUM_CTRL_8125 = 0x4800, + RSS_KEY_8125 = 0x4600, + RSS_INDIRECTION_TBL_8125_V2 = 0x4700, + EEE_TXIDLE_TIMER_8125 = 0x6048, + PTP_CTRL_8125 = 0x6800, + PTP_STATUS_8125 = 0x6802, + PTP_ISR_8125 = 0x6804, + PTP_IMR_8125 = 0x6805, + PTP_TIME_CORRECT_CMD_8125 = 0x6806, + PTP_SOFT_CONFIG_Time_NS_8125 = 0x6808, + PTP_SOFT_CONFIG_Time_S_8125 = 0x680C, + PTP_SOFT_CONFIG_Time_Sign = 0x6812, + PTP_LOCAL_Time_SUB_NS_8125 = 0x6814, + PTP_LOCAL_Time_NS_8125 = 0x6818, + PTP_LOCAL_Time_S_8125 = 0x681C, + PTP_Time_SHIFTER_S_8125 = 0x6856, + PPS_RISE_TIME_NS_8125 = 0x68A0, + PPS_RISE_TIME_S_8125 = 0x68A4, + PTP_EGRESS_TIME_BASE_NS_8125 = 0XCF20, + PTP_EGRESS_TIME_BASE_S_8125 = 0XCF24, + PTP_CTL = 0xE400, + PTP_INER = 0xE402, + PTP_INSR = 0xE404, + PTP_SYNCE_CTL = 0xE406, + PTP_GEN_CFG = 0xE408, + PTP_CLK_CFG_8126 = 0xE410, + PTP_CFG_NS_LO_8126 = 0xE412, + PTP_CFG_NS_HI_8126 = 0xE414, + PTP_CFG_S_LO_8126 = 0xE416, + PTP_CFG_S_MI_8126 = 0xE418, + PTP_CFG_S_HI_8126 = 0xE41A, + PTP_TAI_CFG = 0xE420, + PTP_TAI_TS_S_LO = 0xE42A, + PTP_TAI_TS_S_HI = 0xE42C, + PTP_TRX_TS_STA = 0xE430, + PTP_TRX_TS_NS_LO = 0xE446, + PTP_TRX_TS_NS_HI = 0xE448, + PTP_TRX_TS_S_LO = 0xE44A, + PTP_TRX_TS_S_MI = 0xE44C, + PTP_TRX_TS_S_HI = 0xE44E, + + //TCAM + TCAM_NOTVALID_ADDR = 0xA000, + TCAM_VALID_ADDR = 0xA800, + TCAM_MAC_ADDR = 448, + TCAM_VLAN_TAG = 496, + //TCAM V2 + TCAM_NOTVALID_ADDR_V2 = 0xA000, + TCAM_VALID_ADDR_V2 = 0xB000, + TCAM_MAC_ADDR_V2 = 0x00, + TCAM_VLAN_TAG_V2 = 0x03, +}; + +enum RTL8127_register_content { + /* InterruptStatusBits */ + SYSErr = 0x8000, + PCSTimeout = 0x4000, + SWInt = 0x0100, + TxDescUnavail = 0x0080, + RxFIFOOver = 0x0040, + LinkChg = 0x0020, + RxDescUnavail = 0x0010, + TxErr = 0x0008, + TxOK = 0x0004, + RxErr = 0x0002, + RxOK = 0x0001, + RxDU1 = 0x0002, + RxOK1 = 0x0001, + + /* RxStatusDesc */ + RxRWT = (1 << 22), + RxRES = (1 << 21), + RxRUNT = (1 << 20), + RxCRC = (1 << 19), + + RxRWT_V3 = (1 << 18), + RxRES_V3 = (1 << 20), + RxRUNT_V3 = (1 << 19), + RxCRC_V3 = (1 << 17), + + RxRES_V4 = (1 << 22), + RxRUNT_V4 = (1 << 21), + RxCRC_V4 = (1 << 20), + + /* ChipCmdBits */ + StopReq = 0x80, + CmdReset = 0x10, + CmdRxEnb = 0x08, + CmdTxEnb = 0x04, + RxBufEmpty = 0x01, + + /* Cfg9346Bits */ + Cfg9346_EEM_MASK = 0xC0, + Cfg9346_Lock = 0x00, + Cfg9346_Unlock = 0xC0, + Cfg9346_EEDO = (1 << 0), + Cfg9346_EEDI = (1 << 1), + Cfg9346_EESK = (1 << 2), + Cfg9346_EECS = (1 << 3), + Cfg9346_EEM0 = (1 << 6), + Cfg9346_EEM1 = (1 << 7), + + /* rx_mode_bits */ + AcceptErr = 0x20, + AcceptRunt = 0x10, + AcceptBroadcast = 0x08, + AcceptMulticast = 0x04, + AcceptMyPhys = 0x02, + AcceptAllPhys = 0x01, + AcceppVlanPhys = 0x8000, + + /* Transmit Priority Polling*/ + HPQ = 0x80, + NPQ = 0x40, + FSWInt = 0x01, + + /* RxConfigBits */ + Reserved2_shift = 13, + RxCfgDMAShift = 8, + EnableRxDescV3 = (1 << 24), + EnableRxDescV4_1 = (1 << 24), + EnableOuterVlan = (1 << 23), + EnableInnerVlan = (1 << 22), + RxCfg_128_int_en = (1 << 15), + RxCfg_fet_multi_en = (1 << 14), + RxCfg_half_refetch = (1 << 13), + RxCfg_pause_slot_en = (1 << 11), + RxCfg_9356SEL = (1 << 6), + EnableRxDescV4_0 = (1 << 1), //not in rcr + + /* TxConfigBits */ + TxInterFrameGapShift = 24, + TxDMAShift = 8, /* DMA burst value (0-7) is shift this many bits */ + TxMACLoopBack = (1 << 17), /* MAC loopback */ + + /* Config1 register */ + LEDS1 = (1 << 7), + LEDS0 = (1 << 6), + Speed_down = (1 << 4), + MEMMAP = (1 << 3), + IOMAP = (1 << 2), + VPD = (1 << 1), + PMEnable = (1 << 0), /* Power Management Enable */ + + /* Config2 register */ + PMSTS_En = (1 << 5), + + /* Config3 register */ + Isolate_en = (1 << 12), /* Isolate enable */ + MagicPacket = (1 << 5), /* Wake up when receives a Magic Packet */ + LinkUp = (1 << 4), /* This bit is reserved in RTL8125B.*/ + /* Wake up when the cable connection is re-established */ + ECRCEN = (1 << 3), /* This bit is reserved in RTL8125B*/ + Jumbo_En0 = (1 << 2), /* This bit is reserved in RTL8125B*/ + RDY_TO_L23 = (1 << 1), /* This bit is reserved in RTL8125B*/ + Beacon_en = (1 << 0), /* This bit is reserved in RTL8125B*/ + + /* Config4 register */ + Jumbo_En1 = (1 << 1), /* This bit is reserved in RTL8125B*/ + + /* Config5 register */ + BWF = (1 << 6), /* Accept Broadcast wakeup frame */ + MWF = (1 << 5), /* Accept Multicast wakeup frame */ + UWF = (1 << 4), /* Accept Unicast wakeup frame */ + LanWake = (1 << 1), /* LanWake enable/disable */ + PMEStatus = (1 << 0), /* PME status can be reset by PCI RST# */ + + /* CPlusCmd */ + EnableBist = (1 << 15), + Macdbgo_oe = (1 << 14), + Normal_mode = (1 << 13), + Force_halfdup = (1 << 12), + Force_rxflow_en = (1 << 11), + Force_txflow_en = (1 << 10), + Cxpl_dbg_sel = (1 << 9),//This bit is reserved in RTL8125B + ASF = (1 << 8),//This bit is reserved in RTL8125C + PktCntrDisable = (1 << 7), + RxVlan = (1 << 6), + RxChkSum = (1 << 5), + Macdbgo_sel = 0x001C, + INTT_0 = 0x0000, + INTT_1 = 0x0001, + INTT_2 = 0x0002, + INTT_3 = 0x0003, + + /* rtl8127_PHYstatus */ + PowerSaveStatus = 0x80, + _1000bpsL = 0x80000, + _10000bpsF = 0x4000, + _10000bpsL = 0x2000, + _5000bpsF = 0x1000, + _5000bpsL = 0x800, + _2500bpsF = 0x400, + _2500bpsL = 0x200, + TxFlowCtrl = 0x40, + RxFlowCtrl = 0x20, + _1000bpsF = 0x10, + _100bps = 0x08, + _10bps = 0x04, + LinkStatus = 0x02, + FullDup = 0x01, + + /* DBG_reg */ + Fix_Nak_1 = (1 << 4), + Fix_Nak_2 = (1 << 3), + DBGPIN_E2 = (1 << 0), + + /* ResetCounterCommand */ + CounterReset = 0x1, + /* DumpCounterCommand */ + CounterDump = 0x8, + + /* PHY access */ + PHYAR_Flag = 0x80000000, + PHYAR_Write = 0x80000000, + PHYAR_Read = 0x00000000, + PHYAR_Reg_Mask = 0x1f, + PHYAR_Reg_shift = 16, + PHYAR_Data_Mask = 0xffff, + + /* EPHY access */ + EPHYAR_Flag = 0x80000000, + EPHYAR_Write = 0x80000000, + EPHYAR_Read = 0x00000000, + EPHYAR_Reg_Mask = 0x3f, + EPHYAR_Reg_Mask_v2 = 0x7f, + EPHYAR_Reg_shift = 16, + EPHYAR_Data_Mask = 0xffff, + EPHYAR_EXT_ADDR = 0x0ffe, + + /* CSI access */ + CSIAR_Flag = 0x80000000, + CSIAR_Write = 0x80000000, + CSIAR_Read = 0x00000000, + CSIAR_ByteEn = 0x0f, + CSIAR_ByteEn_shift = 12, + CSIAR_Addr_Mask = 0x0fff, + + /* ERI access */ + ERIAR_Flag = 0x80000000, + ERIAR_Write = 0x80000000, + ERIAR_Read = 0x00000000, + ERIAR_Addr_Align = 4, /* ERI access register address must be 4 byte alignment */ + ERIAR_ExGMAC = 0, + ERIAR_MSIX = 1, + ERIAR_ASF = 2, + ERIAR_OOB = 2, + ERIAR_Type_shift = 16, + ERIAR_ByteEn = 0x0f, + ERIAR_ByteEn_shift = 12, + + /* OCP GPHY access */ + OCPDR_Write = 0x80000000, + OCPDR_Read = 0x00000000, + OCPDR_Reg_Mask = 0xFF, + OCPDR_Data_Mask = 0xFFFF, + OCPDR_GPHY_Reg_shift = 16, + OCPAR_Flag = 0x80000000, + OCPAR_GPHY_Write = 0x8000F060, + OCPAR_GPHY_Read = 0x0000F060, + OCPR_Write = 0x80000000, + OCPR_Read = 0x00000000, + OCPR_Addr_Reg_shift = 16, + OCPR_Flag = 0x80000000, + OCP_STD_PHY_BASE_PAGE = 0x0A40, + + /* MCU Command */ + Now_is_oob = (1 << 7), + Txfifo_empty = (1 << 5), + Rxfifo_empty = (1 << 4), + + /* E-FUSE access */ + EFUSE_WRITE = 0x80000000, + EFUSE_WRITE_OK = 0x00000000, + EFUSE_READ = 0x00000000, + EFUSE_READ_OK = 0x80000000, + EFUSE_WRITE_V3 = 0x40000000, + EFUSE_WRITE_OK_V3 = 0x00000000, + EFUSE_READ_V3 = 0x80000000, + EFUSE_READ_OK_V3 = 0x00000000, + EFUSE_Reg_Mask = 0x03FF, + EFUSE_Reg_Shift = 8, + EFUSE_Check_Cnt = 300, + EFUSE_READ_FAIL = 0xFF, + EFUSE_Data_Mask = 0x000000FF, + + /* GPIO */ + GPIO_en = (1 << 0), + + /* PTP */ + PTP_ISR_TOK = (1 << 1), + PTP_ISR_TER = (1 << 2), + PTP_EXEC_CMD = (1 << 7), + PTP_ADJUST_TIME_NS_NEGATIVE = (1 << 30), + PTP_ADJUST_TIME_S_NEGATIVE = (1ULL << 48), + PTP_SOFT_CONFIG_TIME_NS_NEGATIVE = (1 << 30), + PTP_SOFT_CONFIG_TIME_S_NEGATIVE = (1ULL << 48), + + /* New Interrupt Bits */ + INT_CFG0_ENABLE_8125 = (1 << 0), + INT_CFG0_TIMEOUT0_BYPASS_8125 = (1 << 1), + INT_CFG0_MITIGATION_BYPASS_8125 = (1 << 2), + INT_CFG0_RDU_BYPASS_8126 = (1 << 4), + INT_CFG0_MSIX_ENTRY_NUM_MODE = (1 << 5), + INT_CFG0_AUTO_CLEAR_IMR = (1 << 5), + INT_CFG0_AVOID_MISS_INTR = (1 << 6), + ISRIMR_V2_ROK_Q0 = (1 << 0), + ISRIMR_TOK_Q0 = (1 << 16), + ISRIMR_TOK_Q1 = (1 << 18), + ISRIMR_V2_LINKCHG = (1 << 21), + + ISRIMR_V4_ROK_Q0 = (1 << 0), + ISRIMR_V4_LINKCHG = (1 << 29), + + ISRIMR_V5_ROK_Q0 = (1 << 0), + ISRIMR_V5_TOK_Q0 = (1 << 16), + ISRIMR_V5_TOK_Q1 = (1 << 17), + ISRIMR_V5_LINKCHG = (1 << 18), + + ISRIMR_V6_ROK_Q0 = (1 << 0), + ISRIMR_V6_TOK_Q0 = (1 << 8), + ISRIMR_V6_TOK_Q1 = (1 << 9), + ISRIMR_V6_LINKCHG = (1 << 29), + + /* Magic Number */ + RTL8127_MAGIC_NUMBER = 0x0badbadbadbadbadull, +}; + +enum _DescStatusBit { + DescOwn = (1 << 31), /* Descriptor is owned by NIC */ + RingEnd = (1 << 30), /* End of descriptor ring */ + FirstFrag = (1 << 29), /* First segment of a packet */ + LastFrag = (1 << 28), /* Final segment of a packet */ + + DescOwn_V3 = (DescOwn), /* Descriptor is owned by NIC */ + RingEnd_V3 = (RingEnd), /* End of descriptor ring */ + FirstFrag_V3 = (1 << 25), /* First segment of a packet */ + LastFrag_V3 = (1 << 24), /* Final segment of a packet */ + + DescOwn_V4 = (DescOwn), /* Descriptor is owned by NIC */ + RingEnd_V4 = (RingEnd), /* End of descriptor ring */ + FirstFrag_V4 = (FirstFrag), /* First segment of a packet */ + LastFrag_V4 = (LastFrag), /* Final segment of a packet */ + + /* Tx private */ + /*------ offset 0 of tx descriptor ------*/ + LargeSend = (1 << 27), /* TCP Large Send Offload (TSO) */ + GiantSendv4 = (1 << 26), /* TCP Giant Send Offload V4 (GSOv4) */ + GiantSendv6 = (1 << 25), /* TCP Giant Send Offload V6 (GSOv6) */ + LargeSend_DP = (1 << 16), /* TCP Large Send Offload (TSO) */ + MSSShift = 16, /* MSS value position */ + MSSMask = 0x7FFU, /* MSS value 11 bits */ + TxIPCS = (1 << 18), /* Calculate IP checksum */ + TxUDPCS = (1 << 17), /* Calculate UDP/IP checksum */ + TxTCPCS = (1 << 16), /* Calculate TCP/IP checksum */ + TxVlanTag = (1 << 17), /* Add VLAN tag */ + + /*@@@@@@ offset 4 of tx descriptor => bits for RTL8125 only begin @@@@@@*/ + TxUDPCS_C = (1 << 31), /* Calculate UDP/IP checksum */ + TxTCPCS_C = (1 << 30), /* Calculate TCP/IP checksum */ + TxIPCS_C = (1 << 29), /* Calculate IP checksum */ + TxIPV6F_C = (1 << 28), /* Indicate it is an IPv6 packet */ + /*@@@@@@ offset 4 of tx descriptor => bits for RTL8125 only end @@@@@@*/ + + + /* Rx private */ + /*------ offset 0 of rx descriptor ------*/ + PID1 = (1 << 18), /* Protocol ID bit 1/2 */ + PID0 = (1 << 17), /* Protocol ID bit 2/2 */ + +#define RxProtoUDP (PID1) +#define RxProtoTCP (PID0) +#define RxProtoIP (PID1 | PID0) +#define RxProtoMask RxProtoIP + + RxIPF = (1 << 16), /* IP checksum failed */ + RxUDPF = (1 << 15), /* UDP/IP checksum failed */ + RxTCPF = (1 << 14), /* TCP/IP checksum failed */ + RxVlanTag = (1 << 16), /* VLAN tag available */ + + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxUDPT = (1 << 18), + RxTCPT = (1 << 17), + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only end @@@@@@*/ + + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxV6F = (1 << 31), + RxV4F = (1 << 30), + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only end @@@@@@*/ + + + PID1_v3 = (1 << 29), /* Protocol ID bit 1/2 */ + PID0_v3 = (1 << 28), /* Protocol ID bit 2/2 */ + +#define RxProtoUDP_v3 (PID1_v3) +#define RxProtoTCP_v3 (PID0_v3) +#define RxProtoIP_v3 (PID1_v3 | PID0_v3) +#define RxProtoMask_v3 RxProtoIP_v3 + + RxIPF_v3 = (1 << 26), /* IP checksum failed */ + RxUDPF_v3 = (1 << 25), /* UDP/IP checksum failed */ + RxTCPF_v3 = (1 << 24), /* TCP/IP checksum failed */ + RxSCTPF_v3 = (1 << 23), /* SCTP checksum failed */ + RxVlanTag_v3 = (RxVlanTag), /* VLAN tag available */ + + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxUDPT_v3 = (1 << 29), + RxTCPT_v3 = (1 << 28), + RxSCTP_v3 = (1 << 27), + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only end @@@@@@*/ + + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxV6F_v3 = (RxV6F), + RxV4F_v3 = (RxV4F), + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only end @@@@@@*/ + + RxIPF_v4 = (1 << 17), /* IP checksum failed */ + RxUDPF_v4 = (1 << 16), /* UDP/IP checksum failed */ + RxTCPF_v4 = (1 << 15), /* TCP/IP checksum failed */ + RxSCTPF_v4 = (1 << 19), /* SCTP checksum failed */ + RxVlanTag_v4 = (RxVlanTag), /* VLAN tag available */ + + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxUDPT_v4 = (1 << 19), + RxTCPT_v4 = (1 << 18), + RxSCTP_v4 = (1 << 19), + /*@@@@@@ offset 0 of rx descriptor => bits for RTL8125 only end @@@@@@*/ + + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only begin @@@@@@*/ + RxV6F_v4 = (RxV6F), + RxV4F_v4 = (RxV4F), + /*@@@@@@ offset 4 of rx descriptor => bits for RTL8125 only end @@@@@@*/ +}; + +enum features { +// RTL_FEATURE_WOL = (1 << 0), + RTL_FEATURE_MSI = (1 << 1), + RTL_FEATURE_MSIX = (1 << 2), +}; + +enum wol_capability { + WOL_DISABLED = 0, + WOL_ENABLED = 1 +}; + +enum bits { + BIT_0 = (1 << 0), + BIT_1 = (1 << 1), + BIT_2 = (1 << 2), + BIT_3 = (1 << 3), + BIT_4 = (1 << 4), + BIT_5 = (1 << 5), + BIT_6 = (1 << 6), + BIT_7 = (1 << 7), + BIT_8 = (1 << 8), + BIT_9 = (1 << 9), + BIT_10 = (1 << 10), + BIT_11 = (1 << 11), + BIT_12 = (1 << 12), + BIT_13 = (1 << 13), + BIT_14 = (1 << 14), + BIT_15 = (1 << 15), + BIT_16 = (1 << 16), + BIT_17 = (1 << 17), + BIT_18 = (1 << 18), + BIT_19 = (1 << 19), + BIT_20 = (1 << 20), + BIT_21 = (1 << 21), + BIT_22 = (1 << 22), + BIT_23 = (1 << 23), + BIT_24 = (1 << 24), + BIT_25 = (1 << 25), + BIT_26 = (1 << 26), + BIT_27 = (1 << 27), + BIT_28 = (1 << 28), + BIT_29 = (1 << 29), + BIT_30 = (1 << 30), + BIT_31 = (1 << 31) +}; + +#define RTL8127_CP_NUM 4 +#define RTL8127_MAX_SUPPORT_CP_LEN 110 + +enum rtl8127_cp_status { + rtl8127_cp_normal = 0, + rtl8127_cp_short, + rtl8127_cp_open, + rtl8127_cp_mismatch, + rtl8127_cp_unknown +}; + +enum efuse { + EFUSE_NOT_SUPPORT = 0, + EFUSE_SUPPORT_V1, + EFUSE_SUPPORT_V2, + EFUSE_SUPPORT_V3, + EFUSE_SUPPORT_V4, +}; +#define RsvdMask 0x3fffc000 +#define RsvdMaskV3 0x3fff8000 +#define RsvdMaskV4 RsvdMaskV3 + +struct TxDesc { + u32 opts1; + u32 opts2; + u64 addr; + u32 reserved0; + u32 reserved1; + u32 reserved2; + u32 reserved3; +}; + +struct RxDesc { + u32 opts1; + u32 opts2; + u64 addr; +}; + +struct RxDescV3 { + union { + struct { + u32 rsv1; + u32 rsv2; + } RxDescDDWord1; + }; + + union { + struct { + u32 RSSResult; + u16 HeaderBufferLen; + u16 HeaderInfo; + } RxDescNormalDDWord2; + + struct { + u32 rsv5; + u32 rsv6; + } RxDescDDWord2; + }; + + union { + u64 addr; + + struct { + u32 TimeStampLow; + u32 TimeStampHigh; + } RxDescTimeStamp; + + struct { + u32 rsv8; + u32 rsv9; + } RxDescDDWord3; + }; + + union { + struct { + u32 opts2; + u32 opts1; + } RxDescNormalDDWord4; + + struct { + u16 TimeStampHHigh; + u16 rsv11; + u32 opts1; + } RxDescPTPDDWord4; + }; +}; + +struct RxDescV4 { + union { + u64 addr; + + struct { + u32 RSSInfo; + u32 RSSResult; + } RxDescNormalDDWord1; + }; + + struct { + u32 opts2; + u32 opts1; + } RxDescNormalDDWord2; +}; + +enum rxdesc_type { + RXDESC_TYPE_NORMAL=0, + RXDESC_TYPE_NEXT, + RXDESC_TYPE_PTP, + RXDESC_TYPE_MAX +}; + +//Rx Desc Type +enum rx_desc_ring_type { + RX_DESC_RING_TYPE_UNKNOWN=0, + RX_DESC_RING_TYPE_1, + RX_DESC_RING_TYPE_2, + RX_DESC_RING_TYPE_3, + RX_DESC_RING_TYPE_4, + RX_DESC_RING_TYPE_MAX +}; + +enum rx_desc_len { + RX_DESC_LEN_TYPE_1 = (sizeof(struct RxDesc)), + RX_DESC_LEN_TYPE_3 = (sizeof(struct RxDescV3)), + RX_DESC_LEN_TYPE_4 = (sizeof(struct RxDescV4)) +}; + +struct ring_info { + struct sk_buff *skb; + u32 len; + unsigned int bytecount; + unsigned short gso_segs; + u8 __pad[sizeof(void *) - sizeof(u32)]; +}; + +struct pci_resource { + u8 cmd; + u8 cls; + u16 io_base_h; + u16 io_base_l; + u16 mem_base_h; + u16 mem_base_l; + u8 ilr; + u16 resv_0x1c_h; + u16 resv_0x1c_l; + u16 resv_0x20_h; + u16 resv_0x20_l; + u16 resv_0x24_h; + u16 resv_0x24_l; + u16 resv_0x2c_h; + u16 resv_0x2c_l; + u32 pci_sn_l; + u32 pci_sn_h; +}; + +enum r8127_flag { + R8127_FLAG_DOWN = 0, + R8127_FLAG_TASK_RESET_PENDING, + R8127_FLAG_TASK_ESD_CHECK_PENDING, + R8127_FLAG_TASK_LINKCHG_CHECK_PENDING, + R8127_FLAG_MAX +}; + +enum r8127_sysfs_flag { + R8127_SYSFS_RTL_ADV = 0, + R8127_SYSFS_FLAG_MAX +}; + +struct rtl8127_tx_ring { + void* priv; + struct net_device *netdev; + u32 index; + u32 cur_tx; /* Index into the Tx descriptor buffer of next Rx pkt. */ + u32 dirty_tx; + u32 num_tx_desc; /* Number of Tx descriptor registers */ + struct TxDesc *TxDescArray; /* 256-aligned Tx descriptor ring */ + dma_addr_t TxPhyAddr; + u32 TxDescAllocSize; + struct ring_info tx_skb[MAX_NUM_TX_DESC]; /* Tx data buffers */ + + u32 NextHwDesCloPtr; + u32 BeginHwDesCloPtr; + + u16 hw_clo_ptr_reg; + u16 sw_tail_ptr_reg; + + u16 tdsar_reg; /* Transmit Descriptor Start Address */ +}; + +struct rtl8127_rx_buffer { + struct page *page; + u32 page_offset; + dma_addr_t dma; + void* data; + struct sk_buff *skb; +}; + +struct rtl8127_rx_ring { + void* priv; + struct net_device *netdev; + u32 index; + u32 cur_rx; /* Index into the Rx descriptor buffer of next Rx pkt. */ + u32 dirty_rx; + u32 num_rx_desc; /* Number of Rx descriptor registers */ + struct RxDesc *RxDescArray; /* 256-aligned Rx descriptor ring */ + u32 RxDescAllocSize; + u64 RxDescPhyAddr[MAX_NUM_RX_DESC]; /* Rx desc physical address*/ + dma_addr_t RxPhyAddr; +#ifdef ENABLE_PAGE_REUSE + struct rtl8127_rx_buffer rx_buffer[MAX_NUM_RX_DESC]; + u16 rx_offset; +#else + struct sk_buff *Rx_skbuff[MAX_NUM_RX_DESC]; /* Rx data buffers */ +#endif //ENABLE_PAGE_REUSE + + u16 rdsar_reg; /* Receive Descriptor Start Address */ +}; + +struct r8127_napi { +#ifdef CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,24) + struct napi_struct napi; +#endif +#endif + void* priv; + int index; +}; + +struct r8127_irq { + irq_handler_t handler; + unsigned int vector; + u8 requested; + char name[IFNAMSIZ + 10]; +}; + +#pragma pack(1) +struct rtl8127_regs { + //00 + u8 mac_id[6]; + u16 reg_06; + u8 mar[8]; + //10 + u64 dtccr; + u16 ledsel0; + u16 legreg; + u32 tctr3; + //20 + u32 txq0_dsc_st_addr_0; + u32 txq0_dsc_st_addr_2; + u64 reg_28; + //30 + u16 rit; + u16 ritc; + u16 reg_34; + u8 reg_36; + u8 command; + u32 imr0; + u32 isr0; + //40 + u32 tcr; + u32 rcr; + u32 tctr0; + u32 tctr1; + //50 + u8 cr93c46; + u8 config0; + u8 config1; + u8 config2; + u8 config3; + u8 config4; + u8 config5; + u8 tdfnr; + u32 timer_int0; + u32 timer_int1; + //60 + u32 gphy_mdcmdio; + u32 csidr; + u32 csiar; + u16 phy_status; + u8 config6; + u8 pmch; + //70 + u32 eridr; + u32 eriar; + u16 config7; + u16 reg_7a; + u32 ephy_rxerr_cnt; + //80 + u32 ephy_mdcmdio; + u16 ledsel2; + u16 ledsel1; + u32 tctr2; + u32 timer_int2; + //90 + u8 tppoll0; + u8 reg_91; + u16 reg_92; + u16 led_feature; + u16 ledsel3; + u16 eee_led_config; + u16 reg_9a; + u32 reg_9c; + //a0 + u32 reg_a0; + u32 reg_a4; + u32 reg_a8; + u32 reg_ac; + //b0 + u32 patch_dbg; + u32 reg_b4; + u32 gphy_ocp; + u32 reg_bc; + //c0 + u32 reg_c0; + u32 reg_c4; + u32 reg_c8; + u16 otp_cmd; + u16 otp_pg_config; + //d0 + u16 phy_pwr; + u8 twsi_ctrl; + u8 oob_ctrl; + u16 mac_dbgo; + u16 mac_dbg; + u16 reg_d8; + u16 rms; + u32 efuse_data; + //e0 + u16 cplus_cmd; + u16 reg_e2; + u32 rxq0_dsc_st_addr_0; + u32 rxq0_dsc_st_addr_2; + u16 reg_ec; + u16 tx10midle_cnt; + //f0 + u16 misc0; + u16 misc1; + u32 timer_int3; + u32 cmac_ib; + u16 reg_fc; + u16 sw_rst; +}; +#pragma pack() + +struct rtl8127_regs_save { + union { + u8 mac_io[R8127_MAC_REGS_SIZE]; + + struct rtl8127_regs mac_reg; + }; + u16 pcie_phy[R8127_EPHY_REGS_SIZE/2]; + u16 eth_phy[R8127_PHY_REGS_SIZE/2]; + u32 eri_reg[R8127_ERI_REGS_SIZE/4]; + u32 pci_reg[R8127_PCI_REGS_SIZE/4]; + u16 sw_tail_ptr_reg[R8127_MAX_TX_QUEUES]; + u16 hw_clo_ptr_reg[R8127_MAX_TX_QUEUES]; + + //ktime_t begin_ktime; + //ktime_t end_ktime; + //u64 duration_ns; + + u16 sw0_tail_ptr; + u16 next_hwq0_clo_ptr; + u16 sw1_tail_ptr; + u16 next_hwq1_clo_ptr; + + u16 int_miti_rxq0; + u16 int_miti_txq0; + u16 int_miti_rxq1; + u16 int_miti_txq1; + u8 int_config; + u32 imr_new; + u32 isr_new; + + u8 tdu_status; + u16 rdu_status; + + u16 tc_mode; + + u32 txq1_dsc_st_addr_0; + u32 txq1_dsc_st_addr_2; + + u32 pla_tx_q0_idle_credit; + u32 pla_tx_q1_idle_credit; + + u32 rxq1_dsc_st_addr_0; + u32 rxq1_dsc_st_addr_2; + + u32 rss_ctrl; + u8 rss_key[RTL8127_RSS_KEY_SIZE]; + u8 rss_i_table[RTL8127_MAX_INDIRECTION_TABLE_ENTRIES]; + u16 rss_queue_num_sel_r; +}; + +struct rtl8127_counters { + /* legacy */ + u64 tx_packets; + u64 rx_packets; + u64 tx_errors; + u32 rx_errors; + u16 rx_missed; + u16 align_errors; + u32 tx_one_collision; + u32 tx_multi_collision; + u64 rx_unicast; + u64 rx_broadcast; + u32 rx_multicast; + u16 tx_aborted; + u16 tx_underrun; + + /* extended */ + u64 tx_octets; + u64 rx_octets; + u64 rx_multicast64; + u64 tx_unicast64; + u64 tx_broadcast64; + u64 tx_multicast64; + u32 tx_pause_on; + u32 tx_pause_off; + u32 tx_pause_all; + u32 tx_deferred; + u32 tx_late_collision; + u32 tx_all_collision; + u32 tx_aborted32; + u32 align_errors32; + u32 rx_frame_too_long; + u32 rx_runt; + u32 rx_pause_on; + u32 rx_pause_off; + u32 rx_pause_all; + u32 rx_unknown_opcode; + u32 rx_mac_error; + u32 tx_underrun32; + u32 rx_mac_missed; + u32 rx_tcam_dropped; + u32 tdu; + u32 rdu; +}; + +/* Flow Control Settings */ +enum rtl8127_fc_mode { + rtl8127_fc_none = 0, + rtl8127_fc_rx_pause, + rtl8127_fc_tx_pause, + rtl8127_fc_full, + rtl8127_fc_default +}; + +enum rtl8127_state_t { + __RTL8127_TESTING = 0, + __RTL8127_RESETTING, + __RTL8127_DOWN, + __RTL8127_PTP_TX_IN_PROGRESS, +}; + +#define RTL_FLAG_RX_HWTSTAMP_ENABLED BIT_0 + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) +struct ethtool_eee { + __u32 cmd; + __u32 supported; + __u32 advertised; + __u32 lp_advertised; + __u32 eee_active; + __u32 eee_enabled; + __u32 tx_lpi_enabled; + __u32 tx_lpi_timer; + __u32 reserved[2]; +}; +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,6,0) */ + +struct rtl8127_private { + void __iomem *mmio_addr; /* memory map physical address */ + struct pci_dev *pci_dev; /* Index of PCI device */ + struct net_device *dev; + struct r8127_napi r8127napi[R8127_MAX_MSIX_VEC]; + struct r8127_irq irq_tbl[R8127_MAX_MSIX_VEC]; + unsigned int irq_nvecs; + unsigned int max_irq_nvecs; + unsigned int min_irq_nvecs; + unsigned int hw_supp_irq_nvecs; + //struct msix_entry msix_entries[R8127_MAX_MSIX_VEC]; + struct net_device_stats stats; /* statistics of net device */ + unsigned long state; + u32 flags; + + u32 msg_enable; + u32 tx_tcp_csum_cmd; + u32 tx_udp_csum_cmd; + u32 tx_ip_csum_cmd; + u32 tx_ipv6_csum_cmd; + int max_jumbo_frame_size; + int chipset; + u32 mcfg; + //u32 cur_rx; /* Index into the Rx descriptor buffer of next Rx pkt. */ + //u32 cur_tx; /* Index into the Tx descriptor buffer of next Rx pkt. */ + //u32 dirty_rx; + //u32 dirty_tx; + //struct TxDesc *TxDescArray; /* 256-aligned Tx descriptor ring */ + //struct RxDesc *RxDescArray; /* 256-aligned Rx descriptor ring */ + //dma_addr_t TxPhyAddr; + //dma_addr_t RxPhyAddr; + //struct sk_buff *Rx_skbuff[MAX_NUM_RX_DESC]; /* Rx data buffers */ + //struct ring_info tx_skb[MAX_NUM_TX_DESC]; /* Tx data buffers */ + unsigned rx_buf_sz; +#ifdef ENABLE_PAGE_REUSE + unsigned rx_buf_page_order; + unsigned rx_buf_page_size; + u32 page_reuse_fail_cnt; +#endif //ENABLE_PAGE_REUSE + u16 HwSuppNumTxQueues; + u16 HwSuppNumRxQueues; + unsigned int num_tx_rings; + unsigned int num_rx_rings; + struct rtl8127_tx_ring tx_ring[R8127_MAX_TX_QUEUES]; + struct rtl8127_rx_ring rx_ring[R8127_MAX_RX_QUEUES]; +#ifdef ENABLE_LIB_SUPPORT + struct blocking_notifier_head lib_nh; + struct rtl8127_ring lib_tx_ring[R8127_MAX_TX_QUEUES]; + struct rtl8127_ring lib_rx_ring[R8127_MAX_RX_QUEUES]; +#endif + //struct timer_list esd_timer; + //struct timer_list link_timer; + struct pci_resource pci_cfg_space; + unsigned int esd_flag; + unsigned int pci_cfg_is_read; + unsigned int rtl8127_rx_config; + u16 rms; + u16 cp_cmd; + u32 intr_mask; + u32 timer_intr_mask; + u16 isr_reg[R8127_MAX_MSIX_VEC]; + u16 imr_reg[R8127_MAX_MSIX_VEC]; + int phy_auto_nego_reg; + int phy_1000_ctrl_reg; + int phy_2500_ctrl_reg; + u8 org_mac_addr[NODE_ADDRESS_SIZE]; + struct rtl8127_counters *tally_vaddr; + dma_addr_t tally_paddr; + +#ifdef CONFIG_R8127_VLAN + struct vlan_group *vlgrp; +#endif + u8 wol_enabled; + u32 wol_opts; + u8 efuse_ver; + u8 eeprom_type; + u8 autoneg; + u8 duplex; + u32 speed; + u64 advertising; + enum rtl8127_fc_mode fcpause; + u32 HwSuppMaxPhyLinkSpeed; + u16 eeprom_len; + u16 cur_page; + u32 bios_setting; + + int (*set_speed)(struct net_device *, u8 autoneg, u32 speed, u8 duplex, u64 adv); +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + void (*get_settings)(struct net_device *, struct ethtool_cmd *); +#else + void (*get_settings)(struct net_device *, struct ethtool_link_ksettings *); +#endif + void (*phy_reset_enable)(struct net_device *); + unsigned int (*phy_reset_pending)(struct net_device *); + unsigned int (*link_ok)(struct net_device *); +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) + struct work_struct reset_task; + struct work_struct esd_task; + struct work_struct linkchg_task; +#else + struct delayed_work reset_task; + struct delayed_work esd_task; + struct delayed_work linkchg_task; +#endif + DECLARE_BITMAP(task_flags, R8127_FLAG_MAX); + unsigned features; + + u8 org_pci_offset_99; + u8 org_pci_offset_180; + u8 issue_offset_99_event; + + u8 org_pci_offset_80; + u8 org_pci_offset_81; + u8 use_timer_interrupt; + + u32 keep_intr_cnt; + + u8 HwIcVerUnknown; + u8 NotWrRamCodeToMicroP; + u8 NotWrMcuPatchCode; + u8 HwHasWrRamCodeToMicroP; + + u16 sw_ram_code_ver; + u16 hw_ram_code_ver; + + u8 rtk_enable_diag; + + u8 ShortPacketSwChecksum; + + u8 UseSwPaddingShortPkt; + + u8 RequireAdcBiasPatch; + u16 AdcBiasPatchIoffset; + + u8 RequireAdjustUpsTxLinkPulseTiming; + u16 SwrCnt1msIni; + + u8 HwSuppNowIsOobVer; + + u8 RequiredSecLanDonglePatch; + + u8 RequirePhyMdiSwapPatch; + + u32 HwFiberModeVer; + u32 HwFiberStat; + u8 HwSwitchMdiToFiber; + + u16 NicCustLedValue; + + u8 HwSuppMagicPktVer; + + u8 HwSuppLinkChgWakeUpVer; + + u8 HwSuppCheckPhyDisableModeVer; + + u8 random_mac; + + u16 phy_reg_aner; + u16 phy_reg_anlpar; + u16 phy_reg_gbsr; + u16 phy_reg_status_2500; + + u32 HwPcieSNOffset; + + u32 MaxTxDescPtrMask; + u8 HwSuppTxNoCloseVer; + u8 EnableTxNoClose; + + u8 HwSuppIsrVer; + u8 HwCurrIsrVer; + + u8 HwSuppIntMitiVer; + + u8 HwSuppExtendTallyCounterVer; + + u8 check_keep_link_speed; + u8 resume_not_chg_speed; + + u8 HwSuppD0SpeedUpVer; + u8 D0SpeedUpSpeed; + + u8 ring_lib_enabled; + + const char *fw_name; + struct rtl8127_fw *rtl_fw; + u32 ocp_base; + + //Dash+++++++++++++++++ + u8 HwSuppDashVer; + u8 DASH; + u8 dash_printer_enabled; + u8 HwPkgDet; + u8 AllowAccessDashOcp; + void __iomem *mapped_cmac_ioaddr; /* mapped cmac memory map physical address */ + void __iomem *cmac_ioaddr; /* cmac memory map physical address */ + +#ifdef ENABLE_DASH_SUPPORT + u16 AfterRecvFromFwBufLen; + u8 AfterRecvFromFwBuf[RECV_FROM_FW_BUF_SIZE]; + u16 AfterSendToFwBufLen; + u8 AfterSendToFwBuf[SEND_TO_FW_BUF_SIZE]; + u16 SendToFwBufferLen; + u32 SizeOfSendToFwBuffer; + u32 SizeOfSendToFwBufferMemAlloc; + u32 NumOfSendToFwBuffer; + + u8 OobReq; + u8 OobAck; + u32 OobReqComplete; + u32 OobAckComplete; + + u8 RcvFwReqSysOkEvt; + u8 RcvFwDashOkEvt; + u8 SendFwHostOkEvt; + + u8 DashFwDisableRx; + + void *UnalignedSendToFwBufferVa; + void *SendToFwBuffer; + u64 SendToFwBufferPhy; + u8 SendingToFw; + dma_addr_t UnalignedSendToFwBufferPa; + PTX_DASH_SEND_FW_DESC TxDashSendFwDesc; + u64 TxDashSendFwDescPhy; + u8 *UnalignedTxDashSendFwDescVa; + u32 SizeOfTxDashSendFwDescMemAlloc; + u32 SizeOfTxDashSendFwDesc; + u32 NumTxDashSendFwDesc; + u32 CurrNumTxDashSendFwDesc; + u32 LastSendNumTxDashSendFwDesc; + dma_addr_t UnalignedTxDashSendFwDescPa; + + u32 NumRecvFromFwBuffer; + u32 SizeOfRecvFromFwBuffer; + u32 SizeOfRecvFromFwBufferMemAlloc; + void *RecvFromFwBuffer; + u64 RecvFromFwBufferPhy; + + void *UnalignedRecvFromFwBufferVa; + dma_addr_t UnalignedRecvFromFwBufferPa; + PRX_DASH_FROM_FW_DESC RxDashRecvFwDesc; + u64 RxDashRecvFwDescPhy; + u8 *UnalignedRxDashRecvFwDescVa; + u32 SizeOfRxDashRecvFwDescMemAlloc; + u32 SizeOfRxDashRecvFwDesc; + u32 NumRxDashRecvFwDesc; + u32 CurrNumRxDashRecvFwDesc; + dma_addr_t UnalignedRxDashRecvFwDescPa; + u8 DashReqRegValue; + u16 HostReqValue; + + u32 CmacResetIsrCounter; + u8 CmacResetIntr; + u8 CmacResetting; + u8 CmacOobIssueCmacReset; + u32 CmacResetbyFwCnt; + +#if defined(ENABLE_DASH_PRINTER_SUPPORT) + struct completion fw_ack; + struct completion fw_req; + struct completion fw_host_ok; +#endif + //Dash----------------- +#endif //ENABLE_DASH_SUPPORT + + //Realwow++++++++++++++ + u8 HwSuppKCPOffloadVer; + + u8 EnableDhcpTimeoutWake; + u8 EnableTeredoOffload; + u8 EnableKCPOffload; +#ifdef ENABLE_REALWOW_SUPPORT + u32 DhcpTimeout; + MP_KCP_INFO MpKCPInfo; + //Realwow-------------- +#endif //ENABLE_REALWOW_SUPPORT + + struct ethtool_keee eee; + +#ifdef ENABLE_R8127_PROCFS + //Procfs support + struct proc_dir_entry *proc_dir; + struct proc_dir_entry *proc_dir_debug; + struct proc_dir_entry *proc_dir_test; +#endif +#ifdef ENABLE_R8127_SYSFS + //sysfs support + DECLARE_BITMAP(sysfs_flag, R8127_SYSFS_FLAG_MAX); + u32 testmode; +#endif + u8 HwSuppRxDescType; + u8 InitRxDescType; + u16 RxDescLength; //V1 16 Byte V2 32 Bytes + + spinlock_t phy_lock; + + u8 HwSuppPtpVer; + u8 EnablePtp; +#ifdef ENABLE_PTP_SUPPORT + u32 tx_hwtstamp_timeouts; + u32 tx_hwtstamp_skipped; + struct work_struct ptp_tx_work; + struct sk_buff *ptp_tx_skb; + struct hwtstamp_config hwtstamp_config; + unsigned long ptp_tx_start; + struct ptp_clock_info ptp_clock_info; + struct ptp_clock *ptp_clock; + u8 syncE_en; + u8 pps_enable; + struct hrtimer pps_timer; +#endif + + u8 HwSuppRssVer; + u8 EnableRss; + u16 HwSuppIndirTblEntries; +#ifdef ENABLE_RSS_SUPPORT + u32 rss_flags; + /* Receive Side Scaling settings */ + u8 rss_key[RTL8127_RSS_KEY_SIZE]; + u8 rss_indir_tbl[RTL8127_MAX_INDIRECTION_TABLE_ENTRIES]; + u32 rss_options; +#endif + + u8 HwSuppMacMcuVer; + u16 MacMcuPageSize; + u64 hw_mcu_patch_code_ver; + u64 bin_mcu_patch_code_ver; + + u8 HwSuppTcamVer; + + u16 TcamNotValidReg; + u16 TcamValidReg; + u16 TcamMaAddrcOffset; + u16 TcamVlanTagOffset; +}; + +#ifdef ENABLE_LIB_SUPPORT +static inline unsigned int +rtl8127_num_lib_tx_rings(struct rtl8127_private *tp) +{ + int count, i; + + for (count = 0, i = tp->num_tx_rings; i < tp->HwSuppNumTxQueues; i++) + if(tp->lib_tx_ring[i].enabled) + count++; + + return count; +} + +static inline unsigned int +rtl8127_num_lib_rx_rings(struct rtl8127_private *tp) +{ + int count, i; + + for (count = 0, i = tp->num_rx_rings; i < tp->HwSuppNumRxQueues; i++) + if(tp->lib_rx_ring[i].enabled) + count++; + + return count; +} + +#else +static inline unsigned int +rtl8127_num_lib_tx_rings(struct rtl8127_private *tp) +{ + return 0; +} + +static inline unsigned int +rtl8127_num_lib_rx_rings(struct rtl8127_private *tp) +{ + return 0; +} +#endif + +static inline unsigned int +rtl8127_tot_tx_rings(struct rtl8127_private *tp) +{ + return tp->num_tx_rings + rtl8127_num_lib_tx_rings(tp); +} + +static inline unsigned int +rtl8127_tot_rx_rings(struct rtl8127_private *tp) +{ + return tp->num_rx_rings + rtl8127_num_lib_rx_rings(tp); +} + +static inline struct netdev_queue *txring_txq(const struct rtl8127_tx_ring *ring) +{ + return netdev_get_tx_queue(ring->netdev, ring->index); +} + +enum eetype { + EEPROM_TYPE_NONE=0, + EEPROM_TYPE_93C46, + EEPROM_TYPE_93C56, + EEPROM_TWSI +}; + +enum mcfg { + CFG_METHOD_1=1, + CFG_METHOD_2, + CFG_METHOD_DEFAULT, + CFG_METHOD_MAX +}; + +#define LSO_32K 32000 +#define LSO_64K 64000 + +#define NIC_MIN_PHYS_BUF_COUNT (2) +#define NIC_MAX_PHYS_BUF_COUNT_LSO_64K (24) +#define NIC_MAX_PHYS_BUF_COUNT_LSO2 (16*4) + +#define GTTCPHO_SHIFT 18 +#define GTTCPHO_MAX 0x70U +#define GTPKTSIZE_MAX 0x3ffffU +#define TCPHO_SHIFT 18 +#define TCPHO_MAX 0x3ffU +#define LSOPKTSIZE_MAX 0xffffU +#define MSS_MAX 0x07ffu /* MSS value */ + +#define OOB_CMD_RESET 0x00 +#define OOB_CMD_DRIVER_START 0x05 +#define OOB_CMD_DRIVER_STOP 0x06 +#define OOB_CMD_SET_IPMAC 0x41 + +#define WAKEUP_MAGIC_PACKET_NOT_SUPPORT (0) +#define WAKEUP_MAGIC_PACKET_V1 (1) +#define WAKEUP_MAGIC_PACKET_V2 (2) +#define WAKEUP_MAGIC_PACKET_V3 (3) + +//Ram Code Version +#define NIC_RAMCODE_VERSION_CFG_METHOD_1 (0x0015) +#define NIC_RAMCODE_VERSION_CFG_METHOD_2 (0x0015) + +//hwoptimize +#define HW_PATCH_SOC_LAN (BIT_0) +#define HW_PATCH_SAMSUNG_LAN_DONGLE (BIT_2) + +static const u16 other_q_intr_mask = (RxOK1 | RxDU1); + +void rtl8127_mdio_write(struct rtl8127_private *tp, u16 RegAddr, u16 value); +void rtl8127_mdio_prot_write(struct rtl8127_private *tp, u32 RegAddr, u32 value); +void rtl8127_mdio_prot_direct_write_phy_ocp(struct rtl8127_private *tp, u32 RegAddr, u32 value); +u32 rtl8127_mdio_read(struct rtl8127_private *tp, u16 RegAddr); +u32 rtl8127_mdio_prot_read(struct rtl8127_private *tp, u32 RegAddr); +u32 rtl8127_mdio_prot_direct_read_phy_ocp(struct rtl8127_private *tp, u32 RegAddr); +void rtl8127_ephy_write(struct rtl8127_private *tp, int RegAddr, int value); +void rtl8127_mac_ocp_write(struct rtl8127_private *tp, u16 reg_addr, u16 value); +u16 rtl8127_mac_ocp_read(struct rtl8127_private *tp, u16 reg_addr); +void rtl8127_clear_eth_phy_bit(struct rtl8127_private *tp, u8 addr, u16 mask); +void rtl8127_set_eth_phy_bit(struct rtl8127_private *tp, u8 addr, u16 mask); +void rtl8127_ocp_write(struct rtl8127_private *tp, u16 addr, u8 len, u32 data); +void rtl8127_oob_notify(struct rtl8127_private *tp, u8 cmd); +void rtl8127_init_ring_indexes(struct rtl8127_private *tp); +void rtl8127_oob_mutex_lock(struct rtl8127_private *tp); +u32 rtl8127_ocp_read(struct rtl8127_private *tp, u16 addr, u8 len); +u32 rtl8127_ocp_read_with_oob_base_address(struct rtl8127_private *tp, u16 addr, u8 len, u32 base_address); +u32 rtl8127_ocp_write_with_oob_base_address(struct rtl8127_private *tp, u16 addr, u8 len, u32 value, u32 base_address); +u32 rtl8127_eri_read(struct rtl8127_private *tp, int addr, int len, int type); +u32 rtl8127_eri_read_with_oob_base_address(struct rtl8127_private *tp, int addr, int len, int type, u32 base_address); +int rtl8127_eri_write(struct rtl8127_private *tp, int addr, int len, u32 value, int type); +int rtl8127_eri_write_with_oob_base_address(struct rtl8127_private *tp, int addr, int len, u32 value, int type, u32 base_address); +u16 rtl8127_ephy_read(struct rtl8127_private *tp, int RegAddr); +void rtl8127_wait_txrx_fifo_empty(struct net_device *dev); +void rtl8127_enable_now_is_oob(struct rtl8127_private *tp); +void rtl8127_disable_now_is_oob(struct rtl8127_private *tp); +void rtl8127_oob_mutex_unlock(struct rtl8127_private *tp); +void rtl8127_dash2_disable_tx(struct rtl8127_private *tp); +void rtl8127_dash2_enable_tx(struct rtl8127_private *tp); +void rtl8127_dash2_disable_rx(struct rtl8127_private *tp); +void rtl8127_dash2_enable_rx(struct rtl8127_private *tp); +void rtl8127_hw_disable_mac_mcu_bps(struct net_device *dev); +void rtl8127_mark_to_asic(struct rtl8127_private *tp, struct RxDesc *desc, u32 rx_buf_sz); +void rtl8127_mark_as_last_descriptor(struct rtl8127_private *tp, struct RxDesc *desc); + +static inline void +rtl8127_make_unusable_by_asic(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + ((struct RxDescV3 *)desc)->addr = RTL8127_MAGIC_NUMBER; + ((struct RxDescV3 *)desc)->RxDescNormalDDWord4.opts1 &= ~cpu_to_le32(DescOwn | RsvdMaskV3); + break; + case RX_DESC_RING_TYPE_4: + ((struct RxDescV4 *)desc)->addr = RTL8127_MAGIC_NUMBER; + ((struct RxDescV4 *)desc)->RxDescNormalDDWord2.opts1 &= ~cpu_to_le32(DescOwn | RsvdMaskV4); + break; + default: + desc->addr = RTL8127_MAGIC_NUMBER; + desc->opts1 &= ~cpu_to_le32(DescOwn | RsvdMask); + break; + } +} + +static inline struct RxDesc* +rtl8127_get_rxdesc(struct rtl8127_private *tp, struct RxDesc *RxDescBase, u32 const cur_rx) +{ + return (struct RxDesc*)((u8*)RxDescBase + (cur_rx * tp->RxDescLength)); +} + +static inline void +rtl8127_disable_hw_interrupt_v2(struct rtl8127_private *tp, + u32 message_id) +{ + RTL_W32(tp, IMR_V2_CLEAR_REG_8125, BIT(message_id)); +} + +static inline void +rtl8127_enable_hw_interrupt_v2(struct rtl8127_private *tp, u32 message_id) +{ + RTL_W32(tp, IMR_V2_SET_REG_8125, BIT(message_id)); +} + +int rtl8127_open(struct net_device *dev); +int rtl8127_close(struct net_device *dev); +void rtl8127_hw_config(struct net_device *dev); +void rtl8127_hw_set_timer_int(struct rtl8127_private *tp, u32 message_id, u8 timer_intmiti_val); +void rtl8127_set_rx_q_num(struct rtl8127_private *tp, unsigned int num_rx_queues); +void rtl8127_set_tx_q_num(struct rtl8127_private *tp, unsigned int num_tx_queues); +void rtl8127_enable_mcu(struct rtl8127_private *tp, bool enable); +void rtl8127_hw_start(struct net_device *dev); +void rtl8127_hw_reset(struct net_device *dev); +void rtl8127_tx_clear(struct rtl8127_private *tp); +void rtl8127_rx_clear(struct rtl8127_private *tp); +int rtl8127_init_ring(struct net_device *dev); +void rtl8127_hw_set_rx_packet_filter(struct net_device *dev); +void rtl8127_enable_hw_linkchg_interrupt(struct rtl8127_private *tp); +int rtl8127_dump_tally_counter(struct rtl8127_private *tp, dma_addr_t paddr); +void rtl8127_enable_napi(struct rtl8127_private *tp); +void _rtl8127_wait_for_quiescence(struct net_device *dev); + +void rtl8127_mdio_direct_write_phy_ocp(struct rtl8127_private *tp, u16 RegAddr,u16 value); +u32 rtl8127_mdio_direct_read_phy_ocp(struct rtl8127_private *tp, u16 RegAddr); +void rtl8127_clear_and_set_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 clearmask, u16 setmask); +void rtl8127_clear_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 mask); +void rtl8127_set_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 mask); + +void rtl8127_clear_mac_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 mask); + +#ifndef ENABLE_LIB_SUPPORT +static inline void rtl8127_lib_reset_prepare(struct rtl8127_private *tp) { } +static inline void rtl8127_lib_reset_complete(struct rtl8127_private *tp) { } +#endif + +#define HW_SUPPORT_CHECK_PHY_DISABLE_MODE(_M) ((_M)->HwSuppCheckPhyDisableModeVer > 0) +#define HW_HAS_WRITE_PHY_MCU_RAM_CODE(_M) (((_M)->HwHasWrRamCodeToMicroP == TRUE) ? 1 : 0) +#define HW_SUPPORT_D0_SPEED_UP(_M) ((_M)->HwSuppD0SpeedUpVer > 0) +#define HW_SUPPORT_MAC_MCU(_M) ((_M)->HwSuppMacMcuVer > 0) +#define HW_SUPPORT_TCAM(_M) ((_M)->HwSuppTcamVer > 0) + +#define HW_SUPP_PHY_LINK_SPEED_GIGA(_M) ((_M)->HwSuppMaxPhyLinkSpeed >= 1000) +#define HW_SUPP_PHY_LINK_SPEED_2500M(_M) ((_M)->HwSuppMaxPhyLinkSpeed >= 2500) +#define HW_SUPP_PHY_LINK_SPEED_5000M(_M) ((_M)->HwSuppMaxPhyLinkSpeed >= 5000) +#define HW_SUPP_PHY_LINK_SPEED_10000M(_M) ((_M)->HwSuppMaxPhyLinkSpeed >= 10000) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,34) +#define netdev_mc_count(dev) ((dev)->mc_count) +#define netdev_mc_empty(dev) (netdev_mc_count(dev) == 0) +#define netdev_for_each_mc_addr(mclist, dev) \ + for (mclist = dev->mc_list; mclist; mclist = mclist->next) +#endif + +#endif /* __R8127_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_dash.h b/drivers/net/ethernet/realtek/r8127/src/r8127_dash.h new file mode 100755 index 0000000000000..0f6a3d150a005 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_dash.h @@ -0,0 +1,261 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_R8127_DASH_H +#define _LINUX_R8127_DASH_H + +#include + +#define SIOCDEVPRIVATE_RTLDASH SIOCDEVPRIVATE+2 + +enum rtl_dash_cmd { + RTL_DASH_ARP_NS_OFFLOAD = 0, + RTL_DASH_SET_OOB_IPMAC, + RTL_DASH_NOTIFY_OOB, + + RTL_DASH_SEND_BUFFER_DATA_TO_DASH_FW, + RTL_DASH_CHECK_SEND_BUFFER_TO_DASH_FW_COMPLETE, + RTL_DASH_GET_RCV_FROM_FW_BUFFER_DATA, + RTL_DASH_OOB_REQ, + RTL_DASH_OOB_ACK, + RTL_DASH_DETACH_OOB_REQ, + RTL_DASH_DETACH_OOB_ACK, + + RTL_FW_SET_IPV4 = 0x10, + RTL_FW_GET_IPV4, + RTL_FW_SET_IPV6, + RTL_FW_GET_IPV6, + RTL_FW_SET_EXT_SNMP, + RTL_FW_GET_EXT_SNMP, + RTL_FW_SET_WAKEUP_PATTERN, + RTL_FW_GET_WAKEUP_PATTERN, + RTL_FW_DEL_WAKEUP_PATTERN, + + RTLT_DASH_COMMAND_INVALID, +}; + +struct rtl_dash_ip_mac { + struct sockaddr ifru_addr; + struct sockaddr ifru_netmask; + struct sockaddr ifru_hwaddr; +}; + +struct rtl_dash_ioctl_struct { + __u32 cmd; + __u32 offset; + __u32 len; + union { + __u32 data; + void *data_buffer; + }; +}; + +struct settings_ipv4 { + __u32 IPv4addr; + __u32 IPv4mask; + __u32 IPv4Gateway; +}; + +struct settings_ipv6 { + __u32 reserved; + __u32 prefixLen; + __u16 IPv6addr[8]; + __u16 IPv6Gateway[8]; +}; + +struct settings_ext_snmp { + __u16 index; + __u16 oid_get_len; + __u8 oid_for_get[24]; + __u8 reserved0[26]; + __u16 value_len; + __u8 value[256]; + __u8 supported; + __u8 reserved1[27]; +}; + +struct wakeup_pattern { + __u8 index; + __u8 valid; + __u8 start; + __u8 length; + __u8 name[36]; + __u8 mask[16]; + __u8 pattern[128]; + __u32 reserved[2]; +}; + +typedef struct _RX_DASH_FROM_FW_DESC { + u16 length; + u8 statusLowByte; + u8 statusHighByte; + u32 resv; + u64 BufferAddress; +} +RX_DASH_FROM_FW_DESC, *PRX_DASH_FROM_FW_DESC; + +typedef struct _TX_DASH_SEND_FW_DESC { + u16 length; + u8 statusLowByte; + u8 statusHighByte; + u32 resv; + u64 BufferAddress; +} +TX_DASH_SEND_FW_DESC, *PTX_DASH_SEND_FW_DESC; + +typedef struct _OSOOBHdr { + u32 len; + u8 type; + u8 flag; + u8 hostReqV; + u8 res; +} +OSOOBHdr, *POSOOBHdr; + +typedef struct _RX_DASH_BUFFER_TYPE_2 { + OSOOBHdr oobhdr; + u8 RxDataBuffer[0]; +} +RX_DASH_BUFFER_TYPE_2, *PRX_DASH_BUFFER_TYPE_2; + +#define ALIGN_8 (0x7) +#define ALIGN_16 (0xf) +#define ALIGN_32 (0x1f) +#define ALIGN_64 (0x3f) +#define ALIGN_256 (0xff) +#define ALIGN_4096 (0xfff) + +#define OCP_REG_CONFIG0 (0x10) +#define OCP_REG_CONFIG0_REV_F (0xB8) +#define OCP_REG_DASH_POLL (0x30) +#define OCP_REG_HOST_REQ (0x34) +#define OCP_REG_DASH_REQ (0x35) +#define OCP_REG_CR (0x36) +#define OCP_REG_DMEMSTA (0x38) +#define OCP_REG_GPHYAR (0x60) + + +#define OCP_REG_CONFIG0_DASHEN BIT_15 +#define OCP_REG_CONFIG0_OOBRESET BIT_14 +#define OCP_REG_CONFIG0_APRDY BIT_13 +#define OCP_REG_CONFIG0_FIRMWARERDY BIT_12 +#define OCP_REG_CONFIG0_DRIVERRDY BIT_11 +#define OCP_REG_CONFIG0_OOB_WDT BIT_9 +#define OCP_REG_CONFIG0_DRV_WAIT_OOB BIT_8 +#define OCP_REG_CONFIG0_TLSEN BIT_7 + +#define HW_DASH_SUPPORT_DASH(_M) ((_M)->HwSuppDashVer > 0) +#define HW_DASH_SUPPORT_TYPE_1(_M) ((_M)->HwSuppDashVer == 1) +#define HW_DASH_SUPPORT_TYPE_2(_M) ((_M)->HwSuppDashVer == 2) +#define HW_DASH_SUPPORT_TYPE_3(_M) ((_M)->HwSuppDashVer == 3) + +#define RECV_FROM_FW_BUF_SIZE (1520) +#define SEND_TO_FW_BUF_SIZE (1520) + +#define RX_DASH_FROM_FW_OWN BIT_15 +#define TX_DASH_SEND_FW_OWN BIT_15 +#define TX_DASH_SEND_FW_OWN_HIGHBYTE BIT_7 + +#define TXS_CC3_0 (BIT_0|BIT_1|BIT_2|BIT_3) +#define TXS_EXC BIT_4 +#define TXS_LNKF BIT_5 +#define TXS_OWC BIT_6 +#define TXS_TES BIT_7 +#define TXS_UNF BIT_9 +#define TXS_LGSEN BIT_11 +#define TXS_LS BIT_12 +#define TXS_FS BIT_13 +#define TXS_EOR BIT_14 +#define TXS_OWN BIT_15 + +#define TPPool_HRDY 0x20 + +#define HostReqReg (0xC0) +#define SystemMasterDescStartAddrLow (0xF0) +#define SystemMasterDescStartAddrHigh (0xF4) +#define SystemSlaveDescStartAddrLow (0xF8) +#define SystemSlaveDescStartAddrHigh (0xFC) + +//DASH Request Type +#define WSMANREG 0x01 +#define OSPUSHDATA 0x02 + +#define RXS_OWN BIT_15 +#define RXS_EOR BIT_14 +#define RXS_FS BIT_13 +#define RXS_LS BIT_12 + +#define ISRIMR_DP_DASH_OK BIT_15 +#define ISRIMR_DP_HOST_OK BIT_13 +#define ISRIMR_DP_REQSYS_OK BIT_11 + +#define ISRIMR_DASH_INTR_EN BIT_12 +#define ISRIMR_DASH_INTR_CMAC_RESET BIT_15 + +#define ISRIMR_DASH_TYPE2_ROK BIT_0 +#define ISRIMR_DASH_TYPE2_RDU BIT_1 +#define ISRIMR_DASH_TYPE2_TOK BIT_2 +#define ISRIMR_DASH_TYPE2_TDU BIT_3 +#define ISRIMR_DASH_TYPE2_TX_FIFO_FULL BIT_4 +#define ISRIMR_DASH_TYPE2_TX_DISABLE_IDLE BIT_5 +#define ISRIMR_DASH_TYPE2_RX_DISABLE_IDLE BIT_6 + +#define CMAC_OOB_STOP 0x25 +#define CMAC_OOB_INIT 0x26 +#define CMAC_OOB_RESET 0x2a + +#define NO_BASE_ADDRESS 0x00000000 +#define RTL8168FP_OOBMAC_BASE 0xBAF70000 +#define RTL8168FP_CMAC_IOBASE 0xBAF20000 +#define RTL8168FP_KVM_BASE 0xBAF80400 +#define CMAC_SYNC_REG 0x20 +#define CMAC_RXDESC_OFFSET 0x90 //RX: 0x90 - 0x98 +#define CMAC_TXDESC_OFFSET 0x98 //TX: 0x98 - 0x9F + +/* cmac write/read MMIO register */ +#define RTL_CMAC_W8(tp, reg, val8) writeb ((val8), tp->cmac_ioaddr + (reg)) +#define RTL_CMAC_W16(tp, reg, val16) writew ((val16), tp->cmac_ioaddr + (reg)) +#define RTL_CMAC_W32(tp, reg, val32) writel ((val32), tp->cmac_ioaddr + (reg)) +#define RTL_CMAC_R8(tp, reg) readb (tp->cmac_ioaddr + (reg)) +#define RTL_CMAC_R16(tp, reg) readw (tp->cmac_ioaddr + (reg)) +#define RTL_CMAC_R32(tp, reg) ((unsigned long) readl (tp->cmac_ioaddr + (reg))) + +int rtl8127_dash_ioctl(struct net_device *dev, struct ifreq *ifr); +void HandleDashInterrupt(struct net_device *dev); +int AllocateDashShareMemory(struct net_device *dev); +void FreeAllocatedDashShareMemory(struct net_device *dev); +void DashHwInit(struct net_device *dev); + + +#endif /* _LINUX_R8127_DASH_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c b/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c new file mode 100755 index 0000000000000..7ab59f641e77a --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#include +#include +#include + +#include "r8127_firmware.h" + +enum rtl_fw_opcode { + PHY_READ = 0x0, + PHY_DATA_OR = 0x1, + PHY_DATA_AND = 0x2, + PHY_BJMPN = 0x3, + PHY_MDIO_CHG = 0x4, + PHY_CLEAR_READCOUNT = 0x7, + PHY_WRITE = 0x8, + PHY_READCOUNT_EQ_SKIP = 0x9, + PHY_COMP_EQ_SKIPN = 0xa, + PHY_COMP_NEQ_SKIPN = 0xb, + PHY_WRITE_PREVIOUS = 0xc, + PHY_SKIPN = 0xd, + PHY_DELAY_MS = 0xe, +}; + +struct fw_info { + u32 magic; + char version[RTL8127_VER_SIZE]; + __le32 fw_start; + __le32 fw_len; + u8 chksum; +} __packed; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,16,0) +#define sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER)) +#endif +#define FW_OPCODE_SIZE sizeof_field(struct rtl8127_fw_phy_action, code[0]) + +static bool rtl8127_fw_format_ok(struct rtl8127_fw *rtl_fw) +{ + const struct firmware *fw = rtl_fw->fw; + struct fw_info *fw_info = (struct fw_info *)fw->data; + struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; + + if (fw->size < FW_OPCODE_SIZE) + return false; + + if (!fw_info->magic) { + size_t i, size, start; + u8 checksum = 0; + + if (fw->size < sizeof(*fw_info)) + return false; + + for (i = 0; i < fw->size; i++) + checksum += fw->data[i]; + if (checksum != 0) + return false; + + start = le32_to_cpu(fw_info->fw_start); + if (start > fw->size) + return false; + + size = le32_to_cpu(fw_info->fw_len); + if (size > (fw->size - start) / FW_OPCODE_SIZE) + return false; + + strscpy(rtl_fw->version, fw_info->version, RTL8127_VER_SIZE); + + pa->code = (__le32 *)(fw->data + start); + pa->size = size; + } else { + if (fw->size % FW_OPCODE_SIZE) + return false; + + strscpy(rtl_fw->version, rtl_fw->fw_name, RTL8127_VER_SIZE); + + pa->code = (__le32 *)fw->data; + pa->size = fw->size / FW_OPCODE_SIZE; + } + + return true; +} + +static bool rtl8127_fw_data_ok(struct rtl8127_fw *rtl_fw) +{ + struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; + size_t index; + + for (index = 0; index < pa->size; index++) { + u32 action = le32_to_cpu(pa->code[index]); + u32 val = action & 0x0000ffff; + u32 regno = (action & 0x0fff0000) >> 16; + + switch (action >> 28) { + case PHY_READ: + case PHY_DATA_OR: + case PHY_DATA_AND: + case PHY_CLEAR_READCOUNT: + case PHY_WRITE: + case PHY_WRITE_PREVIOUS: + case PHY_DELAY_MS: + break; + + case PHY_MDIO_CHG: + if (val > 1) + goto out; + break; + + case PHY_BJMPN: + if (regno > index) + goto out; + break; + case PHY_READCOUNT_EQ_SKIP: + if (index + 2 >= pa->size) + goto out; + break; + case PHY_COMP_EQ_SKIPN: + case PHY_COMP_NEQ_SKIPN: + case PHY_SKIPN: + if (index + 1 + regno >= pa->size) + goto out; + break; + + default: + dev_err(rtl_fw->dev, "Invalid action 0x%08x\n", action); + return false; + } + } + + return true; +out: + dev_err(rtl_fw->dev, "Out of range of firmware\n"); + return false; +} + +void rtl8127_fw_write_firmware(struct rtl8127_private *tp, struct rtl8127_fw *rtl_fw) +{ + struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; + rtl8127_fw_write_t fw_write = rtl_fw->phy_write; + rtl8127_fw_read_t fw_read = rtl_fw->phy_read; + int predata = 0, count = 0; + size_t index; + + for (index = 0; index < pa->size; index++) { + u32 action = le32_to_cpu(pa->code[index]); + u32 data = action & 0x0000ffff; + u32 regno = (action & 0x0fff0000) >> 16; + enum rtl_fw_opcode opcode = action >> 28; + + if (!action) + break; + + switch (opcode) { + case PHY_READ: + predata = fw_read(tp, regno); + count++; + break; + case PHY_DATA_OR: + predata |= data; + break; + case PHY_DATA_AND: + predata &= data; + break; + case PHY_BJMPN: + index -= (regno + 1); + break; + case PHY_MDIO_CHG: + if (data) { + fw_write = rtl_fw->mac_mcu_write; + fw_read = rtl_fw->mac_mcu_read; + } else { + fw_write = rtl_fw->phy_write; + fw_read = rtl_fw->phy_read; + } + + break; + case PHY_CLEAR_READCOUNT: + count = 0; + break; + case PHY_WRITE: + fw_write(tp, regno, data); + break; + case PHY_READCOUNT_EQ_SKIP: + if (count == data) + index++; + break; + case PHY_COMP_EQ_SKIPN: + if (predata == data) + index += regno; + break; + case PHY_COMP_NEQ_SKIPN: + if (predata != data) + index += regno; + break; + case PHY_WRITE_PREVIOUS: + fw_write(tp, regno, predata); + break; + case PHY_SKIPN: + index += regno; + break; + case PHY_DELAY_MS: + mdelay(data); + break; + } + } +} + +void rtl8127_fw_release_firmware(struct rtl8127_fw *rtl_fw) +{ + release_firmware(rtl_fw->fw); +} + +int rtl8127_fw_request_firmware(struct rtl8127_fw *rtl_fw) +{ + int rc; + + rc = request_firmware(&rtl_fw->fw, rtl_fw->fw_name, rtl_fw->dev); + if (rc < 0) + goto out; + + if (!rtl8127_fw_format_ok(rtl_fw) || !rtl8127_fw_data_ok(rtl_fw)) { + release_firmware(rtl_fw->fw); + rc = -EINVAL; + goto out; + } + + return 0; +out: + dev_err(rtl_fw->dev, "Unable to load firmware %s (%d)\n", + rtl_fw->fw_name, rc); + return rc; +} diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h b/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h new file mode 100755 index 0000000000000..6b1acea98a3f8 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_R8127_FIRMWARE_H +#define _LINUX_R8127_FIRMWARE_H + +#include +#include + +struct rtl8127_private; +typedef void (*rtl8127_fw_write_t)(struct rtl8127_private *tp, u16 reg, u16 val); +typedef u32 (*rtl8127_fw_read_t)(struct rtl8127_private *tp, u16 reg); + +#define RTL8127_VER_SIZE 32 + +struct rtl8127_fw { + rtl8127_fw_write_t phy_write; + rtl8127_fw_read_t phy_read; + rtl8127_fw_write_t mac_mcu_write; + rtl8127_fw_read_t mac_mcu_read; + const struct firmware *fw; + const char *fw_name; + struct device *dev; + + char version[RTL8127_VER_SIZE]; + + struct rtl8127_fw_phy_action { + __le32 *code; + size_t size; + } phy_action; +}; + +int rtl8127_fw_request_firmware(struct rtl8127_fw *rtl_fw); +void rtl8127_fw_release_firmware(struct rtl8127_fw *rtl_fw); +void rtl8127_fw_write_firmware(struct rtl8127_private *tp, struct rtl8127_fw *rtl_fw); + +#endif /* _LINUX_R8127_FIRMWARE_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_n.c b/drivers/net/ethernet/realtek/r8127/src/r8127_n.c new file mode 100755 index 0000000000000..4f83e44869deb --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_n.c @@ -0,0 +1,17824 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +/* + * This driver is modified from r8169.c in Linux kernel 2.6.18 + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22) +#include +#include +#endif +#include +#include +#include +#include + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,26) +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,4,0) +#include +#endif +#endif +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,4,37) +#include +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) +#define dev_printk(A,B,fmt,args...) printk(A fmt,##args) +#else +#include +#include +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,31) +#include +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,4,10) +#include +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,4,10) */ + +#include +#include + +#include "r8127.h" +#include "rtl_eeprom.h" +#include "rtltool.h" +#include "r8127_firmware.h" + +#ifdef ENABLE_R8127_PROCFS +#include +#include +#endif + +#define FIRMWARE_8127_1 "rtl_nic/rtl8127-1.fw" +#define FIRMWARE_8127_2 "rtl_nic/rtl8127-2.fw" + +static const struct { + const char *name; + const char *fw_name; +} rtl_chip_fw_infos[] = { + /* PCI-E devices. */ + [CFG_METHOD_1] = {"RTL8127", FIRMWARE_8127_1}, + [CFG_METHOD_2] = {"RTL8127", FIRMWARE_8127_2}, + [CFG_METHOD_DEFAULT] = {"Unknown", }, +}; + +#define _R(NAME,MAC,RCR,MASK,JumFrameSz) \ + { .name = NAME, .mcfg = MAC, .RCR_Cfg = RCR, .RxConfigMask = MASK, .jumbo_frame_sz = JumFrameSz } + +static const struct { + const char *name; + u8 mcfg; + u32 RCR_Cfg; + u32 RxConfigMask; /* Clears the bits supported by this chip */ + u32 jumbo_frame_sz; +} rtl_chip_info[] = { + _R("RTL8127", + CFG_METHOD_1, + Rx_Fetch_Number_8 | Rx_Close_Multiple | RxCfg_pause_slot_en | EnableInnerVlan | EnableOuterVlan | (RX_DMA_BURST_512 << RxCfgDMAShift), + 0xff7e5880, + Jumbo_Frame_9k), + + _R("RTL8127", + CFG_METHOD_2, + Rx_Fetch_Number_8 | Rx_Close_Multiple | RxCfg_pause_slot_en | EnableInnerVlan | EnableOuterVlan | (RX_DMA_BURST_512 << RxCfgDMAShift), + 0xff7e5880, + Jumbo_Frame_9k), + + _R("Unknown", + CFG_METHOD_DEFAULT, + (RX_DMA_BURST_512 << RxCfgDMAShift), + 0xff7e5880, + Jumbo_Frame_1k) +}; +#undef _R + + +static struct pci_device_id rtl8127_pci_tbl[] = { + { PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8127), }, + { PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x0E10), }, + {0,}, +}; + +MODULE_DEVICE_TABLE(pci, rtl8127_pci_tbl); + +static int use_dac = 1; +static int timer_count = 0x2600; +static int timer_count_v2 = (0x2600 / 0x200); + +static struct { + u32 msg_enable; +} debug = { -1 }; + +static unsigned int speed_mode = SPEED_10000; +static unsigned int duplex_mode = DUPLEX_FULL; +static unsigned int autoneg_mode = AUTONEG_ENABLE; +#ifdef CONFIG_ASPM +static int aspm = 1; +#else +static int aspm = 0; +#endif +#ifdef ENABLE_S5WOL +static int s5wol = 1; +#else +static int s5wol = 0; +#endif +#ifdef ENABLE_S5_KEEP_CURR_MAC +static int s5_keep_curr_mac = 1; +#else +static int s5_keep_curr_mac = 0; +#endif +#ifdef ENABLE_EEE +static int eee_enable = 1; +#else +static int eee_enable = 0; +#endif +#ifdef CONFIG_SOC_LAN +static ulong hwoptimize = HW_PATCH_SOC_LAN; +#else +static ulong hwoptimize = 0; +#endif +#ifdef ENABLE_S0_MAGIC_PACKET +static int s0_magic_packet = 1; +#else +static int s0_magic_packet = 0; +#endif +#ifdef ENABLE_TX_NO_CLOSE +static int tx_no_close_enable = 1; +#else +static int tx_no_close_enable = 0; +#endif +#ifdef DISABLE_WOL_SUPPORT +static int disable_wol_support = 1; +#else +static int disable_wol_support = 0; +#endif +#ifdef ENABLE_DOUBLE_VLAN +static int enable_double_vlan = 1; +#else +static int enable_double_vlan = 0; +#endif +#ifdef ENABLE_GIGA_LITE +static int eee_giga_lite = 1; +#else +static int eee_giga_lite = 0; +#endif + +MODULE_AUTHOR("Realtek and the Linux r8127 crew "); +MODULE_DESCRIPTION("Realtek r8127 Ethernet controller driver"); + +module_param(speed_mode, uint, 0); +MODULE_PARM_DESC(speed_mode, "force phy operation. Deprecated by ethtool (8)."); + +module_param(duplex_mode, uint, 0); +MODULE_PARM_DESC(duplex_mode, "force phy operation. Deprecated by ethtool (8)."); + +module_param(autoneg_mode, uint, 0); +MODULE_PARM_DESC(autoneg_mode, "force phy operation. Deprecated by ethtool (8)."); + +module_param(aspm, int, 0); +MODULE_PARM_DESC(aspm, "Enable ASPM."); + +module_param(s5wol, int, 0); +MODULE_PARM_DESC(s5wol, "Enable Shutdown Wake On Lan."); + +module_param(s5_keep_curr_mac, int, 0); +MODULE_PARM_DESC(s5_keep_curr_mac, "Enable Shutdown Keep Current MAC Address."); + +module_param(use_dac, int, 0); +MODULE_PARM_DESC(use_dac, "Enable PCI DAC. Unsafe on 32 bit PCI slot."); + +module_param(timer_count, int, 0); +MODULE_PARM_DESC(timer_count, "Timer Interrupt Interval."); + +module_param(eee_enable, int, 0); +MODULE_PARM_DESC(eee_enable, "Enable Energy Efficient Ethernet."); + +module_param(hwoptimize, ulong, 0); +MODULE_PARM_DESC(hwoptimize, "Enable HW optimization function."); + +module_param(s0_magic_packet, int, 0); +MODULE_PARM_DESC(s0_magic_packet, "Enable S0 Magic Packet."); + +module_param(tx_no_close_enable, int, 0); +MODULE_PARM_DESC(tx_no_close_enable, "Enable TX No Close."); + +module_param(disable_wol_support, int, 0); +MODULE_PARM_DESC(disable_wol_support, "Disable PM support."); + +module_param(enable_double_vlan, int, 0); +MODULE_PARM_DESC(enable_double_vlan, "Enable Double VLAN."); + +module_param(eee_giga_lite, int, 0); +MODULE_PARM_DESC(eee_giga_lite, "Enable Giga Lite."); + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) +module_param_named(debug, debug.msg_enable, int, 0); +MODULE_PARM_DESC(debug, "Debug verbosity level (0=none, ..., 16=all)"); +#endif//LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + +MODULE_LICENSE("GPL"); +#ifdef ENABLE_USE_FIRMWARE_FILE +MODULE_FIRMWARE(FIRMWARE_8127_1); +MODULE_FIRMWARE(FIRMWARE_8127_2); +#endif + +MODULE_VERSION(RTL8127_VERSION); + +/* +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) +static void rtl8127_esd_timer(unsigned long __opaque); +#else +static void rtl8127_esd_timer(struct timer_list *t); +#endif +*/ +/* +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) +static void rtl8127_link_timer(unsigned long __opaque); +#else +static void rtl8127_link_timer(struct timer_list *t); +#endif +*/ + +static netdev_tx_t rtl8127_start_xmit(struct sk_buff *skb, struct net_device *dev); +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) +static irqreturn_t rtl8127_interrupt(int irq, void *dev_instance, struct pt_regs *regs); +#else +static irqreturn_t rtl8127_interrupt(int irq, void *dev_instance); +#endif +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) +static irqreturn_t rtl8127_interrupt_msix(int irq, void *dev_instance, struct pt_regs *regs); +#else +static irqreturn_t rtl8127_interrupt_msix(int irq, void *dev_instance); +#endif +static void rtl8127_set_rx_mode(struct net_device *dev); +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,6,0) +static void rtl8127_tx_timeout(struct net_device *dev, unsigned int txqueue); +#else +static void rtl8127_tx_timeout(struct net_device *dev); +#endif +static int rtl8127_rx_interrupt(struct net_device *, struct rtl8127_private *, struct rtl8127_rx_ring *, napi_budget); +static int rtl8127_tx_interrupt(struct rtl8127_tx_ring *ring, int budget); +static int rtl8127_tx_interrupt_with_vector(struct rtl8127_private *tp, const int message_id, int budget); +static void rtl8127_wait_for_quiescence(struct net_device *dev); +static int rtl8127_change_mtu(struct net_device *dev, int new_mtu); +static void rtl8127_down(struct net_device *dev); + +static int rtl8127_set_mac_address(struct net_device *dev, void *p); +static void rtl8127_rar_set(struct rtl8127_private *tp, const u8 *addr); +static void rtl8127_desc_addr_fill(struct rtl8127_private *); +static void rtl8127_tx_desc_init(struct rtl8127_private *tp); +static void rtl8127_rx_desc_init(struct rtl8127_private *tp); + +static u16 rtl8127_get_hw_phy_mcu_code_ver(struct rtl8127_private *tp); +static void rtl8127_phy_power_up(struct net_device *dev); +static void rtl8127_phy_power_down(struct net_device *dev); +static int rtl8127_set_speed(struct net_device *dev, u8 autoneg, u32 speed, u8 duplex, u64 adv); +static bool rtl8127_set_phy_mcu_patch_request(struct rtl8127_private *tp); +static bool rtl8127_clear_phy_mcu_patch_request(struct rtl8127_private *tp); + +#ifdef CONFIG_R8127_NAPI +static int rtl8127_poll(napi_ptr napi, napi_budget budget); +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +static void rtl8127_reset_task(void *_data); +static void rtl8127_esd_task(void *_data); +static void rtl8127_linkchg_task(void *_data); +#else +static void rtl8127_reset_task(struct work_struct *work); +static void rtl8127_esd_task(struct work_struct *work); +static void rtl8127_linkchg_task(struct work_struct *work); +#endif +static void rtl8127_schedule_reset_work(struct rtl8127_private *tp); +static void rtl8127_schedule_esd_work(struct rtl8127_private *tp); +static void rtl8127_schedule_linkchg_work(struct rtl8127_private *tp); +static void rtl8127_init_all_schedule_work(struct rtl8127_private *tp); +static void rtl8127_cancel_all_schedule_work(struct rtl8127_private *tp); + +static inline struct device *tp_to_dev(struct rtl8127_private *tp) +{ + return &tp->pci_dev->dev; +} + +#if ((LINUX_VERSION_CODE < KERNEL_VERSION(4,7,0) && \ + LINUX_VERSION_CODE >= KERNEL_VERSION(4,6,00))) +void ethtool_convert_legacy_u32_to_link_mode(unsigned long *dst, + u32 legacy_u32) +{ + bitmap_zero(dst, __ETHTOOL_LINK_MODE_MASK_NBITS); + dst[0] = legacy_u32; +} + +bool ethtool_convert_link_mode_to_legacy_u32(u32 *legacy_u32, + const unsigned long *src) +{ + bool retval = true; + + /* TODO: following test will soon always be true */ + if (__ETHTOOL_LINK_MODE_MASK_NBITS > 32) { + __ETHTOOL_DECLARE_LINK_MODE_MASK(ext); + + bitmap_zero(ext, __ETHTOOL_LINK_MODE_MASK_NBITS); + bitmap_fill(ext, 32); + bitmap_complement(ext, ext, __ETHTOOL_LINK_MODE_MASK_NBITS); + if (bitmap_intersects(ext, src, + __ETHTOOL_LINK_MODE_MASK_NBITS)) { + /* src mask goes beyond bit 31 */ + retval = false; + } + } + *legacy_u32 = src[0]; + return retval; +} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) + +#ifndef LPA_1000FULL +#define LPA_1000FULL 0x0800 +#endif + +#ifndef LPA_1000HALF +#define LPA_1000HALF 0x0400 +#endif + +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0) +static inline void eth_hw_addr_random(struct net_device *dev) +{ + random_ether_addr(dev->dev_addr); +} +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) +#undef ethtool_ops +#define ethtool_ops _kc_ethtool_ops + +struct _kc_ethtool_ops { + int (*get_settings)(struct net_device *, struct ethtool_cmd *); + int (*set_settings)(struct net_device *, struct ethtool_cmd *); + void (*get_drvinfo)(struct net_device *, struct ethtool_drvinfo *); + int (*get_regs_len)(struct net_device *); + void (*get_regs)(struct net_device *, struct ethtool_regs *, void *); + void (*get_wol)(struct net_device *, struct ethtool_wolinfo *); + int (*set_wol)(struct net_device *, struct ethtool_wolinfo *); + u32 (*get_msglevel)(struct net_device *); + void (*set_msglevel)(struct net_device *, u32); + int (*nway_reset)(struct net_device *); + u32 (*get_link)(struct net_device *); + int (*get_eeprom_len)(struct net_device *); + int (*get_eeprom)(struct net_device *, struct ethtool_eeprom *, u8 *); + int (*set_eeprom)(struct net_device *, struct ethtool_eeprom *, u8 *); + int (*get_coalesce)(struct net_device *, struct ethtool_coalesce *); + int (*set_coalesce)(struct net_device *, struct ethtool_coalesce *); + void (*get_ringparam)(struct net_device *, struct ethtool_ringparam *); + int (*set_ringparam)(struct net_device *, struct ethtool_ringparam *); + void (*get_pauseparam)(struct net_device *, + struct ethtool_pauseparam*); + int (*set_pauseparam)(struct net_device *, + struct ethtool_pauseparam*); + u32 (*get_rx_csum)(struct net_device *); + int (*set_rx_csum)(struct net_device *, u32); + u32 (*get_tx_csum)(struct net_device *); + int (*set_tx_csum)(struct net_device *, u32); + u32 (*get_sg)(struct net_device *); + int (*set_sg)(struct net_device *, u32); + u32 (*get_tso)(struct net_device *); + int (*set_tso)(struct net_device *, u32); + int (*self_test_count)(struct net_device *); + void (*self_test)(struct net_device *, struct ethtool_test *, u64 *); + void (*get_strings)(struct net_device *, u32 stringset, u8 *); + int (*phys_id)(struct net_device *, u32); + int (*get_stats_count)(struct net_device *); + void (*get_ethtool_stats)(struct net_device *, struct ethtool_stats *, + u64 *); +} *ethtool_ops = NULL; + +#undef SET_ETHTOOL_OPS +#define SET_ETHTOOL_OPS(netdev, ops) (ethtool_ops = (ops)) + +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,16,0) +#ifndef SET_ETHTOOL_OPS +#define SET_ETHTOOL_OPS(netdev,ops) \ + ((netdev)->ethtool_ops = (ops)) +#endif //SET_ETHTOOL_OPS +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,16,0) + +//#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,5) +#ifndef netif_msg_init +#define netif_msg_init _kc_netif_msg_init +/* copied from linux kernel 2.6.20 include/linux/netdevice.h */ +static inline u32 netif_msg_init(int debug_value, int default_msg_enable_bits) +{ + /* use default */ + if (debug_value < 0 || debug_value >= (sizeof(u32) * 8)) + return default_msg_enable_bits; + if (debug_value == 0) /* no output */ + return 0; + /* set low N bits */ + return (1 << debug_value) - 1; +} + +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,5) + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,22) +static inline void eth_copy_and_sum (struct sk_buff *dest, + const unsigned char *src, + int len, int base) +{ + skb_copy_to_linear_data(dest, src, len); +} +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,22) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,7) +/* copied from linux kernel 2.6.20 /include/linux/time.h */ +/* Parameters used to convert the timespec values: */ +#define MSEC_PER_SEC 1000L + +/* copied from linux kernel 2.6.20 /include/linux/jiffies.h */ +/* + * Change timeval to jiffies, trying to avoid the + * most obvious overflows.. + * + * And some not so obvious. + * + * Note that we don't want to return MAX_LONG, because + * for various timeout reasons we often end up having + * to wait "jiffies+1" in order to guarantee that we wait + * at _least_ "jiffies" - so "jiffies+1" had better still + * be positive. + */ +#define MAX_JIFFY_OFFSET ((~0UL >> 1)-1) + +/* + * Convert jiffies to milliseconds and back. + * + * Avoid unnecessary multiplications/divisions in the + * two most common HZ cases: + */ +static inline unsigned int _kc_jiffies_to_msecs(const unsigned long j) +{ +#if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ) + return (MSEC_PER_SEC / HZ) * j; +#elif HZ > MSEC_PER_SEC && !(HZ % MSEC_PER_SEC) + return (j + (HZ / MSEC_PER_SEC) - 1)/(HZ / MSEC_PER_SEC); +#else + return (j * MSEC_PER_SEC) / HZ; +#endif +} + +static inline unsigned long _kc_msecs_to_jiffies(const unsigned int m) +{ + if (m > _kc_jiffies_to_msecs(MAX_JIFFY_OFFSET)) + return MAX_JIFFY_OFFSET; +#if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ) + return (m + (MSEC_PER_SEC / HZ) - 1) / (MSEC_PER_SEC / HZ); +#elif HZ > MSEC_PER_SEC && !(HZ % MSEC_PER_SEC) + return m * (HZ / MSEC_PER_SEC); +#else + return (m * HZ + MSEC_PER_SEC - 1) / MSEC_PER_SEC; +#endif +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,7) + + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,11) + +/* copied from linux kernel 2.6.12.6 /include/linux/pm.h */ +typedef int __bitwise pci_power_t; + +/* copied from linux kernel 2.6.12.6 /include/linux/pci.h */ +typedef u32 __bitwise pm_message_t; + +#define PCI_D0 ((pci_power_t __force) 0) +#define PCI_D1 ((pci_power_t __force) 1) +#define PCI_D2 ((pci_power_t __force) 2) +#define PCI_D3hot ((pci_power_t __force) 3) +#define PCI_D3cold ((pci_power_t __force) 4) +#define PCI_POWER_ERROR ((pci_power_t __force) -1) + +/* copied from linux kernel 2.6.12.6 /drivers/pci/pci.c */ +/** + * pci_choose_state - Choose the power state of a PCI device + * @dev: PCI device to be suspended + * @state: target sleep state for the whole system. This is the value + * that is passed to suspend() function. + * + * Returns PCI power state suitable for given device and given system + * message. + */ + +pci_power_t pci_choose_state(struct pci_dev *dev, pm_message_t state) +{ + if (!pci_find_capability(dev, PCI_CAP_ID_PM)) + return PCI_D0; + + switch (state) { + case 0: + return PCI_D0; + case 3: + return PCI_D3hot; + default: + printk("They asked me for state %d\n", state); +// BUG(); + } + return PCI_D0; +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,11) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,9) +/** + * msleep_interruptible - sleep waiting for waitqueue interruptions + * @msecs: Time in milliseconds to sleep for + */ +#define msleep_interruptible _kc_msleep_interruptible +unsigned long _kc_msleep_interruptible(unsigned int msecs) +{ + unsigned long timeout = _kc_msecs_to_jiffies(msecs); + + while (timeout && !signal_pending(current)) { + set_current_state(TASK_INTERRUPTIBLE); + timeout = schedule_timeout(timeout); + } + return _kc_jiffies_to_msecs(timeout); +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,9) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,7) +/* copied from linux kernel 2.6.20 include/linux/sched.h */ +#ifndef __sched +#define __sched __attribute__((__section__(".sched.text"))) +#endif + +/* copied from linux kernel 2.6.20 kernel/timer.c */ +signed long __sched schedule_timeout_uninterruptible(signed long timeout) +{ + __set_current_state(TASK_UNINTERRUPTIBLE); + return schedule_timeout(timeout); +} + +/* copied from linux kernel 2.6.20 include/linux/mii.h */ +#undef if_mii +#define if_mii _kc_if_mii +static inline struct mii_ioctl_data *if_mii(struct ifreq *rq) +{ + return (struct mii_ioctl_data *) &rq->ifr_ifru; +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,7) + +static u32 rtl8127_read_thermal_sensor(struct rtl8127_private *tp) +{ + u16 ts_digout; + + ts_digout = rtl8127_mdio_direct_read_phy_ocp(tp, 0xBD84); + ts_digout &= 0x3ff; + + return ts_digout; +} + +int rtl8127_dump_tally_counter(struct rtl8127_private *tp, dma_addr_t paddr) +{ + u32 cmd; + u32 WaitCnt; + int retval = -1; + + RTL_W32(tp, CounterAddrHigh, (u64)paddr >> 32); + cmd = (u64)paddr & DMA_BIT_MASK(32); + RTL_W32(tp, CounterAddrLow, cmd); + RTL_W32(tp, CounterAddrLow, cmd | CounterDump); + + WaitCnt = 0; + while (RTL_R32(tp, CounterAddrLow) & CounterDump) { + udelay(10); + + WaitCnt++; + if (WaitCnt > 20) + break; + } + + if (WaitCnt <= 20) + retval = 0; + + return retval; +} + +static u32 +rtl8127_get_hw_clo_ptr(struct rtl8127_tx_ring *ring) +{ + struct rtl8127_private *tp = ring->priv; + + switch (tp->HwSuppTxNoCloseVer) { + case 3: + return RTL_R16(tp, ring->hw_clo_ptr_reg); + case 4: + case 5: + case 6: + return RTL_R32(tp, ring->hw_clo_ptr_reg); + default: +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + WARN_ON(1); +#endif + return 0; + } +} + +static u32 +rtl8127_get_sw_tail_ptr(struct rtl8127_tx_ring *ring) +{ + struct rtl8127_private *tp = ring->priv; + + switch (tp->HwSuppTxNoCloseVer) { + case 3: + return RTL_R16(tp, ring->sw_tail_ptr_reg); + case 4: + case 5: + case 6: + return RTL_R32(tp, ring->sw_tail_ptr_reg); + default: +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + WARN_ON(1); +#endif + return 0; + } +} + +static bool +rtl8127_sysfs_testmode_on(struct rtl8127_private *tp) +{ +#ifdef ENABLE_R8127_SYSFS + return !!tp->testmode; +#else + return 1; +#endif +} + +static u32 rtl8127_convert_link_speed(u32 status) +{ + u32 speed = SPEED_UNKNOWN; + + if (status & LinkStatus) { + if (status & _10000bpsF) + speed = SPEED_10000; + else if (status & (_5000bpsF | _10000bpsL)) + speed = SPEED_5000; + else if (status & (_2500bpsF | _5000bpsL)) + speed = SPEED_2500; + else if (status & (_1000bpsF | _2500bpsL | _1000bpsL)) + speed = SPEED_1000; + else if (status & _100bps) + speed = SPEED_100; + else if (status & _10bps) + speed = SPEED_10; + } + + return speed; +} + +static void rtl8127_mdi_swap(struct rtl8127_private *tp) +{ + int i; + u16 reg, val, mdi_reverse; + u16 tps_p0, tps_p1, tps_p2, tps_p3, tps_p3_p0; + + switch (tp->mcfg) { + default: + return; + }; + + tps_p3_p0 = rtl8127_mac_ocp_read(tp, 0xD440) & 0xF000; + tps_p3 = !!(tps_p3_p0 & BIT_15); + tps_p2 = !!(tps_p3_p0 & BIT_14); + tps_p1 = !!(tps_p3_p0 & BIT_13); + tps_p0 = !!(tps_p3_p0 & BIT_12); + mdi_reverse = rtl8127_mac_ocp_read(tp, 0xD442); + + if ((mdi_reverse & BIT_5) && tps_p3_p0 == 0xA000) + return; + + if (!(mdi_reverse & BIT_5)) + val = tps_p0 << 8 | + tps_p1 << 9 | + tps_p2 << 10 | + tps_p3 << 11; + else + val = tps_p3 << 8 | + tps_p2 << 9 | + tps_p1 << 10 | + tps_p0 << 11; + + for (i=8; i<12; i++) { + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, reg); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + BIT(i), + val & BIT(i)); + } +} + +static int rtl8127_vcd_test(struct rtl8127_private *tp) +{ + u16 val; + u32 wait_cnt; + int ret = -1; + + rtl8127_mdi_swap(tp); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA422, BIT(0)); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA422, 0x00F0); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA422, BIT(0)); + + wait_cnt = 0; + do { + mdelay(1); + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA422); + wait_cnt++; + } while (!(val & BIT_15) && (wait_cnt < 5000)); + + if (wait_cnt == 5000) + goto exit; + + ret = 0; + +exit: + return ret; +} + +static void rtl8127_get_cp_len(struct rtl8127_private *tp, + int cp_len[RTL8127_CP_NUM]) +{ + int i; + u16 status; + int tmp_cp_len; + + status = RTL_R16(tp, PHYstatus); + if (status & LinkStatus) { + if (status & _10bps) { + tmp_cp_len = -1; + } else if (status & (_100bps | _1000bpsF)) { + rtl8127_mdio_write(tp, 0x1f, 0x0a88); + tmp_cp_len = rtl8127_mdio_read(tp, 0x10); + } else if (status & _2500bpsF) { + rtl8127_mdio_write(tp, 0x1f, 0x0acb); + tmp_cp_len = rtl8127_mdio_read(tp, 0x15); + tmp_cp_len >>= 2; + } else + tmp_cp_len = 0; + } else + tmp_cp_len = 0; + + if (tmp_cp_len > 0) + tmp_cp_len &= 0xff; + for (i=0; i RTL8127_MAX_SUPPORT_CP_LEN) + cp_len[i] = RTL8127_MAX_SUPPORT_CP_LEN; + + return; +} + +static int __rtl8127_get_cp_status(u16 val) +{ + switch (val) { + case 0x0060: + return rtl8127_cp_normal; + case 0x0048: + return rtl8127_cp_open; + case 0x0050: + return rtl8127_cp_short; + case 0x0042: + case 0x0044: + return rtl8127_cp_mismatch; + default: + return rtl8127_cp_normal; + } +} + +static int _rtl8127_get_cp_status(struct rtl8127_private *tp, u8 pair_num) +{ + u16 val; + int cp_status = rtl8127_cp_unknown; + + if (pair_num > 3) + goto exit; + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8027 + 4 * pair_num); + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA438); + + cp_status = __rtl8127_get_cp_status(val); + +exit: + return cp_status; +} + +static const char * rtl8127_get_cp_status_string(int cp_status) +{ + switch(cp_status) { + case rtl8127_cp_normal: + return "normal "; + case rtl8127_cp_short: + return "short "; + case rtl8127_cp_open: + return "open "; + case rtl8127_cp_mismatch: + return "mismatch"; + default: + return "unknown "; + } +} + +static u16 rtl8127_get_cp_pp(struct rtl8127_private *tp, u8 pair_num) +{ + u16 pp = 0; + + if (pair_num > 3) + goto exit; + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8029 + 4 * pair_num); + pp = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA438); + + pp &= 0x3fff; + pp /= 80; + +exit: + return pp; +} + +static void rtl8127_get_cp_status(struct rtl8127_private *tp, + int cp_status[RTL8127_CP_NUM], + bool poe_mode) +{ + u16 status; + int i; + + status = RTL_R16(tp, PHYstatus); + if (status & LinkStatus && !(status & (_10bps | _100bps))) { + for (i=0; i= KERNEL_VERSION(3,10,0) +static int proc_get_driver_variable(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + + seq_puts(m, "\nDump Driver Variable\n"); + + rtnl_lock(); + + seq_puts(m, "Variable\tValue\n----------\t-----\n"); + seq_printf(m, "MODULENAME\t%s\n", MODULENAME); + seq_printf(m, "driver version\t%s\n", RTL8127_VERSION); + seq_printf(m, "mcfg\t%d\n", tp->mcfg); + seq_printf(m, "chipset\t%d\n", tp->chipset); + seq_printf(m, "chipset_name\t%s\n", rtl_chip_info[tp->chipset].name); + seq_printf(m, "mtu\t%d\n", dev->mtu); + seq_printf(m, "NUM_RX_DESC\t0x%x\n", tp->rx_ring[0].num_rx_desc); + seq_printf(m, "cur_rx0\t0x%x\n", tp->rx_ring[0].cur_rx); + seq_printf(m, "dirty_rx0\t0x%x\n", tp->rx_ring[0].dirty_rx); + seq_printf(m, "cur_rx1\t0x%x\n", tp->rx_ring[1].cur_rx); + seq_printf(m, "dirty_rx1\t0x%x\n", tp->rx_ring[1].dirty_rx); + seq_printf(m, "cur_rx2\t0x%x\n", tp->rx_ring[2].cur_rx); + seq_printf(m, "dirty_rx2\t0x%x\n", tp->rx_ring[2].dirty_rx); + seq_printf(m, "cur_rx3\t0x%x\n", tp->rx_ring[3].cur_rx); + seq_printf(m, "dirty_rx3\t0x%x\n", tp->rx_ring[3].dirty_rx); + seq_printf(m, "NUM_TX_DESC\t0x%x\n", tp->tx_ring[0].num_tx_desc); + seq_printf(m, "cur_tx0\t0x%x\n", tp->tx_ring[0].cur_tx); + seq_printf(m, "dirty_tx0\t0x%x\n", tp->tx_ring[0].dirty_tx); + seq_printf(m, "cur_tx1\t0x%x\n", tp->tx_ring[1].cur_tx); + seq_printf(m, "dirty_tx1\t0x%x\n", tp->tx_ring[1].dirty_tx); + seq_printf(m, "rx_buf_sz\t0x%x\n", tp->rx_buf_sz); +#ifdef ENABLE_PAGE_REUSE + seq_printf(m, "rx_buf_page_order\t0x%x\n", tp->rx_buf_page_order); + seq_printf(m, "rx_buf_page_size\t0x%x\n", tp->rx_buf_page_size); + seq_printf(m, "page_reuse_fail_cnt\t0x%x\n", tp->page_reuse_fail_cnt); +#endif //ENABLE_PAGE_REUSE + seq_printf(m, "esd_flag\t0x%x\n", tp->esd_flag); + seq_printf(m, "pci_cfg_is_read\t0x%x\n", tp->pci_cfg_is_read); + seq_printf(m, "rtl8127_rx_config\t0x%x\n", tp->rtl8127_rx_config); + seq_printf(m, "cp_cmd\t0x%x\n", tp->cp_cmd); + seq_printf(m, "intr_mask\t0x%x\n", tp->intr_mask); + seq_printf(m, "timer_intr_mask\t0x%x\n", tp->timer_intr_mask); + seq_printf(m, "wol_enabled\t0x%x\n", tp->wol_enabled); + seq_printf(m, "wol_opts\t0x%x\n", tp->wol_opts); + seq_printf(m, "efuse_ver\t0x%x\n", tp->efuse_ver); + seq_printf(m, "eeprom_type\t0x%x\n", tp->eeprom_type); + seq_printf(m, "autoneg\t0x%x\n", tp->autoneg); + seq_printf(m, "duplex\t0x%x\n", tp->duplex); + seq_printf(m, "speed\t%d\n", tp->speed); + seq_printf(m, "advertising\t0x%llx\n", tp->advertising); + seq_printf(m, "eeprom_len\t0x%x\n", tp->eeprom_len); + seq_printf(m, "cur_page\t0x%x\n", tp->cur_page); + seq_printf(m, "features\t0x%x\n", tp->features); + seq_printf(m, "org_pci_offset_99\t0x%x\n", tp->org_pci_offset_99); + seq_printf(m, "org_pci_offset_180\t0x%x\n", tp->org_pci_offset_180); + seq_printf(m, "issue_offset_99_event\t0x%x\n", tp->issue_offset_99_event); + seq_printf(m, "org_pci_offset_80\t0x%x\n", tp->org_pci_offset_80); + seq_printf(m, "org_pci_offset_81\t0x%x\n", tp->org_pci_offset_81); + seq_printf(m, "use_timer_interrupt\t0x%x\n", tp->use_timer_interrupt); + seq_printf(m, "HwIcVerUnknown\t0x%x\n", tp->HwIcVerUnknown); + seq_printf(m, "NotWrRamCodeToMicroP\t0x%x\n", tp->NotWrRamCodeToMicroP); + seq_printf(m, "NotWrMcuPatchCode\t0x%x\n", tp->NotWrMcuPatchCode); + seq_printf(m, "HwHasWrRamCodeToMicroP\t0x%x\n", tp->HwHasWrRamCodeToMicroP); + seq_printf(m, "sw_ram_code_ver\t0x%x\n", tp->sw_ram_code_ver); + seq_printf(m, "hw_ram_code_ver\t0x%x\n", tp->hw_ram_code_ver); + seq_printf(m, "rtk_enable_diag\t0x%x\n", tp->rtk_enable_diag); + seq_printf(m, "ShortPacketSwChecksum\t0x%x\n", tp->ShortPacketSwChecksum); + seq_printf(m, "UseSwPaddingShortPkt\t0x%x\n", tp->UseSwPaddingShortPkt); + seq_printf(m, "RequireAdcBiasPatch\t0x%x\n", tp->RequireAdcBiasPatch); + seq_printf(m, "AdcBiasPatchIoffset\t0x%x\n", tp->AdcBiasPatchIoffset); + seq_printf(m, "RequireAdjustUpsTxLinkPulseTiming\t0x%x\n", tp->RequireAdjustUpsTxLinkPulseTiming); + seq_printf(m, "SwrCnt1msIni\t0x%x\n", tp->SwrCnt1msIni); + seq_printf(m, "HwSuppNowIsOobVer\t0x%x\n", tp->HwSuppNowIsOobVer); + seq_printf(m, "HwFiberModeVer\t0x%x\n", tp->HwFiberModeVer); + seq_printf(m, "HwFiberStat\t0x%x\n", tp->HwFiberStat); + seq_printf(m, "HwSwitchMdiToFiber\t0x%x\n", tp->HwSwitchMdiToFiber); + seq_printf(m, "NicCustLedValue\t0x%x\n", tp->NicCustLedValue); + seq_printf(m, "RequiredSecLanDonglePatch\t0x%x\n", tp->RequiredSecLanDonglePatch); + seq_printf(m, "HwSuppDashVer\t0x%x\n", tp->HwSuppDashVer); + seq_printf(m, "DASH\t0x%x\n", tp->DASH); + seq_printf(m, "dash_printer_enabled\t0x%x\n", tp->dash_printer_enabled); + seq_printf(m, "HwSuppKCPOffloadVer\t0x%x\n", tp->HwSuppKCPOffloadVer); + seq_printf(m, "speed_mode\t0x%x\n", speed_mode); + seq_printf(m, "duplex_mode\t0x%x\n", duplex_mode); + seq_printf(m, "autoneg_mode\t0x%x\n", autoneg_mode); + seq_printf(m, "aspm\t0x%x\n", aspm); + seq_printf(m, "s5wol\t0x%x\n", s5wol); + seq_printf(m, "s5_keep_curr_mac\t0x%x\n", s5_keep_curr_mac); + seq_printf(m, "eee_enable\t0x%x\n", tp->eee.eee_enabled); + seq_printf(m, "hwoptimize\t0x%lx\n", hwoptimize); + seq_printf(m, "proc_init_num\t0x%x\n", proc_init_num); + seq_printf(m, "s0_magic_packet\t0x%x\n", s0_magic_packet); + seq_printf(m, "disable_wol_support\t0x%x\n", disable_wol_support); + seq_printf(m, "enable_double_vlan\t0x%x\n", enable_double_vlan); + seq_printf(m, "eee_giga_lite\t0x%x\n", eee_giga_lite); + seq_printf(m, "HwSuppMagicPktVer\t0x%x\n", tp->HwSuppMagicPktVer); + seq_printf(m, "HwSuppLinkChgWakeUpVer\t0x%x\n", tp->HwSuppLinkChgWakeUpVer); + seq_printf(m, "HwSuppD0SpeedUpVer\t0x%x\n", tp->HwSuppD0SpeedUpVer); + seq_printf(m, "D0SpeedUpSpeed\t0x%x\n", tp->D0SpeedUpSpeed); + seq_printf(m, "HwSuppCheckPhyDisableModeVer\t0x%x\n", tp->HwSuppCheckPhyDisableModeVer); + seq_printf(m, "HwPkgDet\t0x%x\n", tp->HwPkgDet); + seq_printf(m, "HwSuppTxNoCloseVer\t0x%x\n", tp->HwSuppTxNoCloseVer); + seq_printf(m, "EnableTxNoClose\t0x%x\n", tp->EnableTxNoClose); + seq_printf(m, "NextHwDesCloPtr0\t0x%x\n", tp->tx_ring[0].NextHwDesCloPtr); + seq_printf(m, "BeginHwDesCloPtr0\t0x%x\n", tp->tx_ring[0].BeginHwDesCloPtr); + seq_printf(m, "hw_clo_ptr_reg0\t0x%x\n", rtl8127_get_hw_clo_ptr(&tp->tx_ring[0])); + seq_printf(m, "sw_tail_ptr_reg0\t0x%x\n", rtl8127_get_sw_tail_ptr(&tp->tx_ring[0])); + seq_printf(m, "NextHwDesCloPtr1\t0x%x\n", tp->tx_ring[1].NextHwDesCloPtr); + seq_printf(m, "BeginHwDesCloPtr1\t0x%x\n", tp->tx_ring[1].BeginHwDesCloPtr); + seq_printf(m, "hw_clo_ptr_reg1\t0x%x\n", rtl8127_get_hw_clo_ptr(&tp->tx_ring[1])); + seq_printf(m, "sw_tail_ptr_reg1\t0x%x\n", rtl8127_get_sw_tail_ptr(&tp->tx_ring[1])); + seq_printf(m, "InitRxDescType\t0x%x\n", tp->InitRxDescType); + seq_printf(m, "RxDescLength\t0x%x\n", tp->RxDescLength); + seq_printf(m, "num_rx_rings\t0x%x\n", tp->num_rx_rings); + seq_printf(m, "num_tx_rings\t0x%x\n", tp->num_tx_rings); + seq_printf(m, "tot_rx_rings\t0x%x\n", rtl8127_tot_rx_rings(tp)); + seq_printf(m, "tot_tx_rings\t0x%x\n", rtl8127_tot_tx_rings(tp)); + seq_printf(m, "HwSuppNumRxQueues\t0x%x\n", tp->HwSuppNumRxQueues); + seq_printf(m, "HwSuppNumTxQueues\t0x%x\n", tp->HwSuppNumTxQueues); + seq_printf(m, "EnableRss\t0x%x\n", tp->EnableRss); + seq_printf(m, "EnablePtp\t0x%x\n", tp->EnablePtp); + seq_printf(m, "min_irq_nvecs\t0x%x\n", tp->min_irq_nvecs); + seq_printf(m, "irq_nvecs\t0x%x\n", tp->irq_nvecs); + seq_printf(m, "hw_supp_irq_nvecs\t0x%x\n", tp->hw_supp_irq_nvecs); + seq_printf(m, "ring_lib_enabled\t0x%x\n", tp->ring_lib_enabled); + seq_printf(m, "HwSuppIsrVer\t0x%x\n", tp->HwSuppIsrVer); + seq_printf(m, "HwCurrIsrVer\t0x%x\n", tp->HwCurrIsrVer); + seq_printf(m, "HwSuppMacMcuVer\t0x%x\n", tp->HwSuppMacMcuVer); + seq_printf(m, "MacMcuPageSize\t0x%x\n", tp->MacMcuPageSize); + seq_printf(m, "hw_mcu_patch_code_ver\t0x%llx\n", tp->hw_mcu_patch_code_ver); + seq_printf(m, "bin_mcu_patch_code_ver\t0x%llx\n", tp->bin_mcu_patch_code_ver); +#ifdef ENABLE_PTP_SUPPORT + seq_printf(m, "tx_hwtstamp_timeouts\t0x%x\n", tp->tx_hwtstamp_timeouts); + seq_printf(m, "tx_hwtstamp_skipped\t0x%x\n", tp->tx_hwtstamp_skipped); +#endif + seq_printf(m, "random_mac\t0x%x\n", tp->random_mac); + seq_printf(m, "org_mac_addr\t%pM\n", tp->org_mac_addr); +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + seq_printf(m, "perm_addr\t%pM\n", dev->perm_addr); +#endif + seq_printf(m, "dev_addr\t%pM\n", dev->dev_addr); + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_tally_counter(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + struct rtl8127_counters *counters; + dma_addr_t paddr; + + seq_puts(m, "\nDump Tally Counter\n"); + + rtnl_lock(); + + counters = tp->tally_vaddr; + paddr = tp->tally_paddr; + if (!counters) { + seq_puts(m, "\nDump Tally Counter Fail\n"); + goto out_unlock; + } + + rtl8127_dump_tally_counter(tp, paddr); + + seq_puts(m, "Statistics\tValue\n----------\t-----\n"); + seq_printf(m, "tx_packets\t%lld\n", le64_to_cpu(counters->tx_packets)); + seq_printf(m, "rx_packets\t%lld\n", le64_to_cpu(counters->rx_packets)); + seq_printf(m, "tx_errors\t%lld\n", le64_to_cpu(counters->tx_errors)); + seq_printf(m, "rx_errors\t%d\n", le32_to_cpu(counters->rx_errors)); + seq_printf(m, "rx_missed\t%d\n", le16_to_cpu(counters->rx_missed)); + seq_printf(m, "align_errors\t%d\n", le16_to_cpu(counters->align_errors)); + seq_printf(m, "tx_one_collision\t%d\n", le32_to_cpu(counters->tx_one_collision)); + seq_printf(m, "tx_multi_collision\t%d\n", le32_to_cpu(counters->tx_multi_collision)); + seq_printf(m, "rx_unicast\t%lld\n", le64_to_cpu(counters->rx_unicast)); + seq_printf(m, "rx_broadcast\t%lld\n", le64_to_cpu(counters->rx_broadcast)); + seq_printf(m, "rx_multicast\t%d\n", le32_to_cpu(counters->rx_multicast)); + seq_printf(m, "tx_aborted\t%d\n", le16_to_cpu(counters->tx_aborted)); + seq_printf(m, "tx_underrun\t%d\n", le16_to_cpu(counters->tx_underrun)); + + seq_printf(m, "tx_octets\t%lld\n", le64_to_cpu(counters->tx_octets)); + seq_printf(m, "rx_octets\t%lld\n", le64_to_cpu(counters->rx_octets)); + seq_printf(m, "rx_multicast64\t%lld\n", le64_to_cpu(counters->rx_multicast64)); + seq_printf(m, "tx_unicast64\t%lld\n", le64_to_cpu(counters->tx_unicast64)); + seq_printf(m, "tx_broadcast64\t%lld\n", le64_to_cpu(counters->tx_broadcast64)); + seq_printf(m, "tx_multicast64\t%lld\n", le64_to_cpu(counters->tx_multicast64)); + seq_printf(m, "tx_pause_on\t%d\n", le32_to_cpu(counters->tx_pause_on)); + seq_printf(m, "tx_pause_off\t%d\n", le32_to_cpu(counters->tx_pause_off)); + seq_printf(m, "tx_pause_all\t%d\n", le32_to_cpu(counters->tx_pause_all)); + seq_printf(m, "tx_deferred\t%d\n", le32_to_cpu(counters->tx_deferred)); + seq_printf(m, "tx_late_collision\t%d\n", le32_to_cpu(counters->tx_late_collision)); + seq_printf(m, "tx_all_collision\t%d\n", le32_to_cpu(counters->tx_all_collision)); + seq_printf(m, "tx_aborted32\t%d\n", le32_to_cpu(counters->tx_aborted32)); + seq_printf(m, "align_errors32\t%d\n", le32_to_cpu(counters->align_errors32)); + seq_printf(m, "rx_frame_too_long\t%d\n", le32_to_cpu(counters->rx_frame_too_long)); + seq_printf(m, "rx_runt\t%d\n", le32_to_cpu(counters->rx_runt)); + seq_printf(m, "rx_pause_on\t%d\n", le32_to_cpu(counters->rx_pause_on)); + seq_printf(m, "rx_pause_off\t%d\n", le32_to_cpu(counters->rx_pause_off)); + seq_printf(m, "rx_pause_all\t%d\n", le32_to_cpu(counters->rx_pause_all)); + seq_printf(m, "rx_unknown_opcode\t%d\n", le32_to_cpu(counters->rx_unknown_opcode)); + seq_printf(m, "rx_mac_error\t%d\n", le32_to_cpu(counters->rx_mac_error)); + seq_printf(m, "tx_underrun32\t%d\n", le32_to_cpu(counters->tx_underrun32)); + seq_printf(m, "rx_mac_missed\t%d\n", le32_to_cpu(counters->rx_mac_missed)); + seq_printf(m, "rx_tcam_dropped\t%d\n", le32_to_cpu(counters->rx_tcam_dropped)); + seq_printf(m, "tdu\t%d\n", le32_to_cpu(counters->tdu)); + seq_printf(m, "rdu\t%d\n", le32_to_cpu(counters->rdu)); + + seq_putc(m, '\n'); + +out_unlock: + rtnl_unlock(); + + return 0; +} + +static int proc_get_registers(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max = R8127_MAC_REGS_SIZE; + u8 byte_rd; + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + + seq_puts(m, "\nDump MAC Registers\n"); + seq_puts(m, "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + for (n = 0; n < max;) { + seq_printf(m, "\n0x%04x:\t", n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + seq_printf(m, "%02x ", byte_rd); + } + } + + max = 0xB00; + for (n = 0xA00; n < max;) { + seq_printf(m, "\n0x%04x:\t", n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + seq_printf(m, "%02x ", byte_rd); + } + } + + max = 0xD40; + for (n = 0xD00; n < max;) { + seq_printf(m, "\n0x%04x:\t", n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + seq_printf(m, "%02x ", byte_rd); + } + } + + max = 0x2840; + for (n = 0x2800; n < max;) { + seq_printf(m, "\n0x%04x:\t", n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + seq_printf(m, "%02x ", byte_rd); + } + } + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_all_registers(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max; + u8 byte_rd; + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + struct pci_dev *pdev = tp->pci_dev; + + seq_puts(m, "\nDump All MAC Registers\n"); + seq_puts(m, "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + max = pci_resource_len(pdev, 2); + + for (n = 0; n < max;) { + seq_printf(m, "\n0x%04x:\t", n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + seq_printf(m, "%02x ", byte_rd); + } + } + + rtnl_unlock(); + + seq_printf(m, "\nTotal length:0x%X", max); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_pcie_phy(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max = R8127_EPHY_REGS_SIZE/2; + u16 word_rd; + struct rtl8127_private *tp = netdev_priv(dev); + + seq_puts(m, "\nDump PCIE PHY\n"); + seq_puts(m, "\nOffset\tValue\n------\t-----\n "); + + rtnl_lock(); + + for (n = 0; n < max;) { + seq_printf(m, "\n0x%02x:\t", n); + + for (i = 0; i < 8 && n < max; i++, n++) { + word_rd = rtl8127_ephy_read(tp, n); + seq_printf(m, "%04x ", word_rd); + } + } + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_eth_phy(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max = R8127_PHY_REGS_SIZE/2; + unsigned long flags; + u16 word_rd; + struct rtl8127_private *tp = netdev_priv(dev); + + seq_puts(m, "\nDump Ethernet PHY\n"); + seq_puts(m, "\nOffset\tValue\n------\t-----\n "); + + spin_lock_irqsave(&tp->phy_lock, flags); + + seq_puts(m, "\n####################page 0##################\n "); + rtl8127_mdio_write(tp, 0x1f, 0x0000); + for (n = 0; n < max;) { + seq_printf(m, "\n0x%02x:\t", n); + + for (i = 0; i < 8 && n < max; i++, n++) { + word_rd = rtl8127_mdio_read(tp, n); + seq_printf(m, "%04x ", word_rd); + } + } + + seq_puts(m, "\n####################extra reg##################\n "); + n = 0xA400; + seq_printf(m, "\n0x%02x:\t", n); + for (i = 0; i < 8; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + } + + n = 0xA410; + seq_printf(m, "\n0x%02x:\t", n); + for (i = 0; i < 3; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + } + + n = 0xA434; + seq_printf(m, "\n0x%02x:\t", n); + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + + n = 0xA5D0; + seq_printf(m, "\n0x%02x:\t", n); + for (i = 0; i < 4; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + } + + n = 0xA61A; + seq_printf(m, "\n0x%02x:\t", n); + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + + n = 0xA6D0; + seq_printf(m, "\n0x%02x:\t", n); + for (i = 0; i < 3; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + seq_printf(m, "%04x ", word_rd); + } + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_extended_registers(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max = R8127_ERI_REGS_SIZE; + u32 dword_rd; + struct rtl8127_private *tp = netdev_priv(dev); + + seq_puts(m, "\nDump Extended Registers\n"); + seq_puts(m, "\nOffset\tValue\n------\t-----\n "); + + rtnl_lock(); + + for (n = 0; n < max;) { + seq_printf(m, "\n0x%02x:\t", n); + + for (i = 0; i < 4 && n < max; i++, n+=4) { + dword_rd = rtl8127_eri_read(tp, n, 4, ERIAR_ExGMAC); + seq_printf(m, "%08x ", dword_rd); + } + } + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_pci_registers(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + int i, n, max = R8127_PCI_REGS_SIZE; + u32 dword_rd; + struct rtl8127_private *tp = netdev_priv(dev); + + seq_puts(m, "\nDump PCI Registers\n"); + seq_puts(m, "\nOffset\tValue\n------\t-----\n "); + + rtnl_lock(); + + for (n = 0; n < max;) { + seq_printf(m, "\n0x%03x:\t", n); + + for (i = 0; i < 4 && n < max; i++, n+=4) { + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + seq_printf(m, "%08x ", dword_rd); + } + } + + n = 0x110; + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + seq_printf(m, "\n0x%03x:\t%08x ", n, dword_rd); + n = 0x70c; + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + seq_printf(m, "\n0x%03x:\t%08x ", n, dword_rd); + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_get_temperature(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + u16 ts_digout, tj, fah; + + seq_puts(m, "\nChip Temperature\n"); + + rtnl_lock(); + + if (!rtl8127_sysfs_testmode_on(tp)) { + seq_puts(m, "\nPlease turn on ""/sys/class/net//rtk_adv/testmode"".\n\n"); + rtnl_unlock(); + return 0; + } + + netif_testing_on(dev); + ts_digout = rtl8127_read_thermal_sensor(tp); + netif_testing_off(dev); + + rtnl_unlock(); + + tj = ts_digout / 2; + if (ts_digout <= 512) { + tj = ts_digout / 2; + seq_printf(m, "Cel:%d\n", tj); + fah = tj * (9/5) + 32; + seq_printf(m, "Fah:%d\n", fah); + } else { + tj = (512 - ((ts_digout / 2) - 512)) / 2; + seq_printf(m, "Cel:-%d\n", tj); + fah = tj * (9/5) + 32; + seq_printf(m, "Fah:-%d\n", fah); + } + + seq_putc(m, '\n'); + return 0; +} + +static int _proc_get_cable_info(struct seq_file *m, void *v, bool poe_mode) +{ + int i; + u16 status; + int cp_status[RTL8127_CP_NUM]; + int cp_len[RTL8127_CP_NUM] = {0}; + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + const char *pair_str[RTL8127_CP_NUM] = {"1-2", "3-6", "4-5", "7-8"}; + int ret; + + switch (tp->mcfg) { + default: + ret = -EOPNOTSUPP; + goto error_out; + } + + rtnl_lock(); + + if (!rtl8127_sysfs_testmode_on(tp)) { + seq_puts(m, "\nPlease turn on ""/sys/class/net//rtk_adv/testmode"".\n\n"); + ret = 0; + goto error_unlock; + } + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + if (rtl8127_mdio_read(tp, MII_BMCR) & BMCR_PDOWN) { + ret = -EIO; + goto error_unlock; + } + + netif_testing_on(dev); + + status = RTL_R16(tp, PHYstatus); + if (status & LinkStatus) + seq_printf(m, "\nlink speed:%d", + rtl8127_convert_link_speed(status)); + else + seq_puts(m, "\nlink status:off"); + + rtl8127_get_cp_len(tp, cp_len); + + rtl8127_get_cp_status(tp, cp_status, poe_mode); + + seq_puts(m, "\npair\tlength\tstatus \tpp\n"); + + for (i=0; iprivate; + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + rtnl_lock(); + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + + if (!ring) + continue; + + seq_printf(m, "\ndump rx %d desc:%d\n", i, ring->num_rx_desc); + + _proc_dump_desc(m, (void*)ring->RxDescArray, ring->RxDescAllocSize); + } + +#ifdef ENABLE_LIB_SUPPORT + if (rtl8127_num_lib_rx_rings(tp) > 0) { + for (i = 0; i < tp->HwSuppNumRxQueues; i++) { + struct rtl8127_ring *lib_ring = &tp->lib_rx_ring[i]; + if (lib_ring->enabled) { + seq_printf(m, "\ndump lib rx %d desc:%d\n", i, + lib_ring->ring_size); + _proc_dump_desc(m, (void*)lib_ring->desc_addr, + lib_ring->desc_size); + } + } + } +#endif //ENABLE_LIB_SUPPORT + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_dump_tx_desc(struct seq_file *m, void *v) +{ + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + rtnl_lock(); + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + + if (!ring) + continue; + + seq_printf(m, "\ndump tx %d desc:%d\n", i, ring->num_tx_desc); + + _proc_dump_desc(m, (void*)ring->TxDescArray, ring->TxDescAllocSize); + } + +#ifdef ENABLE_LIB_SUPPORT + if (rtl8127_num_lib_tx_rings(tp) > 0) { + for (i = 0; i < tp->HwSuppNumTxQueues; i++) { + struct rtl8127_ring *lib_ring = &tp->lib_tx_ring[i]; + if (lib_ring->enabled) { + seq_printf(m, "\ndump lib tx %d desc:%d\n", i, + lib_ring->ring_size); + _proc_dump_desc(m, (void*)lib_ring->desc_addr, + lib_ring->desc_size); + } + } + } +#endif //ENABLE_LIB_SUPPORT + + rtnl_unlock(); + + seq_putc(m, '\n'); + return 0; +} + +static int proc_dump_msix_tbl(struct seq_file *m, void *v) +{ + int i, j; + void __iomem *ioaddr; + struct net_device *dev = m->private; + struct rtl8127_private *tp = netdev_priv(dev); + + /* ioremap MMIO region */ + ioaddr = ioremap(pci_resource_start(tp->pci_dev, 4), pci_resource_len(tp->pci_dev, 4)); + if (!ioaddr) + return -EFAULT; + + rtnl_lock(); + + seq_printf(m, "\ndump MSI-X Table. Total Entry %d. \n", tp->hw_supp_irq_nvecs); + + for (i=0; ihw_supp_irq_nvecs; i++) { + seq_printf(m, "\n%04x ", i); + for (j=0; j<4; j++) + seq_printf(m, "%08x ", + readl(ioaddr + i*0x10 + 4*j)); + } + + rtnl_unlock(); + + iounmap(ioaddr); + + seq_putc(m, '\n'); + return 0; +} + +#else //LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + +static int proc_get_driver_variable(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump Driver Driver\n"); + + rtnl_lock(); + + len += snprintf(page + len, count - len, + "Variable\tValue\n----------\t-----\n"); + + len += snprintf(page + len, count - len, + "MODULENAME\t%s\n" + "driver version\t%s\n" + "mcfg\t%d\n" + "chipset\t%d\n" + "chipset_name\t%s\n" + "mtu\t%d\n" + "NUM_RX_DESC\t0x%x\n" + "cur_rx0\t0x%x\n" + "dirty_rx0\t0x%x\n" + "cur_rx1\t0x%x\n" + "dirty_rx1\t0x%x\n" + "cur_rx2\t0x%x\n" + "dirty_rx2\t0x%x\n" + "cur_rx3\t0x%x\n" + "dirty_rx3\t0x%x\n" + "NUM_TX_DESC\t0x%x\n" + "cur_tx0\t0x%x\n" + "dirty_tx0\t0x%x\n" + "cur_tx1\t0x%x\n" + "dirty_tx1\t0x%x\n" + "rx_buf_sz\t0x%x\n" +#ifdef ENABLE_PAGE_REUSE + "rx_buf_page_order\t0x%x\n" + "rx_buf_page_size\t0x%x\n" + "page_reuse_fail_cnt\t0x%x\n" +#endif //ENABLE_PAGE_REUSE + "esd_flag\t0x%x\n" + "pci_cfg_is_read\t0x%x\n" + "rtl8127_rx_config\t0x%x\n" + "cp_cmd\t0x%x\n" + "intr_mask\t0x%x\n" + "timer_intr_mask\t0x%x\n" + "wol_enabled\t0x%x\n" + "wol_opts\t0x%x\n" + "efuse_ver\t0x%x\n" + "eeprom_type\t0x%x\n" + "autoneg\t0x%x\n" + "duplex\t0x%x\n" + "speed\t%d\n" + "advertising\t0x%llx\n" + "eeprom_len\t0x%x\n" + "cur_page\t0x%x\n" + "features\t0x%x\n" + "org_pci_offset_99\t0x%x\n" + "org_pci_offset_180\t0x%x\n" + "issue_offset_99_event\t0x%x\n" + "org_pci_offset_80\t0x%x\n" + "org_pci_offset_81\t0x%x\n" + "use_timer_interrupt\t0x%x\n" + "HwIcVerUnknown\t0x%x\n" + "NotWrRamCodeToMicroP\t0x%x\n" + "NotWrMcuPatchCode\t0x%x\n" + "HwHasWrRamCodeToMicroP\t0x%x\n" + "sw_ram_code_ver\t0x%x\n" + "hw_ram_code_ver\t0x%x\n" + "rtk_enable_diag\t0x%x\n" + "ShortPacketSwChecksum\t0x%x\n" + "UseSwPaddingShortPkt\t0x%x\n" + "RequireAdcBiasPatch\t0x%x\n" + "AdcBiasPatchIoffset\t0x%x\n" + "RequireAdjustUpsTxLinkPulseTiming\t0x%x\n" + "SwrCnt1msIni\t0x%x\n" + "HwSuppNowIsOobVer\t0x%x\n" + "HwFiberModeVer\t0x%x\n" + "HwFiberStat\t0x%x\n" + "HwSwitchMdiToFiber\t0x%x\n" + "NicCustLedValue\t0x%x\n" + "RequiredSecLanDonglePatch\t0x%x\n" + "HwSuppDashVer\t0x%x\n" + "DASH\t0x%x\n" + "dash_printer_enabled\t0x%x\n" + "HwSuppKCPOffloadVer\t0x%x\n" + "speed_mode\t0x%x\n" + "duplex_mode\t0x%x\n" + "autoneg_mode\t0x%x\n" + "aspm\t0x%x\n" + "s5wol\t0x%x\n" + "s5_keep_curr_mac\t0x%x\n" + "eee_enable\t0x%x\n" + "hwoptimize\t0x%lx\n" + "proc_init_num\t0x%x\n" + "s0_magic_packet\t0x%x\n" + "disable_wol_support\t0x%x\n" + "enable_double_vlan\t0x%x\n" + "eee_giga_lite\t0x%x\n" + "HwSuppMagicPktVer\t0x%x\n" + "HwSuppLinkChgWakeUpVer\t0x%x\n" + "HwSuppD0SpeedUpVer\t0x%x\n" + "D0SpeedUpSpeed\t0x%x\n" + "HwSuppCheckPhyDisableModeVer\t0x%x\n" + "HwPkgDet\t0x%x\n" + "HwSuppTxNoCloseVer\t0x%x\n" + "EnableTxNoClose\t0x%x\n" + "NextHwDesCloPtr0\t0x%x\n" + "BeginHwDesCloPtr0\t0x%x\n" + "hw_clo_ptr_reg0\t0x%x\n" + "sw_tail_ptr_reg0\t0x%x\n" + "NextHwDesCloPtr1\t0x%x\n" + "BeginHwDesCloPtr1\t0x%x\n" + "hw_clo_ptr_reg1\t0x%x\n" + "sw_tail_ptr_reg1\t0x%x\n" + "InitRxDescType\t0x%x\n" + "RxDescLength\t0x%x\n" + "num_rx_rings\t0x%x\n" + "num_tx_rings\t0x%x\n" + "tot_rx_rings\t0x%x\n" + "tot_tx_rings\t0x%x\n" + "HwSuppNumRxQueues\t0x%x\n" + "HwSuppNumTxQueues\t0x%x\n" + "EnableRss\t0x%x\n" + "EnablePtp\t0x%x\n" + "min_irq_nvecs\t0x%x\n" + "irq_nvecs\t0x%x\n" + "hw_supp_irq_nvecs\t0x%x\n" + "ring_lib_enabled\t0x%x\n" + "HwSuppIsrVer\t0x%x\n" + "HwCurrIsrVer\t0x%x\n" + "HwSuppMacMcuVer\t0x%x\n" + "MacMcuPageSize\t0x%x\n" + "hw_mcu_patch_code_ver\t0x%llx\n" + "bin_mcu_patch_code_ver\t0x%llx\n" +#ifdef ENABLE_PTP_SUPPORT + "tx_hwtstamp_timeouts\t0x%x\n" + "tx_hwtstamp_skipped\t0x%x\n" +#endif + "random_mac\t0x%x\n" + "org_mac_addr\t%pM\n" +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + "perm_addr\t%pM\n" +#endif + "dev_addr\t%pM\n", + MODULENAME, + RTL8127_VERSION, + tp->mcfg, + tp->chipset, + rtl_chip_info[tp->chipset].name, + dev->mtu, + tp->rx_ring[0].num_rx_desc, + tp->rx_ring[0].cur_rx, + tp->rx_ring[0].dirty_rx, + tp->rx_ring[1].cur_rx, + tp->rx_ring[1].dirty_rx, + tp->rx_ring[2].cur_rx, + tp->rx_ring[2].dirty_rx, + tp->rx_ring[3].cur_rx, + tp->rx_ring[3].dirty_rx, + tp->tx_ring[0].num_tx_desc, + tp->tx_ring[0].cur_tx, + tp->tx_ring[0].dirty_tx, + tp->tx_ring[1].cur_tx, + tp->tx_ring[1].dirty_tx, + tp->rx_buf_sz, +#ifdef ENABLE_PAGE_REUSE + tp->rx_buf_page_order, + tp->rx_buf_page_size, + tp->page_reuse_fail_cnt, +#endif //ENABLE_PAGE_REUSE + tp->esd_flag, + tp->pci_cfg_is_read, + tp->rtl8127_rx_config, + tp->cp_cmd, + tp->intr_mask, + tp->timer_intr_mask, + tp->wol_enabled, + tp->wol_opts, + tp->efuse_ver, + tp->eeprom_type, + tp->autoneg, + tp->duplex, + tp->speed, + tp->advertising, + tp->eeprom_len, + tp->cur_page, + tp->features, + tp->org_pci_offset_99, + tp->org_pci_offset_180, + tp->issue_offset_99_event, + tp->org_pci_offset_80, + tp->org_pci_offset_81, + tp->use_timer_interrupt, + tp->HwIcVerUnknown, + tp->NotWrRamCodeToMicroP, + tp->NotWrMcuPatchCode, + tp->HwHasWrRamCodeToMicroP, + tp->sw_ram_code_ver, + tp->hw_ram_code_ver, + tp->rtk_enable_diag, + tp->ShortPacketSwChecksum, + tp->UseSwPaddingShortPkt, + tp->RequireAdcBiasPatch, + tp->AdcBiasPatchIoffset, + tp->RequireAdjustUpsTxLinkPulseTiming, + tp->SwrCnt1msIni, + tp->HwSuppNowIsOobVer, + tp->HwFiberModeVer, + tp->HwFiberStat, + tp->HwSwitchMdiToFiber, + tp->NicCustLedValue, + tp->RequiredSecLanDonglePatch, + tp->HwSuppDashVer, + tp->DASH, + tp->dash_printer_enabled, + tp->HwSuppKCPOffloadVer, + speed_mode, + duplex_mode, + autoneg_mode, + aspm, + s5wol, + s5_keep_curr_mac, + tp->eee.eee_enabled, + hwoptimize, + proc_init_num, + s0_magic_packet, + disable_wol_support, + enable_double_vlan, + eee_giga_lite, + tp->HwSuppMagicPktVer, + tp->HwSuppLinkChgWakeUpVer, + tp->HwSuppD0SpeedUpVer, + tp->D0SpeedUpSpeed, + tp->HwSuppCheckPhyDisableModeVer, + tp->HwPkgDet, + tp->HwSuppTxNoCloseVer, + tp->EnableTxNoClose, + tp->tx_ring[0].NextHwDesCloPtr, + tp->tx_ring[0].BeginHwDesCloPtr, + rtl8127_get_hw_clo_ptr(&tp->tx_ring[0]), + rtl8127_get_sw_tail_ptr(&tp->tx_ring[0]), + tp->tx_ring[1].NextHwDesCloPtr, + tp->tx_ring[1].BeginHwDesCloPtr, + rtl8127_get_hw_clo_ptr(&tp->tx_ring[1]), + rtl8127_get_sw_tail_ptr(&tp->tx_ring[1]), + tp->InitRxDescType, + tp->RxDescLength, + tp->num_rx_rings, + tp->num_tx_rings, + rtl8127_tot_rx_rings(tp), + rtl8127_tot_tx_rings(tp), + tp->HwSuppNumRxQueues, + tp->HwSuppNumTxQueues, + tp->EnableRss, + tp->EnablePtp, + tp->min_irq_nvecs, + tp->irq_nvecs, + tp->hw_supp_irq_nvecs, + tp->ring_lib_enabled, + tp->HwSuppIsrVer, + tp->HwCurrIsrVer, + tp->HwSuppMacMcuVer, + tp->MacMcuPageSize, + tp->hw_mcu_patch_code_ver, + tp->bin_mcu_patch_code_ver, +#ifdef ENABLE_PTP_SUPPORT + tp->tx_hwtstamp_timeouts, + tp->tx_hwtstamp_skipped, +#endif + tp->random_mac, + tp->org_mac_addr, +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + dev->perm_addr, +#endif + dev->dev_addr); + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_tally_counter(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + struct rtl8127_counters *counters; + dma_addr_t paddr; + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump Tally Counter\n"); + + rtnl_lock(); + + counters = tp->tally_vaddr; + paddr = tp->tally_paddr; + if (!counters) { + len += snprintf(page + len, count - len, + "\nDump Tally Counter Fail\n"); + goto out_unlock; + } + + rtl8127_dump_tally_counter(tp, paddr); + + len += snprintf(page + len, count - len, + "Statistics\tValue\n----------\t-----\n"); + + len += snprintf(page + len, count - len, + "tx_packets\t%lld\n" + "rx_packets\t%lld\n" + "tx_errors\t%lld\n" + "rx_errors\t%d\n" + "rx_missed\t%d\n" + "align_errors\t%d\n" + "tx_one_collision\t%d\n" + "tx_multi_collision\t%d\n" + "rx_unicast\t%lld\n" + "rx_broadcast\t%lld\n" + "rx_multicast\t%d\n" + "tx_aborted\t%d\n" + "tx_underrun\t%d\n" + + "tx_octets\t%lld\n" + "rx_octets\t%lld\n" + "rx_multicast64\t%lld\n" + "tx_unicast64\t%lld\n" + "tx_broadcast64\t%lld\n" + "tx_multicast64\t%lld\n" + "tx_pause_on\t%d\n" + "tx_pause_off\t%d\n" + "tx_pause_all\t%d\n" + "tx_deferred\t%d\n" + "tx_late_collision\t%d\n" + "tx_all_collision\t%d\n" + "tx_aborted32\t%d\n" + "align_errors32\t%d\n" + "rx_frame_too_long\t%d\n" + "rx_runt\t%d\n" + "rx_pause_on\t%d\n" + "rx_pause_off\t%d\n" + "rx_pause_all\t%d\n" + "rx_unknown_opcode\t%d\n" + "rx_mac_error\t%d\n" + "tx_underrun32\t%d\n" + "rx_mac_missed\t%d\n" + "rx_tcam_dropped\t%d\n" + "tdu\t%d\n" + "rdu\t%d\n", + le64_to_cpu(counters->tx_packets), + le64_to_cpu(counters->rx_packets), + le64_to_cpu(counters->tx_errors), + le32_to_cpu(counters->rx_errors), + le16_to_cpu(counters->rx_missed), + le16_to_cpu(counters->align_errors), + le32_to_cpu(counters->tx_one_collision), + le32_to_cpu(counters->tx_multi_collision), + le64_to_cpu(counters->rx_unicast), + le64_to_cpu(counters->rx_broadcast), + le32_to_cpu(counters->rx_multicast), + le16_to_cpu(counters->tx_aborted), + le16_to_cpu(counters->tx_underrun), + + le64_to_cpu(counters->tx_octets), + le64_to_cpu(counters->rx_octets), + le64_to_cpu(counters->rx_multicast64), + le64_to_cpu(counters->tx_unicast64), + le64_to_cpu(counters->tx_broadcast64), + le64_to_cpu(counters->tx_multicast64), + le32_to_cpu(counters->tx_pause_on), + le32_to_cpu(counters->tx_pause_off), + le32_to_cpu(counters->tx_pause_all), + le32_to_cpu(counters->tx_deferred), + le32_to_cpu(counters->tx_late_collision), + le32_to_cpu(counters->tx_all_collision), + le32_to_cpu(counters->tx_aborted32), + le32_to_cpu(counters->align_errors32), + le32_to_cpu(counters->rx_frame_too_long), + le32_to_cpu(counters->rx_runt), + le32_to_cpu(counters->rx_pause_on), + le32_to_cpu(counters->rx_pause_off), + le32_to_cpu(counters->rx_pause_all), + le32_to_cpu(counters->rx_unknown_opcode), + le32_to_cpu(counters->rx_mac_error), + le32_to_cpu(counters->tx_underrun32), + le32_to_cpu(counters->rx_mac_missed), + le32_to_cpu(counters->rx_tcam_dropped), + le32_to_cpu(counters->tdu), + le32_to_cpu(counters->rdu)); + + len += snprintf(page + len, count - len, "\n"); +out_unlock: + rtnl_unlock(); + + *eof = 1; + return len; +} + +static int proc_get_registers(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max = R8127_MAC_REGS_SIZE; + u8 byte_rd; + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump MAC Registers\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%04x:\t", + n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + len += snprintf(page + len, count - len, + "%02x ", + byte_rd); + } + } + + max = 0xB00; + for (n = 0xA00; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%04x:\t", + n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + len += snprintf(page + len, count - len, + "%02x ", + byte_rd); + } + } + + max = 0xD40; + for (n = 0xD00; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%04x:\t", + n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + len += snprintf(page + len, count - len, + "%02x ", + byte_rd); + } + } + + max = 0x2840; + for (n = 0x2800; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%04x:\t", + n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + len += snprintf(page + len, count - len, + "%02x ", + byte_rd); + } + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_all_registers(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max; + u8 byte_rd; + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + struct pci_dev *pdev = tp->pci_dev; + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump All MAC Registers\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + max = pci_resource_len(pdev, 2); + + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%04x:\t", + n); + + for (i = 0; i < 16 && n < max; i++, n++) { + byte_rd = readb(ioaddr + n); + len += snprintf(page + len, count - len, + "%02x ", + byte_rd); + } + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\nTotal length:0x%X", max); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_pcie_phy(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max = R8127_EPHY_REGS_SIZE/2; + u16 word_rd; + struct rtl8127_private *tp = netdev_priv(dev); + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump PCIE PHY\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + + for (i = 0; i < 8 && n < max; i++, n++) { + word_rd = rtl8127_ephy_read(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_eth_phy(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max = R8127_PHY_REGS_SIZE/2; + u16 word_rd; + struct rtl8127_private *tp = netdev_priv(dev); + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump Ethernet PHY\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + len += snprintf(page + len, count - len, + "\n####################page 0##################\n"); + rtl8127_mdio_write(tp, 0x1f, 0x0000); + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + + for (i = 0; i < 8 && n < max; i++, n++) { + word_rd = rtl8127_mdio_read(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + } + + len += snprintf(page + len, count - len, + "\n####################extra reg##################\n"); + n = 0xA400; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + for (i = 0; i < 8; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + + n = 0xA410; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + for (i = 0; i < 3; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + + n = 0xA434; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + + n = 0xA5D0; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + for (i = 0; i < 4; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + + n = 0xA61A; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + + n = 0xA6D0; + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + for (i = 0; i < 3; i++, n+=2) { + word_rd = rtl8127_mdio_direct_read_phy_ocp(tp, n); + len += snprintf(page + len, count - len, + "%04x ", + word_rd); + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_extended_registers(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max = R8127_ERI_REGS_SIZE; + u32 dword_rd; + struct rtl8127_private *tp = netdev_priv(dev); + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump Extended Registers\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%02x:\t", + n); + + for (i = 0; i < 4 && n < max; i++, n+=4) { + dword_rd = rtl8127_eri_read(tp, n, 4, ERIAR_ExGMAC); + len += snprintf(page + len, count - len, + "%08x ", + dword_rd); + } + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_pci_registers(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + int i, n, max = R8127_PCI_REGS_SIZE; + u32 dword_rd; + struct rtl8127_private *tp = netdev_priv(dev); + int len = 0; + + len += snprintf(page + len, count - len, + "\nDump PCI Registers\n" + "Offset\tValue\n------\t-----\n"); + + rtnl_lock(); + + for (n = 0; n < max;) { + len += snprintf(page + len, count - len, + "\n0x%03x:\t", + n); + + for (i = 0; i < 4 && n < max; i++, n+=4) { + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + len += snprintf(page + len, count - len, + "%08x ", + dword_rd); + } + } + + n = 0x110; + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + len += snprintf(page + len, count - len, + "\n0x%03x:\t%08x ", + n, + dword_rd); + n = 0x70c; + pci_read_config_dword(tp->pci_dev, n, &dword_rd); + len += snprintf(page + len, count - len, + "\n0x%03x:\t%08x ", + n, + dword_rd); + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return len; +} + +static int proc_get_temperature(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + u16 ts_digout, tj, fah; + int len = 0; + + len += snprintf(page + len, count - len, + "\nChip Temperature\n"); + + rtnl_lock(); + + if (!rtl8127_sysfs_testmode_on(tp)) { + len += snprintf(page + len, count - len, + "\nPlease turn on ""/sys/class/net//rtk_adv/testmode"".\n\n"); + goto out_unlock; + } + + ts_digout = rtl8127_read_thermal_sensor(tp); + + tj = ts_digout / 2; + if (ts_digout <= 512) { + tj = ts_digout / 2; + len += snprintf(page + len, count - len, + "Cel:%d\n", + tj); + fah = tj * (9/5) + 32; + len += snprintf(page + len, count - len, + "Fah:%d\n", + fah); + + } else { + tj = (512 - ((ts_digout / 2) - 512)) / 2; + len += snprintf(page + len, count - len, + "Cel:-%d\n", + tj); + fah = tj * (9/5) + 32; + len += snprintf(page + len, count - len, + "Fah:-%d\n", + fah); + } + + len += snprintf(page + len, count - len, "\n"); + +out_unlock: + rtnl_unlock(); + + *eof = 1; + return len; +} + +static int _proc_get_cable_info(char *page, char **start, + off_t offset, int count, + int *eof, void *data, + bool poe_mode) +{ + int i; + u16 status; + int len = 0; + struct net_device *dev = data; + int cp_status[RTL8127_CP_NUM] = {0}; + int cp_len[RTL8127_CP_NUM] = {0}; + struct rtl8127_private *tp = netdev_priv(dev); + const char *pair_str[RTL8127_CP_NUM] = {"1-2", "3-6", "4-5", "7-8"}; + + switch (tp->mcfg) { + default: + return -EOPNOTSUPP; + } + + spin_lock_irqsave(&tp->phy_lock, flags); + + if (!rtl8127_sysfs_testmode_on(tp)) { + len += snprintf(page + len, count - len, + "\nPlease turn on ""/sys/class/net//rtk_adv/testmode"".\n\n"); + goto out_unlock; + } + + status = RTL_R16(tp, PHYstatus); + if (status & LinkStatus) + len += snprintf(page + len, count - len, + "\nlink speed:%d", + rtl8127_convert_link_speed(status)); + else + len += snprintf(page + len, count - len, + "\nlink status:off"); + + rtl8127_get_cp_len(tp, cp_len); + + rtl8127_get_cp_status(tp, cp_status, poe_mode); + + len += snprintf(page + len, count - len, + "\npair\tlength\tstatus \tpp\n"); + + for (i=0; iphy_lock, flags); + + *eof = 1; + return len; +} + +static int proc_get_cable_info(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + return _proc_get_cable_info(page, start, offset, count, eof, data, 0); +} + +static int proc_get_poe_cable_info(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + return _proc_get_cable_info(page, start, offset, count, eof, data, 1); +} + +static void _proc_dump_desc(char *page, int *page_len, int *count, void *desc_base, + u32 alloc_size) +{ + u32 *pdword; + int i, len; + + if (desc_base == NULL || + alloc_size == 0) + return; + + len = *page_len; + pdword = (u32*)desc_base; + for (i=0; i<(alloc_size/4); i++) { + if (!(i % 4)) + len += snprintf(page + len, *count - len, + "\n%04x ", + i); + len += snprintf(page + len, *count - len, + "%08x ", + pdword[i]); + } + + len += snprintf(page + len, *count - len, "\n"); + + *page_len = len; + return; +} + +static int proc_dump_rx_desc(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + int i; + int len = 0; + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + + rtnl_lock(); + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + + if (!ring) + continue; + + len += snprintf(page + len, count - len, + "\ndump rx %d desc:%d", + i, ring->num_rx_desc); + + _proc_dump_desc(page, &len, &count, + ring->RxDescArray, + ring->RxDescAllocSize); + } + +#ifdef ENABLE_LIB_SUPPORT + if (rtl8127_num_lib_rx_rings(tp) > 0) { + for (i = 0; i < tp->HwSuppNumRxQueues; i++) { + struct rtl8127_ring *lib_ring = &tp->lib_rx_ring[i]; + if (lib_ring->enabled) { + len += snprintf(page + len, count - len, + "\ndump lib rx %d desc:%d", + i, + ring->ring_size); + _proc_dump_desc(page, &len, &count, + (void*)lib_ring->desc_addr, + lib_ring->desc_size); + } + } + } +#endif //ENABLE_LIB_SUPPORT + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + + return len; +} + +static int proc_dump_tx_desc(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + int len = 0; + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + rtnl_lock(); + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + + if (!ring) + continue; + + len += snprintf(page + len, count - len, + "\ndump tx desc:%d", + ring->num_tx_desc); + + _proc_dump_desc(page, &len, &count, + ring->TxDescArray, + ring->TxDescAllocSize); + } + +#ifdef ENABLE_LIB_SUPPORT + if (rtl8127_num_lib_tx_rings(tp) > 0) { + for (i = 0; i < tp->HwSuppNumTxQueues; i++) { + struct rtl8127_ring *lib_ring = &tp->lib_tx_ring[i]; + if (lib_ring->enabled) { + len += snprintf(page + len, count - len, + "\ndump lib tx %d desc:%d", + i, + ring->ring_size); + _proc_dump_desc(page, &len, &count, + (void*)lib_ring->desc_addr, + lib_ring->desc_size); + } + } + } +#endif //ENABLE_LIB_SUPPORT + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + + return len; +} + +static int proc_dump_msix_tbl(char *page, char **start, + off_t offset, int count, + int *eof, void *data) +{ + int i, j; + int len = 0; + void __iomem *ioaddr; + struct net_device *dev = data; + struct rtl8127_private *tp = netdev_priv(dev); + + /* ioremap MMIO region */ + ioaddr = ioremap(pci_resource_start(tp->pci_dev, 4), pci_resource_len(tp->pci_dev, 4)); + if (!ioaddr) + return -EFAULT; + + rtnl_lock(); + + len += snprintf(page + len, count - len, + "\ndump MSI-X Table. Total Entry %d. \n", + tp->hw_supp_irq_nvecs); + + for (i=0; ihw_supp_irq_nvecs; i++) { + len += snprintf(page + len, count - len, + "\n%04x ", i); + for (j=0; j<4; j++) + len += snprintf(page + len, count - len, "%08x ", + readl(ioaddr + i*0x10 + 4*j)); + } + + rtnl_unlock(); + + len += snprintf(page + len, count - len, "\n"); + + *eof = 1; + return 0; +} + +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + +static void rtl8127_proc_module_init(void) +{ + //create /proc/net/r8127 +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,32) + rtl8127_proc = proc_mkdir(MODULENAME, init_net.proc_net); +#else + rtl8127_proc = proc_mkdir(MODULENAME, proc_net); +#endif + if (!rtl8127_proc) + dprintk("cannot create %s proc entry \n", MODULENAME); +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) +/* + * seq_file wrappers for procfile show routines. + */ +static int rtl8127_proc_open(struct inode *inode, struct file *file) +{ + struct net_device *dev = proc_get_parent_data(inode); +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) + int (*show)(struct seq_file *, void *) = pde_data(inode); +#else + int (*show)(struct seq_file *, void *) = PDE_DATA(inode); +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) + + return single_open(file, show, dev); +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,6,0) +static const struct proc_ops rtl8127_proc_fops = { + .proc_open = rtl8127_proc_open, + .proc_read = seq_read, + .proc_lseek = seq_lseek, + .proc_release = single_release, +}; +#else +static const struct file_operations rtl8127_proc_fops = { + .open = rtl8127_proc_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; +#endif + +#endif + +/* + * Table of proc files we need to create. + */ +struct rtl8127_proc_file { + char name[16]; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + int (*show)(struct seq_file *, void *); +#else + int (*show)(char *, char **, off_t, int, int *, void *); +#endif +}; + +static const struct rtl8127_proc_file rtl8127_debug_proc_files[] = { + { "driver_var", &proc_get_driver_variable }, + { "tally", &proc_get_tally_counter }, + { "registers", &proc_get_registers }, + { "registers2", &proc_get_all_registers }, + { "pcie_phy", &proc_get_pcie_phy }, + { "eth_phy", &proc_get_eth_phy }, + { "ext_regs", &proc_get_extended_registers }, + { "pci_regs", &proc_get_pci_registers }, + { "tx_desc", &proc_dump_tx_desc }, + { "rx_desc", &proc_dump_rx_desc }, + { "msix_tbl", &proc_dump_msix_tbl }, + { "", NULL } +}; + +static const struct rtl8127_proc_file rtl8127_test_proc_files[] = { + { "temp", &proc_get_temperature }, + { "cdt", &proc_get_cable_info }, + { "cdt_poe", &proc_get_poe_cable_info }, + { "", NULL } +}; + +#define R8127_PROC_DEBUG_DIR "debug" +#define R8127_PROC_TEST_DIR "test" + +static void rtl8127_proc_init(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + const struct rtl8127_proc_file *f; + struct proc_dir_entry *dir; + + if (!rtl8127_proc) + return; + + if (tp->proc_dir_debug || tp->proc_dir_test) + return; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + dir = proc_mkdir_data(dev->name, 0, rtl8127_proc, dev); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s\n", + MODULENAME, dev->name); + return; + } + tp->proc_dir = dir; + proc_init_num++; + + /* create debug entry */ + dir = proc_mkdir_data(R8127_PROC_DEBUG_DIR, 0, tp->proc_dir, dev); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_DEBUG_DIR); + return; + } + + tp->proc_dir_debug = dir; + for (f = rtl8127_debug_proc_files; f->name[0]; f++) { + if (!proc_create_data(f->name, S_IFREG | S_IRUGO, dir, + &rtl8127_proc_fops, f->show)) { + printk("Unable to initialize " + "/proc/net/%s/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_DEBUG_DIR, + f->name); + return; + } + } + + /* create test entry */ + dir = proc_mkdir_data(R8127_PROC_TEST_DIR, 0, tp->proc_dir, dev); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_TEST_DIR); + return; + } + + tp->proc_dir_test = dir; + for (f = rtl8127_test_proc_files; f->name[0]; f++) { + if (!proc_create_data(f->name, S_IFREG | S_IRUGO, dir, + &rtl8127_proc_fops, f->show)) { + printk("Unable to initialize " + "/proc/net/%s/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_TEST_DIR, + f->name); + return; + } + } +#else + dir = proc_mkdir(dev->name, rtl8127_proc); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s\n", + MODULENAME, dev->name); + return; + } + + tp->proc_dir = dir; + proc_init_num++; + + /* create debug entry */ + dir = proc_mkdir(R8127_PROC_DEBUG_DIR, tp->proc_dir); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_DEBUG_DIR); + return; + } + + tp->proc_dir_debug = dir; + for (f = rtl8127_debug_proc_files; f->name[0]; f++) { + if (!create_proc_read_entry(f->name, S_IFREG | S_IRUGO, + dir, f->show, dev)) { + printk("Unable to initialize " + "/proc/net/%s/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_DEBUG_DIR, + f->name); + return; + } + } + + /* create test entry */ + dir = proc_mkdir(R8127_PROC_TEST_DIR, tp->proc_dir); + if (!dir) { + printk("Unable to initialize /proc/net/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_TEST_DIR); + return; + } + + tp->proc_dir_test = dir; + for (f = rtl8127_test_proc_files; f->name[0]; f++) { + if (!create_proc_read_entry(f->name, S_IFREG | S_IRUGO, + dir, f->show, dev)) { + printk("Unable to initialize " + "/proc/net/%s/%s/%s/%s\n", + MODULENAME, dev->name, R8127_PROC_TEST_DIR, + f->name); + return; + } + } +#endif +} + +static void rtl8127_proc_remove(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->proc_dir) { +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + remove_proc_subtree(dev->name, rtl8127_proc); +#else + const struct rtl8127_proc_file *f; + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->proc_dir_debug) { + for (f = rtl8127_debug_proc_files; f->name[0]; f++) + remove_proc_entry(f->name, tp->proc_dir_debug); + remove_proc_entry(R8127_PROC_DEBUG_DIR, tp->proc_dir); + } + + if (tp->proc_dir_test) { + for (f = rtl8127_test_proc_files; f->name[0]; f++) + remove_proc_entry(f->name, tp->proc_dir_test); + remove_proc_entry(R8127_PROC_TEST_DIR, tp->proc_dir); + } + + remove_proc_entry(dev->name, rtl8127_proc); +#endif + proc_init_num--; + + tp->proc_dir_debug = NULL; + tp->proc_dir_test = NULL; + tp->proc_dir = NULL; + } +} + +#endif //ENABLE_R8127_PROCFS + +#ifdef ENABLE_R8127_SYSFS +/**************************************************************************** +* -----------------------------SYSFS STUFF------------------------- +***************************************************************************** +*/ +static ssize_t testmode_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct net_device *netdev = to_net_dev(dev); + struct rtl8127_private *tp = netdev_priv(netdev); + + sprintf(buf, "%u\n", tp->testmode); + + return strlen(buf); +} + +static ssize_t testmode_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct net_device *netdev = to_net_dev(dev); + struct rtl8127_private *tp = netdev_priv(netdev); + u32 testmode; + + if (sscanf(buf, "%u\n", &testmode) != 1) + return -EINVAL; + + if (tp->testmode != testmode) { + rtnl_lock(); + tp->testmode = testmode; + rtnl_unlock(); + } + + return count; +} + +static DEVICE_ATTR_RW(testmode); + +static struct attribute *rtk_adv_attrs[] = { + &dev_attr_testmode.attr, + NULL +}; + +static struct attribute_group rtk_adv_grp = { + .name = "rtl_adv", + .attrs = rtk_adv_attrs, +}; + +static void rtl8127_sysfs_init(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret; + + /* init rtl_adv */ +#ifdef ENABLE_LIB_SUPPORT + tp->testmode = 0; +#else + tp->testmode = 1; +#endif //ENABLE_LIB_SUPPORT + + ret = sysfs_create_group(&dev->dev.kobj, &rtk_adv_grp); + if (ret < 0) + netif_warn(tp, probe, dev, "create rtk_adv_grp fail\n"); + else + set_bit(R8127_SYSFS_RTL_ADV, tp->sysfs_flag); +} + +static void rtl8127_sysfs_remove(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (test_and_clear_bit(R8127_SYSFS_RTL_ADV, tp->sysfs_flag)) + sysfs_remove_group(&dev->dev.kobj, &rtk_adv_grp); +} +#endif //ENABLE_R8127_SYSFS + +static inline u16 map_phy_ocp_addr(u16 PageNum, u8 RegNum) +{ + u16 OcpPageNum = 0; + u8 OcpRegNum = 0; + u16 OcpPhyAddress = 0; + + if (PageNum == 0) { + OcpPageNum = OCP_STD_PHY_BASE_PAGE + (RegNum / 8); + OcpRegNum = 0x10 + (RegNum % 8); + } else { + OcpPageNum = PageNum; + OcpRegNum = RegNum; + } + + OcpPageNum <<= 4; + + if (OcpRegNum < 16) { + OcpPhyAddress = 0; + } else { + OcpRegNum -= 16; + OcpRegNum <<= 1; + + OcpPhyAddress = OcpPageNum + OcpRegNum; + } + + + return OcpPhyAddress; +} + +static void mdio_real_direct_write_phy_ocp(struct rtl8127_private *tp, + u16 RegAddr, + u16 value) +{ + u32 data32; + int i; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,18) + WARN_ON_ONCE(RegAddr % 2); +#endif + data32 = RegAddr/2; + data32 <<= OCPR_Addr_Reg_shift; + data32 |= OCPR_Write | value; + + RTL_W32(tp, PHYOCP, data32); + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + udelay(R8127_CHANNEL_WAIT_TIME); + + if (!(RTL_R32(tp, PHYOCP) & OCPR_Flag)) + break; + } +} + +void rtl8127_mdio_direct_write_phy_ocp(struct rtl8127_private *tp, + u16 RegAddr, + u16 value) +{ + if (tp->rtk_enable_diag) + return; + + mdio_real_direct_write_phy_ocp(tp, RegAddr, value); +} + +/* +static void rtl8127_mdio_write_phy_ocp(struct rtl8127_private *tp, + u16 PageNum, + u32 RegAddr, + u32 value) +{ + u16 ocp_addr; + + ocp_addr = map_phy_ocp_addr(PageNum, RegAddr); + + rtl8127_mdio_direct_write_phy_ocp(tp, ocp_addr, value); +} +*/ + +static void rtl8127_mdio_real_write_phy_ocp(struct rtl8127_private *tp, + u16 PageNum, + u32 RegAddr, + u32 value) +{ + u16 ocp_addr; + + ocp_addr = map_phy_ocp_addr(PageNum, RegAddr); + + mdio_real_direct_write_phy_ocp(tp, ocp_addr, value); +} + +static void mdio_real_write(struct rtl8127_private *tp, + u16 RegAddr, + u16 value) +{ + if (RegAddr == 0x1F) { + tp->cur_page = value; + return; + } + rtl8127_mdio_real_write_phy_ocp(tp, tp->cur_page, RegAddr, value); +} + +void rtl8127_mdio_write(struct rtl8127_private *tp, + u16 RegAddr, + u16 value) +{ + if (tp->rtk_enable_diag) + return; + + mdio_real_write(tp, RegAddr, value); +} + +void rtl8127_mdio_prot_write(struct rtl8127_private *tp, + u32 RegAddr, + u32 value) +{ + mdio_real_write(tp, RegAddr, value); +} + +void rtl8127_mdio_prot_direct_write_phy_ocp(struct rtl8127_private *tp, + u32 RegAddr, + u32 value) +{ + mdio_real_direct_write_phy_ocp(tp, RegAddr, value); +} + +static u32 mdio_real_direct_read_phy_ocp(struct rtl8127_private *tp, + u16 RegAddr) +{ + u32 data32; + int i, value = 0; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,18) + WARN_ON_ONCE(RegAddr % 2); +#endif + data32 = RegAddr/2; + data32 <<= OCPR_Addr_Reg_shift; + + RTL_W32(tp, PHYOCP, data32); + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + udelay(R8127_CHANNEL_WAIT_TIME); + + if (RTL_R32(tp, PHYOCP) & OCPR_Flag) + break; + } + value = RTL_R32(tp, PHYOCP) & OCPDR_Data_Mask; + + return value; +} + +u32 rtl8127_mdio_direct_read_phy_ocp(struct rtl8127_private *tp, + u16 RegAddr) +{ + if (tp->rtk_enable_diag) + return 0xffffffff; + + return mdio_real_direct_read_phy_ocp(tp, RegAddr); +} + +/* +static u32 rtl8127_mdio_read_phy_ocp(struct rtl8127_private *tp, + u16 PageNum, + u32 RegAddr) +{ + u16 ocp_addr; + + ocp_addr = map_phy_ocp_addr(PageNum, RegAddr); + + return rtl8127_mdio_direct_read_phy_ocp(tp, ocp_addr); +} +*/ + +static u32 rtl8127_mdio_real_read_phy_ocp(struct rtl8127_private *tp, + u16 PageNum, + u32 RegAddr) +{ + u16 ocp_addr; + + ocp_addr = map_phy_ocp_addr(PageNum, RegAddr); + + return mdio_real_direct_read_phy_ocp(tp, ocp_addr); +} + +static u32 mdio_real_read(struct rtl8127_private *tp, + u16 RegAddr) +{ + return rtl8127_mdio_real_read_phy_ocp(tp, tp->cur_page, RegAddr); +} + +u32 rtl8127_mdio_read(struct rtl8127_private *tp, + u16 RegAddr) +{ + if (tp->rtk_enable_diag) + return 0xffffffff; + + return mdio_real_read(tp, RegAddr); +} + +u32 rtl8127_mdio_prot_read(struct rtl8127_private *tp, + u32 RegAddr) +{ + return mdio_real_read(tp, RegAddr); +} + +u32 rtl8127_mdio_prot_direct_read_phy_ocp(struct rtl8127_private *tp, + u32 RegAddr) +{ + return mdio_real_direct_read_phy_ocp(tp, RegAddr); +} + +static void rtl8127_clear_and_set_eth_phy_bit(struct rtl8127_private *tp, u8 addr, u16 clearmask, u16 setmask) +{ + u16 PhyRegValue; + + PhyRegValue = rtl8127_mdio_read(tp, addr); + PhyRegValue &= ~clearmask; + PhyRegValue |= setmask; + rtl8127_mdio_write(tp, addr, PhyRegValue); +} + +void rtl8127_clear_eth_phy_bit(struct rtl8127_private *tp, u8 addr, u16 mask) +{ + rtl8127_clear_and_set_eth_phy_bit(tp, + addr, + mask, + 0); +} + +void rtl8127_set_eth_phy_bit(struct rtl8127_private *tp, u8 addr, u16 mask) +{ + rtl8127_clear_and_set_eth_phy_bit(tp, + addr, + 0, + mask); +} + +void rtl8127_clear_and_set_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 clearmask, u16 setmask) +{ + u16 PhyRegValue; + + PhyRegValue = rtl8127_mdio_direct_read_phy_ocp(tp, addr); + PhyRegValue &= ~clearmask; + PhyRegValue |= setmask; + rtl8127_mdio_direct_write_phy_ocp(tp, addr, PhyRegValue); +} + +void rtl8127_clear_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 mask) +{ + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + addr, + mask, + 0); +} + +void rtl8127_set_eth_phy_ocp_bit(struct rtl8127_private *tp, u16 addr, u16 mask) +{ + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + addr, + 0, + mask); +} + +void rtl8127_mac_ocp_write(struct rtl8127_private *tp, u16 reg_addr, u16 value) +{ + u32 data32; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,18) + WARN_ON_ONCE(reg_addr % 2); +#endif + + data32 = reg_addr/2; + data32 <<= OCPR_Addr_Reg_shift; + data32 += value; + data32 |= OCPR_Write; + + RTL_W32(tp, MACOCP, data32); +} + +u16 rtl8127_mac_ocp_read(struct rtl8127_private *tp, u16 reg_addr) +{ + u32 data32; + u16 data16 = 0; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,18) + WARN_ON_ONCE(reg_addr % 2); +#endif + + data32 = reg_addr/2; + data32 <<= OCPR_Addr_Reg_shift; + + RTL_W32(tp, MACOCP, data32); + data16 = (u16)RTL_R32(tp, MACOCP); + + return data16; +} + +#ifdef ENABLE_USE_FIRMWARE_FILE +static void mac_mcu_write(struct rtl8127_private *tp, u16 reg, u16 value) +{ + if (reg == 0x1f) { + tp->ocp_base = value << 4; + return; + } + + rtl8127_mac_ocp_write(tp, tp->ocp_base + reg, value); +} + +static u32 mac_mcu_read(struct rtl8127_private *tp, u16 reg) +{ + return rtl8127_mac_ocp_read(tp, tp->ocp_base + reg); +} +#endif + +static void +rtl8127_clear_set_mac_ocp_bit( + struct rtl8127_private *tp, + u16 addr, + u16 clearmask, + u16 setmask +) +{ + u16 PhyRegValue; + + PhyRegValue = rtl8127_mac_ocp_read(tp, addr); + PhyRegValue &= ~clearmask; + PhyRegValue |= setmask; + rtl8127_mac_ocp_write(tp, addr, PhyRegValue); +} + +void +rtl8127_clear_mac_ocp_bit( + struct rtl8127_private *tp, + u16 addr, + u16 mask +) +{ + rtl8127_clear_set_mac_ocp_bit(tp, + addr, + mask, + 0); +} + +static void +rtl8127_set_mac_ocp_bit( + struct rtl8127_private *tp, + u16 addr, + u16 mask +) +{ + rtl8127_clear_set_mac_ocp_bit(tp, + addr, + 0, + mask); +} + +u32 rtl8127_ocp_read_with_oob_base_address(struct rtl8127_private *tp, u16 addr, u8 len, const u32 base_address) +{ + return rtl8127_eri_read_with_oob_base_address(tp, addr, len, ERIAR_OOB, base_address); +} + +u32 rtl8127_ocp_read(struct rtl8127_private *tp, u16 addr, u8 len) +{ + u32 value = 0; + + if (!tp->AllowAccessDashOcp) + return 0xffffffff; + + if (HW_DASH_SUPPORT_TYPE_2(tp)) + value = rtl8127_ocp_read_with_oob_base_address(tp, addr, len, NO_BASE_ADDRESS); + else if (HW_DASH_SUPPORT_TYPE_3(tp)) + value = rtl8127_ocp_read_with_oob_base_address(tp, addr, len, RTL8168FP_OOBMAC_BASE); + + return value; +} + +u32 rtl8127_ocp_write_with_oob_base_address(struct rtl8127_private *tp, u16 addr, u8 len, u32 value, const u32 base_address) +{ + return rtl8127_eri_write_with_oob_base_address(tp, addr, len, value, ERIAR_OOB, base_address); +} + +void rtl8127_ocp_write(struct rtl8127_private *tp, u16 addr, u8 len, u32 value) +{ + if (!tp->AllowAccessDashOcp) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp)) + rtl8127_ocp_write_with_oob_base_address(tp, addr, len, value, NO_BASE_ADDRESS); + else if (HW_DASH_SUPPORT_TYPE_3(tp)) + rtl8127_ocp_write_with_oob_base_address(tp, addr, len, value, RTL8168FP_OOBMAC_BASE); +} + +void rtl8127_oob_mutex_lock(struct rtl8127_private *tp) +{ + u8 reg_16, reg_a0; + u32 wait_cnt_0, wait_Cnt_1; + u16 ocp_reg_mutex_ib; + u16 ocp_reg_mutex_oob; + u16 ocp_reg_mutex_prio; + + if (!tp->DASH) + return; + + switch (tp->mcfg) { + default: + return; + } + + rtl8127_ocp_write(tp, ocp_reg_mutex_ib, 1, BIT_0); + reg_16 = rtl8127_ocp_read(tp, ocp_reg_mutex_oob, 1); + wait_cnt_0 = 0; + while(reg_16) { + reg_a0 = rtl8127_ocp_read(tp, ocp_reg_mutex_prio, 1); + if (reg_a0) { + rtl8127_ocp_write(tp, ocp_reg_mutex_ib, 1, 0x00); + reg_a0 = rtl8127_ocp_read(tp, ocp_reg_mutex_prio, 1); + wait_Cnt_1 = 0; + while(reg_a0) { + reg_a0 = rtl8127_ocp_read(tp, ocp_reg_mutex_prio, 1); + + wait_Cnt_1++; + + if (wait_Cnt_1 > 2000) + break; + }; + rtl8127_ocp_write(tp, ocp_reg_mutex_ib, 1, BIT_0); + + } + reg_16 = rtl8127_ocp_read(tp, ocp_reg_mutex_oob, 1); + + wait_cnt_0++; + + if (wait_cnt_0 > 2000) + break; + }; +} + +void rtl8127_oob_mutex_unlock(struct rtl8127_private *tp) +{ + //u16 ocp_reg_mutex_ib; + //u16 ocp_reg_mutex_oob; + //u16 ocp_reg_mutex_prio; + + if (!tp->DASH) + return; + + switch (tp->mcfg) { + default: + return; + } + + //rtl8127_ocp_write(tp, ocp_reg_mutex_prio, 1, BIT_0); + //rtl8127_ocp_write(tp, ocp_reg_mutex_ib, 1, 0x00); +} + +static bool +rtl8127_is_allow_access_dash_ocp(struct rtl8127_private *tp) +{ + bool allow_access = false; + + if (!HW_DASH_SUPPORT_DASH(tp)) + goto exit; + + allow_access = true; + switch (tp->mcfg) { + default: + goto exit; + } +exit: + return allow_access; +} + +static int rtl8127_check_dash(struct rtl8127_private *tp) +{ + if (!tp->AllowAccessDashOcp) + return 0; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) { + if (rtl8127_ocp_read(tp, 0x128, 1) & BIT_0) + return 1; + } + + return 0; +} + +void rtl8127_dash2_disable_tx(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) { + u16 WaitCnt; + u8 TmpUchar; + + //Disable oob Tx + RTL_CMAC_W8(tp, CMAC_IBCR2, RTL_CMAC_R8(tp, CMAC_IBCR2) & ~(BIT_0)); + WaitCnt = 0; + + //wait oob tx disable + do { + TmpUchar = RTL_CMAC_R8(tp, CMAC_IBISR0); + + if (TmpUchar & ISRIMR_DASH_TYPE2_TX_DISABLE_IDLE) { + break; + } + + fsleep(50); + WaitCnt++; + } while(WaitCnt < 2000); + + //Clear ISRIMR_DASH_TYPE2_TX_DISABLE_IDLE + RTL_CMAC_W8(tp, CMAC_IBISR0, RTL_CMAC_R8(tp, CMAC_IBISR0) | ISRIMR_DASH_TYPE2_TX_DISABLE_IDLE); + } +} + +void rtl8127_dash2_enable_tx(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) + RTL_CMAC_W8(tp, CMAC_IBCR2, RTL_CMAC_R8(tp, CMAC_IBCR2) | BIT_0); +} + +void rtl8127_dash2_disable_rx(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) + RTL_CMAC_W8(tp, CMAC_IBCR0, RTL_CMAC_R8(tp, CMAC_IBCR0) & ~(BIT_0)); +} + +void rtl8127_dash2_enable_rx(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) + RTL_CMAC_W8(tp, CMAC_IBCR0, RTL_CMAC_R8(tp, CMAC_IBCR0) | BIT_0); +} + +static void rtl8127_dash2_disable_txrx(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) { + rtl8127_dash2_disable_tx(tp); + rtl8127_dash2_disable_rx(tp); + } +} + +static int rtl8127_wait_dash_fw_ready(struct rtl8127_private *tp) +{ + int rc = -1; + int timeout; + + if (HW_DASH_SUPPORT_TYPE_2(tp) == FALSE && + HW_DASH_SUPPORT_TYPE_3(tp) == FALSE) + goto out; + + if (!tp->DASH) + goto out; + + for (timeout = 0; timeout < 10; timeout++) { + fsleep(10000); + if (rtl8127_ocp_read(tp, 0x124, 1) & BIT_0) { + rc = 1; + goto out; + } + } + + rc = 0; + +out: + return rc; +} + +static void rtl8127_driver_start(struct rtl8127_private *tp) +{ + u32 tmp_value; + + if (HW_DASH_SUPPORT_TYPE_2(tp) == FALSE && + HW_DASH_SUPPORT_TYPE_3(tp) == FALSE) + return; + + if (!tp->AllowAccessDashOcp) + return; + + rtl8127_ocp_write(tp, 0x180, 1, OOB_CMD_DRIVER_START); + tmp_value = rtl8127_ocp_read(tp, 0x30, 1); + tmp_value |= BIT_0; + rtl8127_ocp_write(tp, 0x30, 1, tmp_value); + + rtl8127_wait_dash_fw_ready(tp); +} + +static void rtl8127_driver_stop(struct rtl8127_private *tp) +{ + u32 tmp_value; + struct net_device *dev = tp->dev; + + if (HW_DASH_SUPPORT_TYPE_2(tp) == FALSE && + HW_DASH_SUPPORT_TYPE_3(tp) == FALSE) + return; + + if (!tp->AllowAccessDashOcp) + return; + + rtl8127_dash2_disable_txrx(dev); + + rtl8127_ocp_write(tp, 0x180, 1, OOB_CMD_DRIVER_STOP); + tmp_value = rtl8127_ocp_read(tp, 0x30, 1); + tmp_value |= BIT_0; + rtl8127_ocp_write(tp, 0x30, 1, tmp_value); + + rtl8127_wait_dash_fw_ready(tp); +} + +static void _rtl8127_ephy_write(struct rtl8127_private *tp, int addr, int data) +{ + int i; + + RTL_W32(tp, EPHYAR, + EPHYAR_Write | + (addr & EPHYAR_Reg_Mask_v2) << EPHYAR_Reg_shift | + (data & EPHYAR_Data_Mask)); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed EPHY write */ + if (!(RTL_R32(tp, EPHYAR) & EPHYAR_Flag)) + break; + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); +} + +static void rtl8127_set_ephy_ext_addr(struct rtl8127_private *tp, int addr) +{ + _rtl8127_ephy_write(tp, EPHYAR_EXT_ADDR, addr); +} + +static int rtl8127_check_ephy_ext_addr(struct rtl8127_private *tp, int addr) +{ + int data; + + data = ((u16)addr >> 12); + + rtl8127_set_ephy_ext_addr(tp, data); + + return (addr & 0xfff); +} + +void rtl8127_ephy_write(struct rtl8127_private *tp, int addr, int data) +{ + _rtl8127_ephy_write(tp, rtl8127_check_ephy_ext_addr(tp, addr), data); +} + +static u16 _rtl8127_ephy_read(struct rtl8127_private *tp, int addr) +{ + int i; + u16 data = 0xffff; + + RTL_W32(tp, EPHYAR, + EPHYAR_Read | (addr & EPHYAR_Reg_Mask_v2) << EPHYAR_Reg_shift); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed EPHY read */ + if (RTL_R32(tp, EPHYAR) & EPHYAR_Flag) { + data = (u16) (RTL_R32(tp, EPHYAR) & EPHYAR_Data_Mask); + break; + } + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); + + return data; +} + +u16 rtl8127_ephy_read(struct rtl8127_private *tp, int addr) +{ + return _rtl8127_ephy_read(tp, rtl8127_check_ephy_ext_addr(tp, addr)); +} + +/* +static void ClearAndSetPCIePhyBit(struct rtl8127_private *tp, u8 addr, u16 clearmask, u16 setmask) +{ + u16 EphyValue; + + EphyValue = rtl8127_ephy_read(tp, addr); + EphyValue &= ~clearmask; + EphyValue |= setmask; + rtl8127_ephy_write(tp, addr, EphyValue); +} + +static void ClearPCIePhyBit(struct rtl8127_private *tp, u8 addr, u16 mask) +{ + ClearAndSetPCIePhyBit(tp, + addr, + mask, + 0); +} + +static void SetPCIePhyBit(struct rtl8127_private *tp, u8 addr, u16 mask) +{ + ClearAndSetPCIePhyBit(tp, + addr, + 0, + mask); +} +*/ + +static u32 +rtl8127_csi_other_fun_read(struct rtl8127_private *tp, + u8 multi_fun_sel_bit, + u32 addr) +{ + u32 cmd; + int i; + u32 value = 0xffffffff; + + cmd = CSIAR_Read | CSIAR_ByteEn << CSIAR_ByteEn_shift | (addr & CSIAR_Addr_Mask); + + if (tp->mcfg == CFG_METHOD_DEFAULT) + multi_fun_sel_bit = 0; + + if (multi_fun_sel_bit > 7) + goto exit; + + cmd |= multi_fun_sel_bit << 16; + + RTL_W32(tp, CSIAR, cmd); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed CSI read */ + if (RTL_R32(tp, CSIAR) & CSIAR_Flag) { + value = (u32)RTL_R32(tp, CSIDR); + break; + } + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); + +exit: + return value; +} + +static void +rtl8127_csi_other_fun_write(struct rtl8127_private *tp, + u8 multi_fun_sel_bit, + u32 addr, + u32 value) +{ + u32 cmd; + int i; + + RTL_W32(tp, CSIDR, value); + cmd = CSIAR_Write | CSIAR_ByteEn << CSIAR_ByteEn_shift | (addr & CSIAR_Addr_Mask); + if (tp->mcfg == CFG_METHOD_DEFAULT) + multi_fun_sel_bit = 0; + + if (multi_fun_sel_bit > 7) + return; + + cmd |= multi_fun_sel_bit << 16; + + RTL_W32(tp, CSIAR, cmd); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed CSI write */ + if (!(RTL_R32(tp, CSIAR) & CSIAR_Flag)) + break; + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); +} + +static u32 +rtl8127_csi_read(struct rtl8127_private *tp, + u32 addr) +{ + u8 multi_fun_sel_bit; + + multi_fun_sel_bit = 0; + + return rtl8127_csi_other_fun_read(tp, multi_fun_sel_bit, addr); +} + +static void +rtl8127_csi_write(struct rtl8127_private *tp, + u32 addr, + u32 value) +{ + u8 multi_fun_sel_bit; + + multi_fun_sel_bit = 0; + + rtl8127_csi_other_fun_write(tp, multi_fun_sel_bit, addr, value); +} + +static u8 +rtl8127_csi_fun0_read_byte(struct rtl8127_private *tp, + u32 addr) +{ + u8 RetVal = 0; + + if (tp->mcfg == CFG_METHOD_DEFAULT) { + struct pci_dev *pdev = tp->pci_dev; + + pci_read_config_byte(pdev, addr, &RetVal); + } else { + u32 TmpUlong; + u16 RegAlignAddr; + u8 ShiftByte; + + RegAlignAddr = addr & ~(0x3); + ShiftByte = addr & (0x3); + TmpUlong = rtl8127_csi_other_fun_read(tp, 0, RegAlignAddr); + TmpUlong >>= (8*ShiftByte); + RetVal = (u8)TmpUlong; + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); + + return RetVal; +} + +static void +rtl8127_csi_fun0_write_byte(struct rtl8127_private *tp, + u32 addr, + u8 value) +{ + if (tp->mcfg == CFG_METHOD_DEFAULT) { + struct pci_dev *pdev = tp->pci_dev; + + pci_write_config_byte(pdev, addr, value); + } else { + u32 TmpUlong; + u16 RegAlignAddr; + u8 ShiftByte; + + RegAlignAddr = addr & ~(0x3); + ShiftByte = addr & (0x3); + TmpUlong = rtl8127_csi_other_fun_read(tp, 0, RegAlignAddr); + TmpUlong &= ~(0xFF << (8*ShiftByte)); + TmpUlong |= (value << (8*ShiftByte)); + rtl8127_csi_other_fun_write(tp, 0, RegAlignAddr, TmpUlong); + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); +} + +u32 rtl8127_eri_read_with_oob_base_address(struct rtl8127_private *tp, int addr, int len, int type, const u32 base_address) +{ + int i, val_shift, shift = 0; + u32 value1 = 0, value2 = 0, mask; + u32 eri_cmd; + const u32 transformed_base_address = ((base_address & 0x00FFF000) << 6) | (base_address & 0x000FFF); + + if (len > 4 || len <= 0) + return -1; + + while (len > 0) { + val_shift = addr % ERIAR_Addr_Align; + addr = addr & ~0x3; + + eri_cmd = ERIAR_Read | + transformed_base_address | + type << ERIAR_Type_shift | + ERIAR_ByteEn << ERIAR_ByteEn_shift | + (addr & 0x0FFF); + if (addr & 0xF000) { + u32 tmp; + + tmp = addr & 0xF000; + tmp >>= 12; + eri_cmd |= (tmp << 20) & 0x00F00000; + } + + RTL_W32(tp, ERIAR, eri_cmd); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed ERI read */ + if (RTL_R32(tp, ERIAR) & ERIAR_Flag) + break; + } + + if (len == 1) mask = (0xFF << (val_shift * 8)) & 0xFFFFFFFF; + else if (len == 2) mask = (0xFFFF << (val_shift * 8)) & 0xFFFFFFFF; + else if (len == 3) mask = (0xFFFFFF << (val_shift * 8)) & 0xFFFFFFFF; + else mask = (0xFFFFFFFF << (val_shift * 8)) & 0xFFFFFFFF; + + value1 = RTL_R32(tp, ERIDR) & mask; + value2 |= (value1 >> val_shift * 8) << shift * 8; + + if (len <= 4 - val_shift) { + len = 0; + } else { + len -= (4 - val_shift); + shift = 4 - val_shift; + addr += 4; + } + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); + + return value2; +} + +u32 rtl8127_eri_read(struct rtl8127_private *tp, int addr, int len, int type) +{ + return rtl8127_eri_read_with_oob_base_address(tp, addr, len, type, 0); +} + +int rtl8127_eri_write_with_oob_base_address(struct rtl8127_private *tp, int addr, int len, u32 value, int type, const u32 base_address) +{ + int i, val_shift, shift = 0; + u32 value1 = 0, mask; + u32 eri_cmd; + const u32 transformed_base_address = ((base_address & 0x00FFF000) << 6) | (base_address & 0x000FFF); + + if (len > 4 || len <= 0) + return -1; + + while (len > 0) { + val_shift = addr % ERIAR_Addr_Align; + addr = addr & ~0x3; + + if (len == 1) mask = (0xFF << (val_shift * 8)) & 0xFFFFFFFF; + else if (len == 2) mask = (0xFFFF << (val_shift * 8)) & 0xFFFFFFFF; + else if (len == 3) mask = (0xFFFFFF << (val_shift * 8)) & 0xFFFFFFFF; + else mask = (0xFFFFFFFF << (val_shift * 8)) & 0xFFFFFFFF; + + value1 = rtl8127_eri_read_with_oob_base_address(tp, addr, 4, type, base_address) & ~mask; + value1 |= ((value << val_shift * 8) >> shift * 8); + + RTL_W32(tp, ERIDR, value1); + + eri_cmd = ERIAR_Write | + transformed_base_address | + type << ERIAR_Type_shift | + ERIAR_ByteEn << ERIAR_ByteEn_shift | + (addr & 0x0FFF); + if (addr & 0xF000) { + u32 tmp; + + tmp = addr & 0xF000; + tmp >>= 12; + eri_cmd |= (tmp << 20) & 0x00F00000; + } + + RTL_W32(tp, ERIAR, eri_cmd); + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + fsleep(R8127_CHANNEL_WAIT_TIME); + + /* Check if the RTL8125 has completed ERI write */ + if (!(RTL_R32(tp, ERIAR) & ERIAR_Flag)) + break; + } + + if (len <= 4 - val_shift) { + len = 0; + } else { + len -= (4 - val_shift); + shift = 4 - val_shift; + addr += 4; + } + } + + fsleep(R8127_CHANNEL_EXIT_DELAY_TIME); + + return 0; +} + +int rtl8127_eri_write(struct rtl8127_private *tp, int addr, int len, u32 value, int type) +{ + return rtl8127_eri_write_with_oob_base_address(tp, addr, len, value, type, NO_BASE_ADDRESS); +} + +static void +rtl8127_enable_rxdvgate(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W8(tp, 0xF2, RTL_R8(tp, 0xF2) | BIT_3); +} + +static void +rtl8127_disable_rxdvgate(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W8(tp, 0xF2, RTL_R8(tp, 0xF2) & ~BIT_3); +} + +static u8 +rtl8127_is_gpio_low(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u8 gpio_low = FALSE; + + switch (tp->HwSuppCheckPhyDisableModeVer) { + case 3: + if (!(rtl8127_mac_ocp_read(tp, 0xDC04) & BIT_13)) + gpio_low = TRUE; + break; + } + + if (gpio_low) + dprintk("gpio is low.\n"); + + return gpio_low; +} + +static u8 +rtl8127_is_phy_disable_mode_enabled(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u8 phy_disable_mode_enabled = FALSE; + + switch (tp->HwSuppCheckPhyDisableModeVer) { + case 3: + if (RTL_R8(tp, 0xF2) & BIT_5) + phy_disable_mode_enabled = TRUE; + break; + } + + if (phy_disable_mode_enabled) + dprintk("phy disable mode enabled.\n"); + + return phy_disable_mode_enabled; +} + +static u8 +rtl8127_is_in_phy_disable_mode(struct net_device *dev) +{ + u8 in_phy_disable_mode = FALSE; + + if (rtl8127_is_phy_disable_mode_enabled(dev) && rtl8127_is_gpio_low(dev)) + in_phy_disable_mode = TRUE; + + if (in_phy_disable_mode) + dprintk("Hardware is in phy disable mode.\n"); + + return in_phy_disable_mode; +} + +static void +rtl8127_stop_all_request(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W8(tp, ChipCmd, RTL_R8(tp, ChipCmd) | StopReq); + fsleep(200); +} + +static void +rtl8127_clear_stop_all_request(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W8(tp, ChipCmd, RTL_R8(tp, ChipCmd) & (CmdTxEnb | CmdRxEnb)); +} + +void +rtl8127_wait_txrx_fifo_empty(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + /* Txfifo_empty require StopReq been set */ + for (i = 0; i < 3000; i++) { + fsleep(50); + if ((RTL_R8(tp, MCUCmd_reg) & (Txfifo_empty | Rxfifo_empty)) == (Txfifo_empty | Rxfifo_empty)) + break; + } + + for (i = 0; i < 3000; i++) { + fsleep(50); + if ((RTL_R16(tp, IntrMitigate) & (BIT_0 | BIT_1 | BIT_8)) == (BIT_0 | BIT_1 | BIT_8)) + break; + } +} + +#ifdef ENABLE_DASH_SUPPORT + +static inline void +rtl8127_enable_dash2_interrupt(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) + RTL_CMAC_W8(tp, CMAC_IBIMR0, (ISRIMR_DASH_TYPE2_ROK | ISRIMR_DASH_TYPE2_TOK | ISRIMR_DASH_TYPE2_TDU | ISRIMR_DASH_TYPE2_RDU | ISRIMR_DASH_TYPE2_RX_DISABLE_IDLE)); +} + +static inline void +rtl8127_disable_dash2_interrupt(struct rtl8127_private *tp) +{ + if (!tp->DASH) + return; + + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) + RTL_CMAC_W8(tp, CMAC_IBIMR0, 0); +} +#endif + +void +rtl8127_enable_hw_linkchg_interrupt(struct rtl8127_private *tp) +{ + switch (tp->HwCurrIsrVer) { + case 6: + RTL_W32(tp, IMR_V2_SET_REG_8125, ISRIMR_V6_LINKCHG); + break; + case 5: + RTL_W32(tp, IMR_V2_SET_REG_8125, ISRIMR_V5_LINKCHG); + break; + case 4: + RTL_W32(tp, IMR_V2_SET_REG_8125, ISRIMR_V4_LINKCHG); + break; + case 2: + case 3: + RTL_W32(tp, IMR_V2_SET_REG_8125, ISRIMR_V2_LINKCHG); + break; + case 1: + RTL_W32(tp, tp->imr_reg[0], LinkChg | RTL_R32(tp, tp->imr_reg[0])); + break; + } + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + rtl8127_enable_dash2_interrupt(tp); +#endif +} + +static inline void +rtl8127_enable_hw_interrupt(struct rtl8127_private *tp) +{ + switch (tp->HwCurrIsrVer) { + case 2: + case 3: + case 4: + case 5: + case 6: + RTL_W32(tp, IMR_V2_SET_REG_8125, tp->intr_mask); + break; + case 1: + RTL_W32(tp, tp->imr_reg[0], tp->intr_mask); + + if (R8127_MULTI_RX_Q(tp)) { + int i; + for (i=1; inum_rx_rings; i++) + RTL_W16(tp, tp->imr_reg[i], other_q_intr_mask); + } + break; + } + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + rtl8127_enable_dash2_interrupt(tp); +#endif +} + +static inline void rtl8127_clear_hw_isr_v2(struct rtl8127_private *tp, + u32 message_id) +{ + RTL_W32(tp, ISR_V2_8125, BIT(message_id)); +} + +static inline void +rtl8127_disable_hw_interrupt(struct rtl8127_private *tp) +{ + if (tp->HwCurrIsrVer > 1) { + RTL_W32(tp, IMR_V2_CLEAR_REG_8125, 0xFFFFFFFF); + if (tp->HwCurrIsrVer > 3) + RTL_W32(tp, IMR_V4_L2_CLEAR_REG_8125, 0xFFFFFFFF); + } else { + RTL_W32(tp, tp->imr_reg[0], 0x0000); + + if (R8127_MULTI_RX_Q(tp)) { + int i; + for (i=1; inum_rx_rings; i++) + RTL_W16(tp, tp->imr_reg[i], 0); + } + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + rtl8127_disable_dash2_interrupt(tp); +#endif + } +} + +static inline void +rtl8127_switch_to_hw_interrupt(struct rtl8127_private *tp) +{ + RTL_W32(tp, TIMER_INT0_8125, 0x0000); + + rtl8127_enable_hw_interrupt(tp); +} + +static inline void +rtl8127_switch_to_timer_interrupt(struct rtl8127_private *tp) +{ + if (tp->use_timer_interrupt) { + RTL_W32(tp, TIMER_INT0_8125, timer_count); + RTL_W32(tp, TCTR0_8125, timer_count); + RTL_W32(tp, tp->imr_reg[0], tp->timer_intr_mask); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + rtl8127_enable_dash2_interrupt(tp); +#endif + } else { + rtl8127_switch_to_hw_interrupt(tp); + } +} + +static void +rtl8127_irq_mask_and_ack(struct rtl8127_private *tp) +{ + rtl8127_disable_hw_interrupt(tp); + + if (tp->HwCurrIsrVer > 1) { + RTL_W32(tp, ISR_V2_8125, 0xFFFFFFFF); + if (tp->HwCurrIsrVer > 3) + RTL_W32(tp, ISR_V4_L2_8125, 0xFFFFFFFF); + } else { +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) { + if (tp->dash_printer_enabled) { + RTL_W32(tp, tp->isr_reg[0], RTL_R32(tp, tp->isr_reg[0]) & + ~(ISRIMR_DASH_INTR_EN | ISRIMR_DASH_INTR_CMAC_RESET)); + } else { + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) { + RTL_CMAC_W8(tp, CMAC_IBISR0, RTL_CMAC_R8(tp, CMAC_IBISR0)); + } + } + } else { + RTL_W32(tp, tp->isr_reg[0], RTL_R32(tp, tp->isr_reg[0])); + } +#else + RTL_W32(tp, tp->isr_reg[0], RTL_R32(tp, tp->isr_reg[0])); +#endif + if (R8127_MULTI_RX_Q(tp)) { + int i; + for (i=1; inum_rx_rings; i++) + RTL_W16(tp, tp->isr_reg[i], RTL_R16(tp, tp->isr_reg[i])); + } + } +} + +static void +rtl8127_disable_rx_packet_filter(struct rtl8127_private *tp) +{ + + RTL_W32(tp, RxConfig, RTL_R32(tp, RxConfig) & + ~(AcceptErr | AcceptRunt |AcceptBroadcast | AcceptMulticast | + AcceptMyPhys | AcceptAllPhys)); +} + +static void +rtl8127_nic_reset(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + rtl8127_disable_rx_packet_filter(tp); + + rtl8127_enable_rxdvgate(dev); + + rtl8127_stop_all_request(dev); + + rtl8127_wait_txrx_fifo_empty(dev); + + rtl8127_clear_stop_all_request(dev); + + /* Soft reset the chip. */ + RTL_W8(tp, ChipCmd, CmdReset); + + /* Check that the chip has finished the reset. */ + for (i = 100; i > 0; i--) { + fsleep(100); + if ((RTL_R8(tp, ChipCmd) & CmdReset) == 0) + break; + } + + /* reset rcr */ + RTL_W32(tp, RxConfig, (RX_DMA_BURST_512 << RxCfgDMAShift)); +} + +static void +rtl8127_hw_set_interrupt_type(struct rtl8127_private *tp, u8 isr_ver) +{ + u8 tmp; + + if (tp->HwSuppIsrVer < 2) + return; + + tmp = RTL_R8(tp, INT_CFG0_8125); + + switch (tp->HwSuppIsrVer) { + case 6: + tmp &= ~INT_CFG0_AVOID_MISS_INTR; + fallthrough; + case 4: + case 5: + if (tp->HwSuppIsrVer == 6) + tmp &= ~INT_CFG0_AUTO_CLEAR_IMR; + else + tmp &= ~INT_CFG0_MSIX_ENTRY_NUM_MODE; + fallthrough; + case 2: + case 3: + tmp &= ~(INT_CFG0_ENABLE_8125); + if (isr_ver > 1) + tmp |= INT_CFG0_ENABLE_8125; + break; + default: + return; + } + + RTL_W8(tp, INT_CFG0_8125, tmp); +} + +static void +rtl8127_hw_clear_timer_int(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W32(tp, TIMER_INT0_8125, 0x0000); + RTL_W32(tp, TIMER_INT1_8125, 0x0000); + RTL_W32(tp, TIMER_INT2_8125, 0x0000); + RTL_W32(tp, TIMER_INT3_8125, 0x0000); +} + +static void +rtl8127_hw_clear_int_miti(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + switch (tp->HwSuppIntMitiVer) { + case 3: + case 6: + //IntMITI_0-IntMITI_31 + for (i=0xA00; i<0xB00; i+=4) + RTL_W32(tp, i, 0x0000); + break; + case 4: + case 5: + //IntMITI_0-IntMITI_15 + for (i = 0xA00; i < 0xA80; i += 4) + RTL_W32(tp, i, 0x0000); + + if (tp->HwSuppIntMitiVer == 5) + RTL_W8(tp, INT_CFG0_8125, RTL_R8(tp, INT_CFG0_8125) & + ~(INT_CFG0_TIMEOUT0_BYPASS_8125 | + INT_CFG0_MITIGATION_BYPASS_8125 | + INT_CFG0_RDU_BYPASS_8126)); + else + RTL_W8(tp, INT_CFG0_8125, RTL_R8(tp, INT_CFG0_8125) & + ~(INT_CFG0_TIMEOUT0_BYPASS_8125 | INT_CFG0_MITIGATION_BYPASS_8125)); + + RTL_W16(tp, INT_CFG1_8125, 0x0000); + break; + } +} + +static bool +rtl8127_vec_2_tx_q_num( + struct rtl8127_private *tp, + u32 messageId, + u32 *qnum +) +{ + u32 whichQ = 0xffffffff; + bool rc = false; + + switch (tp->HwSuppIsrVer) { + case 2: + if (messageId == 0x10) + whichQ = 0; + else if (messageId == 0x12 && tp->num_tx_rings > 1) + whichQ = 1; + break; + case 3: + case 4: + if (messageId == 0x00) + whichQ = 0; + else if (messageId == 0x01 && tp->num_tx_rings > 1) + whichQ = 1; + break; + case 5: + if (messageId == 0x10) + whichQ = 0; + else if (messageId == 0x11 && tp->num_tx_rings > 1) + whichQ = 1; + break; + case 6: + if (messageId == 0x08) + whichQ = 0; + else if (messageId == 0x09 && tp->num_tx_rings > 1) + whichQ = 1; + break; + case 7: + if (messageId == 0x1B) + whichQ = 0; + else if (messageId == 0x1C && tp->num_tx_rings > 1) + whichQ = 1; + break; + } + + if (whichQ != 0xffffffff) { + *qnum = whichQ; + rc = true; + } + + return rc; +} + +static bool +rtl8127_vec_2_rx_q_num( + struct rtl8127_private *tp, + u32 messageId, + u32 *qnum +) +{ + u32 whichQ = 0xffffffff; + bool rc = false; + + switch (tp->HwSuppIsrVer) { + case 2: + case 3: + case 4: + case 5: + case 6: + case 7: + if (messageId < tp->HwSuppNumRxQueues) + whichQ = messageId; + break; + } + + if (whichQ != 0xffffffff) { + *qnum = whichQ; + rc = true; + } + + return rc; +} + +void +rtl8127_hw_set_timer_int(struct rtl8127_private *tp, + u32 message_id, + u8 timer_intmiti_val) +{ + u32 qnum; + + switch (tp->HwSuppIntMitiVer) { + case 4: + case 5: + case 6: + //ROK + if (rtl8127_vec_2_rx_q_num(tp, message_id, &qnum)) + RTL_W8(tp,INT_MITI_V2_0_RX + 8 * qnum, timer_intmiti_val); + //TOK + if (rtl8127_vec_2_tx_q_num(tp, message_id, &qnum)) + RTL_W8(tp,INT_MITI_V2_0_TX + 8 * qnum, timer_intmiti_val); + break; + } +} + +void +rtl8127_hw_reset(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_lib_reset_prepare(tp); + + /* Disable interrupts */ + rtl8127_irq_mask_and_ack(tp); + + rtl8127_hw_clear_timer_int(dev); + + rtl8127_nic_reset(dev); +} + +static unsigned int +rtl8127_xmii_reset_pending(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned int retval; + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + rtl8127_mdio_write(tp, 0x1f, 0x0000); + retval = rtl8127_mdio_read(tp, MII_BMCR) & BMCR_RESET; + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return retval; +} + +static unsigned int +rtl8127_xmii_link_ok(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u16 status; + + status = RTL_R16(tp, PHYstatus); + if (status == 0xffff) + return 0; + + return (status & LinkStatus) ? 1 : 0; +} + +static int +rtl8127_wait_phy_reset_complete(struct rtl8127_private *tp) +{ + int i, val; + + for (i = 0; i < 2500; i++) { + val = rtl8127_mdio_read(tp, MII_BMCR) & BMCR_RESET; + if (!val) + return 0; + + mdelay(1); + } + + return -1; +} + +static void +rtl8127_xmii_reset_enable(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + int ret; + + if (rtl8127_is_in_phy_disable_mode(dev)) + return; + + spin_lock_irqsave(&tp->phy_lock, flags); + + rtl8127_mdio_write(tp, 0x1f, 0x0000); + rtl8127_mdio_write(tp, MII_ADVERTISE, rtl8127_mdio_read(tp, MII_ADVERTISE) & + ~(ADVERTISE_10HALF | ADVERTISE_10FULL | + ADVERTISE_100HALF | ADVERTISE_100FULL)); + rtl8127_mdio_write(tp, MII_CTRL1000, rtl8127_mdio_read(tp, MII_CTRL1000) & + ~(ADVERTISE_1000HALF | ADVERTISE_1000FULL)); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA5D4, rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D4) & + ~(RTK_ADVERTISE_2500FULL | RTK_ADVERTISE_5000FULL | + RTK_ADVERTISE_10000FULL)); + rtl8127_mdio_write(tp, MII_BMCR, BMCR_RESET | BMCR_ANENABLE); + + ret = rtl8127_wait_phy_reset_complete(tp); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + if (ret != 0 && netif_msg_link(tp)) + printk(KERN_ERR "%s: PHY reset failed.\n", dev->name); +} + +void +rtl8127_init_ring_indexes(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->HwSuppNumTxQueues; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + ring->dirty_tx = ring->cur_tx = 0; + ring->NextHwDesCloPtr = 0; + ring->BeginHwDesCloPtr = 0; + ring->index = i; + ring->priv = tp; + ring->netdev = tp->dev; + + /* reset BQL for queue */ + netdev_tx_reset_queue(txring_txq(ring)); + } + + for (i = 0; i < tp->HwSuppNumRxQueues; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + ring->dirty_rx = ring->cur_rx = 0; + ring->index = i; + ring->priv = tp; + ring->netdev = tp->dev; + } + +#ifdef ENABLE_LIB_SUPPORT + for (i = 0; i < tp->HwSuppNumTxQueues; i++) { + struct rtl8127_ring *ring = &tp->lib_tx_ring[i]; + ring->direction = RTL8127_CH_DIR_TX; + ring->queue_num = i; + ring->private = tp; + } + + for (i = 0; i < tp->HwSuppNumRxQueues; i++) { + struct rtl8127_ring *ring = &tp->lib_rx_ring[i]; + ring->direction = RTL8127_CH_DIR_RX; + ring->queue_num = i; + ring->private = tp; + } +#endif +} + +static void +rtl8127_issue_offset_99_event(struct rtl8127_private *tp) +{ + rtl8127_mac_ocp_write(tp, 0xE09A, rtl8127_mac_ocp_read(tp, 0xE09A) | BIT_0); +} + +#ifdef ENABLE_DASH_SUPPORT +static void +NICChkTypeEnableDashInterrupt(struct rtl8127_private *tp) +{ + if (tp->DASH) { + // + // even disconnected, enable 3 dash interrupt mask bits for in-band/out-band communication + // + if (HW_DASH_SUPPORT_TYPE_2(tp) || HW_DASH_SUPPORT_TYPE_3(tp)) { + rtl8127_enable_dash2_interrupt(tp); + RTL_W16(tp, IntrMask, (ISRIMR_DASH_INTR_EN | ISRIMR_DASH_INTR_CMAC_RESET)); + } + } +} +#endif + +static int rtl8127_enable_eee_plus(struct rtl8127_private *tp) +{ + rtl8127_mac_ocp_write(tp, 0xE080, rtl8127_mac_ocp_read(tp, 0xE080)|BIT_1); + + return 0; +} + +static int rtl8127_disable_eee_plus(struct rtl8127_private *tp) +{ + rtl8127_mac_ocp_write(tp, 0xE080, rtl8127_mac_ocp_read(tp, 0xE080)&~BIT_1); + + return 0; +} + +static void rtl8127_enable_double_vlan(struct rtl8127_private *tp) +{ + RTL_W16(tp, DOUBLE_VLAN_CONFIG, 0xf002); +} + +static void rtl8127_disable_double_vlan(struct rtl8127_private *tp) +{ + RTL_W16(tp, DOUBLE_VLAN_CONFIG, 0); +} + +static void +rtl8127_link_on_patch(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + + rtl8127_hw_config(dev); + + if (RTL_R8(tp, PHYstatus) & _10bps) + rtl8127_enable_eee_plus(tp); + + rtl8127_hw_start(dev); + + netif_carrier_on(dev); + + netif_tx_wake_all_queues(dev); + + spin_lock_irqsave(&tp->phy_lock, flags); + tp->phy_reg_aner = rtl8127_mdio_read(tp, MII_EXPANSION); + tp->phy_reg_anlpar = rtl8127_mdio_read(tp, MII_LPA); + tp->phy_reg_gbsr = rtl8127_mdio_read(tp, MII_STAT1000); + tp->phy_reg_status_2500 = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D6); + spin_unlock_irqrestore(&tp->phy_lock, flags); + +#ifdef ENABLE_PTP_SUPPORT + if (tp->EnablePtp) + rtl8127_set_local_time(tp); +#endif // ENABLE_PTP_SUPPORT +} + +static void +rtl8127_link_down_patch(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + tp->phy_reg_aner = 0; + tp->phy_reg_anlpar = 0; + tp->phy_reg_gbsr = 0; + tp->phy_reg_status_2500 = 0; + + rtl8127_disable_eee_plus(tp); + + netif_carrier_off(dev); + + netif_tx_disable(dev); + + rtl8127_hw_reset(dev); + + rtl8127_tx_clear(tp); + + rtl8127_rx_clear(tp); + + rtl8127_init_ring(dev); + + rtl8127_enable_hw_linkchg_interrupt(tp); + + //rtl8127_set_speed(dev, tp->autoneg, tp->speed, tp->duplex, tp->advertising); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + NICChkTypeEnableDashInterrupt(tp); +#endif +} + +static void +_rtl8127_check_link_status(struct net_device *dev, unsigned int link_state) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (link_state != R8127_LINK_STATE_OFF && + link_state != R8127_LINK_STATE_ON) + link_state = tp->link_ok(dev); + + if (link_state == R8127_LINK_STATE_ON) { + rtl8127_link_on_patch(dev); + + if (netif_msg_ifup(tp)) + printk(KERN_INFO PFX "%s: link up\n", dev->name); + } else { + if (netif_msg_ifdown(tp)) + printk(KERN_INFO PFX "%s: link down\n", dev->name); + + rtl8127_link_down_patch(dev); + } +} + +static void +rtl8127_check_link_status(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned int link_status_on; + + tp->resume_not_chg_speed = 0; + + link_status_on = tp->link_ok(dev); + if (netif_carrier_ok(dev) == link_status_on) + return; + + _rtl8127_check_link_status(dev, link_status_on); +} + +static bool +rtl8127_is_autoneg_mode_valid(u32 autoneg) +{ + switch(autoneg) { + case AUTONEG_ENABLE: + case AUTONEG_DISABLE: + return true; + default: + return false; + } +} + +static bool +rtl8127_is_speed_mode_valid(u32 speed) +{ + switch(speed) { + case SPEED_10000: + case SPEED_5000: + case SPEED_2500: + case SPEED_1000: + case SPEED_100: + case SPEED_10: + return true; + default: + return false; + } +} + +static bool +rtl8127_is_duplex_mode_valid(u8 duplex) +{ + switch(duplex) { + case DUPLEX_FULL: + case DUPLEX_HALF: + return true; + default: + return false; + } +} + +static void +rtl8127_set_link_option(struct rtl8127_private *tp, + u8 autoneg, + u32 speed, + u8 duplex, + enum rtl8127_fc_mode fc) +{ + u64 adv; + + if (!rtl8127_is_speed_mode_valid(speed)) + speed = SPEED_10000; + + if (!rtl8127_is_duplex_mode_valid(duplex)) + duplex = DUPLEX_FULL; + + if (!rtl8127_is_autoneg_mode_valid(autoneg)) + autoneg = AUTONEG_ENABLE; + + speed = min(speed, tp->HwSuppMaxPhyLinkSpeed); + + adv = 0; + switch(speed) { + case SPEED_10000: + adv |= ADVERTISED_10000baseT_Full; + fallthrough; + case SPEED_5000: + adv |= RTK_ADVERTISED_5000baseX_Full; + fallthrough; + case SPEED_2500: + adv |= ADVERTISED_2500baseX_Full; + fallthrough; + default: + adv |= (ADVERTISED_10baseT_Half | ADVERTISED_10baseT_Full | + ADVERTISED_100baseT_Half | ADVERTISED_100baseT_Full | + ADVERTISED_1000baseT_Half | ADVERTISED_1000baseT_Full); + break; + } + + tp->autoneg = autoneg; + tp->speed = speed; + tp->duplex = duplex; + tp->advertising = adv; + tp->fcpause = fc; +} + +static void +rtl8127_wait_ll_share_fifo_ready(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + for (i = 0; i < 10; i++) { + fsleep(100); + if (RTL_R16(tp, 0xD2) & BIT_9) + break; + } +} + +static void +rtl8127_disable_pci_offset_99(struct rtl8127_private *tp) +{ + rtl8127_mac_ocp_write(tp, 0xE032, rtl8127_mac_ocp_read(tp, 0xE032) & ~(BIT_0 | BIT_1)); + + rtl8127_csi_fun0_write_byte(tp, 0x99, 0x00); +} + +static void +rtl8127_enable_pci_offset_99(struct rtl8127_private *tp) +{ + u32 csi_tmp; + + rtl8127_csi_fun0_write_byte(tp, 0x99, tp->org_pci_offset_99); + + csi_tmp = rtl8127_mac_ocp_read(tp, 0xE032); + csi_tmp &= ~(BIT_0 | BIT_1); + if (tp->org_pci_offset_99 & (BIT_5 | BIT_6)) + csi_tmp |= BIT_1; + if (tp->org_pci_offset_99 & BIT_2) + csi_tmp |= BIT_0; + rtl8127_mac_ocp_write(tp, 0xE032, csi_tmp); +} + +static void +rtl8127_init_pci_offset_99(struct rtl8127_private *tp) +{ + rtl8127_mac_ocp_write(tp, 0xCDD0, 0x9003); + rtl8127_set_mac_ocp_bit(tp, 0xE034, (BIT_15 | BIT_14)); + rtl8127_mac_ocp_write(tp, 0xCDD2, 0x8C17); + rtl8127_mac_ocp_write(tp, 0xCDD8, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDD4, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDDA, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDD6, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDDC, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDE8, 0x8C08); + rtl8127_mac_ocp_write(tp, 0xCDEA, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDEC, 0x8C12); + rtl8127_mac_ocp_write(tp, 0xCDEE, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDF0, 0x8C2E); + rtl8127_mac_ocp_write(tp, 0xCDF2, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDF4, 0x8892); + rtl8127_mac_ocp_write(tp, 0xCDF6, 0x9003); + rtl8127_mac_ocp_write(tp, 0xCDF4, 0x8849); + rtl8127_mac_ocp_write(tp, 0xCDF6, 0x9003); + rtl8127_set_mac_ocp_bit(tp, 0xE032, BIT_14); + rtl8127_set_mac_ocp_bit(tp, 0xE0A2, BIT_0); + + rtl8127_enable_pci_offset_99(tp); +} + +static void +rtl8127_disable_pci_offset_180(struct rtl8127_private *tp) +{ + rtl8127_clear_mac_ocp_bit(tp, 0xE092, 0x00FF); +} + +static void +rtl8127_enable_pci_offset_180(struct rtl8127_private *tp) +{ + rtl8127_clear_mac_ocp_bit(tp, 0xE094, 0xFF00); + + rtl8127_clear_set_mac_ocp_bit(tp, 0xE092, 0x00FF, BIT_2); +} + +static void +rtl8127_init_pci_offset_180(struct rtl8127_private *tp) +{ + rtl8127_enable_pci_offset_180(tp); +} + +static void +rtl8127_set_pci_99_exit_driver_para(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->org_pci_offset_99 & BIT_2) + rtl8127_issue_offset_99_event(tp); + rtl8127_disable_pci_offset_99(tp); +} + +static void +rtl8127_enable_cfg9346_write(struct rtl8127_private *tp) +{ + RTL_W8(tp, Cfg9346, RTL_R8(tp, Cfg9346) | Cfg9346_Unlock); +} + +static void +rtl8127_disable_cfg9346_write(struct rtl8127_private *tp) +{ + RTL_W8(tp, Cfg9346, RTL_R8(tp, Cfg9346) & ~Cfg9346_Unlock); +} + +static void +rtl8127_enable_exit_l1_mask(struct rtl8127_private *tp) +{ + //(1)ERI(0xD4)(OCP 0xC0AC).bit[7:12]=6'b111111, L1 Mask + rtl8127_set_mac_ocp_bit(tp, 0xC0AC, (BIT_7 | BIT_8 | BIT_9 | BIT_10 | BIT_11 | BIT_12)); +} + +static void +rtl8127_disable_exit_l1_mask(struct rtl8127_private *tp) +{ + //(1)ERI(0xD4)(OCP 0xC0AC).bit[7:12]=6'b000000, L1 Mask + rtl8127_clear_mac_ocp_bit(tp, 0xC0AC, (BIT_7 | BIT_8 | BIT_9 | BIT_10 | BIT_11 | BIT_12)); +} + +static void +rtl8127_enable_extend_tally_couter(struct rtl8127_private *tp) +{ + switch (tp->HwSuppExtendTallyCounterVer) { + case 1: + rtl8127_set_mac_ocp_bit(tp, 0xEA84, (BIT_1 | BIT_0)); + break; + } +} + +static void +rtl8127_disable_extend_tally_couter(struct rtl8127_private *tp) +{ + switch (tp->HwSuppExtendTallyCounterVer) { + case 1: + rtl8127_clear_mac_ocp_bit(tp, 0xEA84, (BIT_1 | BIT_0)); + break; + } +} + +static void +rtl8127_enable_force_clkreq(struct rtl8127_private *tp, bool enable) +{ + if (enable) + RTL_W8(tp, 0xF1, RTL_R8(tp, 0xF1) | BIT_7); + else + RTL_W8(tp, 0xF1, RTL_R8(tp, 0xF1) & ~BIT_7); +} + +static void +rtl8127_enable_aspm_clkreq_lock(struct rtl8127_private *tp, bool enable) +{ + bool unlock_cfg_wr; + + if ((RTL_R8(tp, Cfg9346) & Cfg9346_EEM_MASK) == Cfg9346_Unlock) + unlock_cfg_wr = false; + else + unlock_cfg_wr = true; + + if (unlock_cfg_wr) + rtl8127_enable_cfg9346_write(tp); + + if (enable) { + RTL_W8(tp, INT_CFG0_8125, RTL_R8(tp, INT_CFG0_8125) | BIT_3); + RTL_W8(tp, Config5, RTL_R8(tp, Config5) | BIT_0); + } else { + RTL_W8(tp, INT_CFG0_8125, RTL_R8(tp, INT_CFG0_8125) & ~BIT_3); + RTL_W8(tp, Config5, RTL_R8(tp, Config5) & ~BIT_0); + } + + if (unlock_cfg_wr) + rtl8127_disable_cfg9346_write(tp); +} + +static void +rtl8127_hw_d3_para(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W16(tp, RxMaxSize, RX_BUF_SIZE); + + rtl8127_enable_force_clkreq(tp, 0); + rtl8127_enable_aspm_clkreq_lock(tp, 0); + + rtl8127_disable_exit_l1_mask(tp); + +#ifdef ENABLE_REALWOW_SUPPORT + rtl8127_set_realwow_d3_para(dev); +#endif + + rtl8127_set_pci_99_exit_driver_para(dev); + + rtl8127_disable_rxdvgate(dev); + + rtl8127_disable_extend_tally_couter(tp); +} + +static void +rtl8127_enable_magic_packet(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppMagicPktVer) { + case WAKEUP_MAGIC_PACKET_V3: + rtl8127_mac_ocp_write(tp, 0xC0B6, rtl8127_mac_ocp_read(tp, 0xC0B6) | BIT_0); + break; + } +} +static void +rtl8127_disable_magic_packet(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppMagicPktVer) { + case WAKEUP_MAGIC_PACKET_V3: + rtl8127_mac_ocp_write(tp, 0xC0B6, rtl8127_mac_ocp_read(tp, 0xC0B6) & ~BIT_0); + break; + } +} + +static void +rtl8127_enable_linkchg_wakeup(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppLinkChgWakeUpVer) { + case 3: + RTL_W8(tp, Config3, RTL_R8(tp, Config3) | LinkUp); + rtl8127_clear_set_mac_ocp_bit(tp, 0xE0C6, (BIT_5 | BIT_3 | BIT_2), (BIT_4 | BIT_1 | BIT_0)); + break; + } +} + +static void +rtl8127_disable_linkchg_wakeup(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppLinkChgWakeUpVer) { + case 3: + RTL_W8(tp, Config3, RTL_R8(tp, Config3) & ~LinkUp); + if (!(rtl8127_mac_ocp_read(tp, 0xE0C6) & BIT_0)) + rtl8127_clear_set_mac_ocp_bit(tp, 0xE0C6, (BIT_5 | BIT_3 | BIT_2 | BIT_1), BIT_4); + break; + } +} + +#define WAKE_ANY (WAKE_PHY | WAKE_MAGIC | WAKE_UCAST | WAKE_BCAST | WAKE_MCAST) + +static u32 +rtl8127_get_hw_wol(struct rtl8127_private *tp) +{ + u8 options; + u32 csi_tmp; + u32 wol_opts = 0; + + if (disable_wol_support) + goto out; + + options = RTL_R8(tp, Config1); + if (!(options & PMEnable)) + goto out; + + options = RTL_R8(tp, Config3); + if (options & LinkUp) + wol_opts |= WAKE_PHY; + + switch (tp->HwSuppMagicPktVer) { + case WAKEUP_MAGIC_PACKET_V3: + csi_tmp = rtl8127_mac_ocp_read(tp, 0xC0B6); + if (csi_tmp & BIT_0) + wol_opts |= WAKE_MAGIC; + break; + } + + options = RTL_R8(tp, Config5); + if (options & UWF) + wol_opts |= WAKE_UCAST; + if (options & BWF) + wol_opts |= WAKE_BCAST; + if (options & MWF) + wol_opts |= WAKE_MCAST; + +out: + return wol_opts; +} + +static void +rtl8127_enable_d0_speedup(struct rtl8127_private *tp) +{ + u16 clearmask; + u16 setmask; + + if (FALSE == HW_SUPPORT_D0_SPEED_UP(tp)) + return; + + if (tp->D0SpeedUpSpeed == D0_SPEED_UP_SPEED_DISABLE) + return; + + if (tp->HwSuppD0SpeedUpVer == 1 || tp->HwSuppD0SpeedUpVer == 2) { + //speed up speed + clearmask = (BIT_10 | BIT_9 | BIT_8 | BIT_7); + if (tp->D0SpeedUpSpeed == D0_SPEED_UP_SPEED_2500) + setmask = BIT_7; + else if (tp->D0SpeedUpSpeed == D0_SPEED_UP_SPEED_5000) + setmask = BIT_8; + else if (tp->D0SpeedUpSpeed == D0_SPEED_UP_SPEED_10000) + setmask = BIT_7 | BIT_8; + else + setmask = 0; + rtl8127_clear_set_mac_ocp_bit(tp, 0xE10A, clearmask, setmask); + + //speed up flowcontrol + clearmask = (BIT_15 | BIT_14); + if (tp->HwSuppD0SpeedUpVer == 2) + clearmask |= BIT_13; + + if (tp->fcpause == rtl8127_fc_full) { + setmask = (BIT_15 | BIT_14); + if (tp->HwSuppD0SpeedUpVer == 2) + setmask |= BIT_13; + } else + setmask = 0; + rtl8127_clear_set_mac_ocp_bit(tp, 0xE860, clearmask, setmask); + } + + RTL_W8(tp, 0xD0, RTL_R8(tp, 0xD0) | BIT_3); +} + +static void +rtl8127_disable_d0_speedup(struct rtl8127_private *tp) +{ + if (FALSE == HW_SUPPORT_D0_SPEED_UP(tp)) + return; + + RTL_W8(tp, 0xD0, RTL_R8(tp, 0xD0) & ~BIT_3); +} + +static void +rtl8127_set_hw_wol(struct net_device *dev, u32 wolopts) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i,tmp = 0; + static struct { + u32 opt; + u16 reg; + u8 mask; + } cfg[] = { + { WAKE_PHY, Config3, LinkUp }, + { WAKE_UCAST, Config5, UWF }, + { WAKE_BCAST, Config5, BWF }, + { WAKE_MCAST, Config5, MWF }, + { WAKE_ANY, Config5, LanWake }, + { WAKE_MAGIC, Config3, MagicPacket }, + }; + + switch (tp->HwSuppMagicPktVer) { + case WAKEUP_MAGIC_PACKET_V3: + tmp = ARRAY_SIZE(cfg) - 1; + + if (wolopts & WAKE_MAGIC) + rtl8127_enable_magic_packet(dev); + else + rtl8127_disable_magic_packet(dev); + break; + default: + break; + } + + rtl8127_enable_cfg9346_write(tp); + + for (i = 0; i < tmp; i++) { + u8 options = RTL_R8(tp, cfg[i].reg) & ~cfg[i].mask; + if (wolopts & cfg[i].opt) + options |= cfg[i].mask; + RTL_W8(tp, cfg[i].reg, options); + } + + switch (tp->HwSuppLinkChgWakeUpVer) { + case 3: + if (wolopts & WAKE_PHY) + rtl8127_enable_linkchg_wakeup(dev); + else + rtl8127_disable_linkchg_wakeup(dev); + break; + } + + rtl8127_disable_cfg9346_write(tp); +} + +static void +rtl8127_phy_restart_nway(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (rtl8127_is_in_phy_disable_mode(dev)) + return; + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + rtl8127_mdio_write(tp, MII_BMCR, BMCR_ANENABLE | BMCR_ANRESTART); +} + +static void +rtl8127_phy_setup_force_mode(struct net_device *dev, u32 speed, u8 duplex) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u16 bmcr_true_force = 0; + + if (rtl8127_is_in_phy_disable_mode(dev)) + return; + + if ((speed == SPEED_10) && (duplex == DUPLEX_HALF)) { + bmcr_true_force = BMCR_SPEED10; + } else if ((speed == SPEED_10) && (duplex == DUPLEX_FULL)) { + bmcr_true_force = BMCR_SPEED10 | BMCR_FULLDPLX; + } else if ((speed == SPEED_100) && (duplex == DUPLEX_HALF)) { + bmcr_true_force = BMCR_SPEED100; + } else if ((speed == SPEED_100) && (duplex == DUPLEX_FULL)) { + bmcr_true_force = BMCR_SPEED100 | BMCR_FULLDPLX; + } else { + netif_err(tp, drv, dev, "Failed to set phy force mode!\n"); + return; + } + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + rtl8127_mdio_write(tp, MII_BMCR, bmcr_true_force); +} + +static void +rtl8127_set_pci_pme(struct rtl8127_private *tp, int set) +{ + struct pci_dev *pdev = tp->pci_dev; + u16 pmc; + + if (!pdev->pm_cap) + return; + + pci_read_config_word(pdev, pdev->pm_cap + PCI_PM_CTRL, &pmc); + pmc |= PCI_PM_CTRL_PME_STATUS; + if (set) + pmc |= PCI_PM_CTRL_PME_ENABLE; + else + pmc &= ~PCI_PM_CTRL_PME_ENABLE; + pci_write_config_word(pdev, pdev->pm_cap + PCI_PM_CTRL, pmc); +} + +static void +rtl8127_enable_giga_lite(struct rtl8127_private *tp, u64 adv) +{ + if (adv & ADVERTISED_1000baseT_Full) + rtl8127_set_eth_phy_ocp_bit(tp, 0xA428, BIT_9); + else + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA428, BIT_9); + + if (adv & ADVERTISED_2500baseX_Full) + rtl8127_set_eth_phy_ocp_bit(tp, 0xA5EA, BIT_0); + else + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5EA, BIT_0); + + if (adv & RTK_ADVERTISED_5000baseX_Full) + rtl8127_set_eth_phy_ocp_bit(tp, 0xA5EA, BIT_1); + else + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5EA, BIT_1); + + if (adv & ADVERTISED_10000baseT_Full) + rtl8127_set_eth_phy_ocp_bit(tp, 0xA5EA, BIT_2); + else + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5EA, BIT_2); +} + +static void +rtl8127_disable_giga_lite(struct rtl8127_private *tp) +{ + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA428, BIT_9); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5EA, BIT_0 | BIT_1 | BIT_2); +} + +static void +rtl8127_set_wol_link_speed(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + int auto_nego; + int giga_ctrl; + int ctrl_2500; + u64 adv; + u16 anlpar; + u16 gbsr; + u16 status_2500; + u16 aner; + + spin_lock_irqsave(&tp->phy_lock, flags); + + if (tp->autoneg != AUTONEG_ENABLE) + goto exit; + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + + auto_nego = rtl8127_mdio_read(tp, MII_ADVERTISE); + auto_nego &= ~(ADVERTISE_10HALF | ADVERTISE_10FULL + | ADVERTISE_100HALF | ADVERTISE_100FULL); + + giga_ctrl = rtl8127_mdio_read(tp, MII_CTRL1000); + giga_ctrl &= ~(ADVERTISE_1000HALF | ADVERTISE_1000FULL); + + ctrl_2500 = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D4); + ctrl_2500 &= ~(RTK_ADVERTISE_2500FULL | RTK_ADVERTISE_5000FULL | + RTK_ADVERTISE_10000FULL); + + aner = tp->phy_reg_aner; + anlpar = tp->phy_reg_anlpar; + gbsr = tp->phy_reg_gbsr; + status_2500 = tp->phy_reg_status_2500; + if (tp->link_ok(dev)) { + aner = rtl8127_mdio_read(tp, MII_EXPANSION); + anlpar = rtl8127_mdio_read(tp, MII_LPA); + gbsr = rtl8127_mdio_read(tp, MII_STAT1000); + status_2500 = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D6); + } + + adv = tp->advertising; + if ((aner | anlpar | gbsr | status_2500) == 0) { + int auto_nego_tmp = 0; + if (adv & ADVERTISED_10baseT_Half) + auto_nego_tmp |= ADVERTISE_10HALF; + if (adv & ADVERTISED_10baseT_Full) + auto_nego_tmp |= ADVERTISE_10FULL; + if (adv & ADVERTISED_100baseT_Half) + auto_nego_tmp |= ADVERTISE_100HALF; + if (adv & ADVERTISED_100baseT_Full) + auto_nego_tmp |= ADVERTISE_100FULL; + + if (auto_nego_tmp == 0) + goto exit; + + auto_nego |= auto_nego_tmp; + goto skip_check_lpa; + } + if (!(aner & EXPANSION_NWAY)) + goto exit; + + if ((adv & ADVERTISED_10baseT_Half) && (anlpar & LPA_10HALF)) + auto_nego |= ADVERTISE_10HALF; + else if ((adv & ADVERTISED_10baseT_Full) && (anlpar & LPA_10FULL)) + auto_nego |= ADVERTISE_10FULL; + else if ((adv & ADVERTISED_100baseT_Half) && (anlpar & LPA_100HALF)) + auto_nego |= ADVERTISE_100HALF; + else if ((adv & ADVERTISED_100baseT_Full) && (anlpar & LPA_100FULL)) + auto_nego |= ADVERTISE_100FULL; + else if (adv & ADVERTISED_1000baseT_Half && (gbsr & LPA_1000HALF)) + giga_ctrl |= ADVERTISE_1000HALF; + else if (adv & ADVERTISED_1000baseT_Full && (gbsr & LPA_1000FULL)) + giga_ctrl |= ADVERTISE_1000FULL; + else if (adv & ADVERTISED_2500baseX_Full && (status_2500 & RTK_LPA_ADVERTISE_2500FULL)) + ctrl_2500 |= RTK_ADVERTISE_2500FULL; + else if (adv & RTK_ADVERTISED_5000baseX_Full && (status_2500 & RTK_LPA_ADVERTISE_5000FULL)) + ctrl_2500 |= RTK_ADVERTISE_5000FULL; + else if (adv & ADVERTISED_10000baseT_Full && (status_2500 & RTK_LPA_ADVERTISE_10000FULL)) + ctrl_2500 |= RTK_ADVERTISE_10000FULL; + else + goto exit; + +skip_check_lpa: + if (tp->DASH) + auto_nego |= (ADVERTISE_100FULL | ADVERTISE_100HALF | ADVERTISE_10HALF | ADVERTISE_10FULL); + +#ifdef CONFIG_DOWN_SPEED_100 + auto_nego |= (ADVERTISE_100FULL | ADVERTISE_100HALF | ADVERTISE_10HALF | ADVERTISE_10FULL); +#endif + + rtl8127_mdio_write(tp, MII_ADVERTISE, auto_nego); + rtl8127_mdio_write(tp, MII_CTRL1000, giga_ctrl); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA5D4, ctrl_2500); + + rtl8127_disable_giga_lite(tp); + + rtl8127_phy_restart_nway(dev); + +exit: + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return; +} + +static bool +rtl8127_keep_wol_link_speed(struct net_device *dev, u8 from_suspend) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (from_suspend && tp->link_ok(dev) && (tp->wol_opts & WAKE_PHY)) + return 1; + + if (!from_suspend && tp->resume_not_chg_speed) + return 1; + + return 0; +} +static void +rtl8127_powerdown_pll(struct net_device *dev, u8 from_suspend) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + /* Reboot not set wol link speed */ + if (system_state == SYSTEM_RESTART) + return; + + tp->check_keep_link_speed = 0; + if (tp->wol_enabled == WOL_ENABLED || tp->DASH || tp->EnableKCPOffload) { + rtl8127_set_hw_wol(dev, tp->wol_opts); + + rtl8127_enable_cfg9346_write(tp); + RTL_W8(tp, Config2, RTL_R8(tp, Config2) | PMSTS_En); + rtl8127_disable_cfg9346_write(tp); + + /* Enable the PME and clear the status */ + rtl8127_set_pci_pme(tp, 1); + + if (rtl8127_keep_wol_link_speed(dev, from_suspend)) { + tp->check_keep_link_speed = 1; + } else { + if (tp->D0SpeedUpSpeed != D0_SPEED_UP_SPEED_DISABLE) { + rtl8127_enable_d0_speedup(tp); + tp->check_keep_link_speed = 1; + } + + rtl8127_set_wol_link_speed(dev); + } + + RTL_W32(tp, RxConfig, RTL_R32(tp, RxConfig) | AcceptBroadcast | AcceptMulticast | AcceptMyPhys); + + return; + } + + if (tp->DASH) + return; + + rtl8127_phy_power_down(dev); + + RTL_W8(tp, 0xF2, RTL_R8(tp, 0xF2) & ~BIT_6); +} + +static void rtl8127_powerup_pll(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + RTL_W8(tp, PMCH, RTL_R8(tp, PMCH) | BIT_7 | BIT_6); + + if (tp->resume_not_chg_speed) + return; + + rtl8127_phy_power_up(dev); +} + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static void +rtl8127_get_wol(struct net_device *dev, + struct ethtool_wolinfo *wol) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u8 options; + + wol->wolopts = 0; + + if (tp->mcfg == CFG_METHOD_DEFAULT || disable_wol_support) { + wol->supported = 0; + return; + } else { + wol->supported = WAKE_ANY; + } + + options = RTL_R8(tp, Config1); + if (!(options & PMEnable)) + return; + + wol->wolopts = tp->wol_opts; +} + +static int +rtl8127_set_wol(struct net_device *dev, + struct ethtool_wolinfo *wol) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->mcfg == CFG_METHOD_DEFAULT || disable_wol_support) + return -EOPNOTSUPP; + + tp->wol_opts = wol->wolopts; + + tp->wol_enabled = (tp->wol_opts) ? WOL_ENABLED : WOL_DISABLED; + + device_set_wakeup_enable(tp_to_dev(tp), wol->wolopts); + + return 0; +} + +static void +rtl8127_get_drvinfo(struct net_device *dev, + struct ethtool_drvinfo *info) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct rtl8127_fw *rtl_fw = tp->rtl_fw; + + strscpy(info->driver, MODULENAME, sizeof(info->driver)); + strscpy(info->version, RTL8127_VERSION, sizeof(info->version)); + strscpy(info->bus_info, pci_name(tp->pci_dev), sizeof(info->bus_info)); + info->regdump_len = R8127_REGS_DUMP_SIZE; + info->eedump_len = tp->eeprom_len; + BUILD_BUG_ON(sizeof(info->fw_version) < sizeof(rtl_fw->version)); + if (rtl_fw) + strscpy(info->fw_version, rtl_fw->version, + sizeof(info->fw_version)); +} + +static int +rtl8127_get_regs_len(struct net_device *dev) +{ + return R8127_REGS_DUMP_SIZE; +} +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + +static void +rtl8127_set_d0_speedup_speed(struct rtl8127_private *tp) +{ + if (FALSE == HW_SUPPORT_D0_SPEED_UP(tp)) + return; + + tp->D0SpeedUpSpeed = D0_SPEED_UP_SPEED_DISABLE; + if (tp->autoneg == AUTONEG_ENABLE) { + if (tp->speed == SPEED_10000) + tp->D0SpeedUpSpeed = D0_SPEED_UP_SPEED_10000; + else if (tp->speed == SPEED_5000) + tp->D0SpeedUpSpeed = D0_SPEED_UP_SPEED_5000; + else if (tp->speed == SPEED_2500) + tp->D0SpeedUpSpeed = D0_SPEED_UP_SPEED_2500; + else if (tp->speed == SPEED_1000) + tp->D0SpeedUpSpeed = D0_SPEED_UP_SPEED_1000; + } +} + +static int +rtl8127_set_speed_xmii(struct net_device *dev, + u8 autoneg, + u32 speed, + u8 duplex, + u64 adv) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + int auto_nego = 0; + int giga_ctrl = 0; + int ctrl_2500 = 0; + int rc = -EINVAL; + + spin_lock_irqsave(&tp->phy_lock, flags); + + if (!rtl8127_is_speed_mode_valid(speed)) { + speed = SPEED_10000; + duplex = DUPLEX_FULL; + adv |= tp->advertising; + } + + if (eee_giga_lite && (autoneg == AUTONEG_ENABLE)) + rtl8127_enable_giga_lite(tp, adv); + else + rtl8127_disable_giga_lite(tp); + + giga_ctrl = rtl8127_mdio_read(tp, MII_CTRL1000); + giga_ctrl &= ~(ADVERTISE_1000HALF | ADVERTISE_1000FULL); + ctrl_2500 = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D4); + ctrl_2500 &= ~(RTK_ADVERTISE_2500FULL | RTK_ADVERTISE_5000FULL | + RTK_ADVERTISE_10000FULL); + + if (autoneg == AUTONEG_ENABLE) { + /*n-way force*/ + auto_nego = rtl8127_mdio_read(tp, MII_ADVERTISE); + auto_nego &= ~(ADVERTISE_10HALF | ADVERTISE_10FULL | + ADVERTISE_100HALF | ADVERTISE_100FULL | + ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM); + + if (adv & ADVERTISED_10baseT_Half) + auto_nego |= ADVERTISE_10HALF; + if (adv & ADVERTISED_10baseT_Full) + auto_nego |= ADVERTISE_10FULL; + if (adv & ADVERTISED_100baseT_Half) + auto_nego |= ADVERTISE_100HALF; + if (adv & ADVERTISED_100baseT_Full) + auto_nego |= ADVERTISE_100FULL; + if (adv & ADVERTISED_1000baseT_Half) + giga_ctrl |= ADVERTISE_1000HALF; + if (adv & ADVERTISED_1000baseT_Full) + giga_ctrl |= ADVERTISE_1000FULL; + if (adv & ADVERTISED_2500baseX_Full) + ctrl_2500 |= RTK_ADVERTISE_2500FULL; + if (HW_SUPP_PHY_LINK_SPEED_5000M(tp)) { + if (adv & RTK_ADVERTISED_5000baseX_Full) + ctrl_2500 |= RTK_ADVERTISE_5000FULL; + } + if (HW_SUPP_PHY_LINK_SPEED_10000M(tp)) { + if (adv & ADVERTISED_10000baseT_Full) + ctrl_2500 |= RTK_ADVERTISE_10000FULL; + } + + //flow control + if (tp->fcpause == rtl8127_fc_full) + auto_nego |= ADVERTISE_PAUSE_CAP | ADVERTISE_PAUSE_ASYM; + + tp->phy_auto_nego_reg = auto_nego; + tp->phy_1000_ctrl_reg = giga_ctrl; + + tp->phy_2500_ctrl_reg = ctrl_2500; + + rtl8127_mdio_write(tp, 0x1f, 0x0000); + rtl8127_mdio_write(tp, MII_ADVERTISE, auto_nego); + rtl8127_mdio_write(tp, MII_CTRL1000, giga_ctrl); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA5D4, ctrl_2500); + rtl8127_phy_restart_nway(dev); + } else { + /*true force*/ + if (speed == SPEED_10 || speed == SPEED_100) + rtl8127_phy_setup_force_mode(dev, speed, duplex); + else + goto out; + } + + tp->autoneg = autoneg; + tp->speed = speed; + tp->duplex = duplex; + tp->advertising = adv; + + rtl8127_set_d0_speedup_speed(tp); + + rc = 0; +out: + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return rc; +} + +static int +rtl8127_set_speed(struct net_device *dev, + u8 autoneg, + u32 speed, + u8 duplex, + u64 adv) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret; + + if (tp->resume_not_chg_speed) + return 0; + + ret = tp->set_speed(dev, autoneg, speed, duplex, adv); + + return ret; +} + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static int +rtl8127_set_settings(struct net_device *dev, +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + struct ethtool_cmd *cmd +#else + const struct ethtool_link_ksettings *cmd +#endif + ) +{ + int ret; + u8 autoneg; + u32 speed; + u8 duplex; + u64 supported = 0, advertising = 0; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + autoneg = cmd->autoneg; + speed = cmd->speed; + duplex = cmd->duplex; + supported = cmd->supported; + advertising = cmd->advertising; +#else + struct rtl8127_private *tp = netdev_priv(dev); + const struct ethtool_link_settings *base = &cmd->base; + autoneg = base->autoneg; + speed = base->speed; + duplex = base->duplex; + ethtool_convert_link_mode_to_legacy_u32((u32*)&supported, + cmd->link_modes.supported); + ethtool_convert_link_mode_to_legacy_u32((u32*)&advertising, + cmd->link_modes.advertising); + if (test_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, + cmd->link_modes.supported)) + supported |= ADVERTISED_2500baseX_Full; + if (test_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, + cmd->link_modes.advertising)) + advertising |= ADVERTISED_2500baseX_Full; + if (HW_SUPP_PHY_LINK_SPEED_5000M(tp)) { + if (test_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, + cmd->link_modes.supported)) + supported |= RTK_ADVERTISED_5000baseX_Full; + if (test_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, + cmd->link_modes.advertising)) + advertising |= RTK_ADVERTISED_5000baseX_Full; + } + if (HW_SUPP_PHY_LINK_SPEED_10000M(tp)) { + if (test_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, + cmd->link_modes.supported)) + supported |= ADVERTISED_10000baseT_Full; + if (test_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, + cmd->link_modes.advertising)) + advertising |= ADVERTISED_10000baseT_Full; + } +#endif + if (advertising & ~supported) + return -EINVAL; + + ret = rtl8127_set_speed(dev, autoneg, speed, duplex, advertising); + + return ret; +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +static u32 +rtl8127_get_tx_csum(struct net_device *dev) +{ + u32 ret; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + ret = ((dev->features & NETIF_F_IP_CSUM) != 0); +#else + ret = ((dev->features & (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)) != 0); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + + return ret; +} + +static u32 +rtl8127_get_rx_csum(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u32 ret; + + ret = tp->cp_cmd & RxChkSum; + + return ret; +} + +static int +rtl8127_set_tx_csum(struct net_device *dev, + u32 data) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->mcfg == CFG_METHOD_DEFAULT) + return -EOPNOTSUPP; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + if (data) + dev->features |= NETIF_F_IP_CSUM; + else + dev->features &= ~NETIF_F_IP_CSUM; +#else + if (data) + dev->features |= (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM); + else + dev->features &= ~(NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + + return 0; +} + +static int +rtl8127_set_rx_csum(struct net_device *dev, + u32 data) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->mcfg == CFG_METHOD_DEFAULT) + return -EOPNOTSUPP; + + if (data) + tp->cp_cmd |= RxChkSum; + else + tp->cp_cmd &= ~RxChkSum; + + RTL_W16(tp, CPlusCmd, tp->cp_cmd); + + return 0; +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + +static u32 +rtl8127_rx_desc_opts1(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + return READ_ONCE(((struct RxDescV3 *)desc)->RxDescNormalDDWord4.opts1); + case RX_DESC_RING_TYPE_4: + return READ_ONCE(((struct RxDescV4 *)desc)->RxDescNormalDDWord2.opts1); + default: + return READ_ONCE(desc->opts1); + } +} + +static u32 +rtl8127_rx_desc_opts2(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + return ((struct RxDescV3 *)desc)->RxDescNormalDDWord4.opts2; + case RX_DESC_RING_TYPE_4: + return ((struct RxDescV4 *)desc)->RxDescNormalDDWord2.opts2; + default: + return desc->opts2; + } +} + +#ifdef CONFIG_R8127_VLAN + +static void +rtl8127_clear_rx_desc_opts2(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + ((struct RxDescV3 *)desc)->RxDescNormalDDWord4.opts2 = 0; + break; + case RX_DESC_RING_TYPE_4: + ((struct RxDescV4 *)desc)->RxDescNormalDDWord2.opts2 = 0; + break; + default: + desc->opts2 = 0; + break; + } +} + + +static inline u32 +rtl8127_tx_vlan_tag(struct rtl8127_private *tp, + struct sk_buff *skb) +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + return (tp->vlgrp && vlan_tx_tag_present(skb)) ? + TxVlanTag | swab16(vlan_tx_tag_get(skb)) : 0x00; +#elif LINUX_VERSION_CODE < KERNEL_VERSION(4,0,0) + return (vlan_tx_tag_present(skb)) ? + TxVlanTag | swab16(vlan_tx_tag_get(skb)) : 0x00; +#else + return (skb_vlan_tag_present(skb)) ? + TxVlanTag | swab16(skb_vlan_tag_get(skb)) : 0x00; +#endif + + return 0; +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + +static void +rtl8127_vlan_rx_register(struct net_device *dev, + struct vlan_group *grp) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + tp->vlgrp = grp; + + if (tp->vlgrp) { + tp->rtl8127_rx_config |= (EnableInnerVlan | EnableOuterVlan); + RTL_W32(tp, RxConfig, RTL_R32(tp, RxConfig) | (EnableInnerVlan | EnableOuterVlan)) + } else { + tp->rtl8127_rx_config &= ~(EnableInnerVlan | EnableOuterVlan); + RTL_W32(tp, RxConfig, RTL_R32(tp, RxConfig) & ~(EnableInnerVlan | EnableOuterVlan)) + } +} + +#endif + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) +static void +rtl8127_vlan_rx_kill_vid(struct net_device *dev, + unsigned short vid) +{ + struct rtl8127_private *tp = netdev_priv(dev); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,21) + if (tp->vlgrp) + tp->vlgrp->vlan_devices[vid] = NULL; +#else + vlan_group_set_device(tp->vlgrp, vid, NULL); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,21) +} +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) + +static int +rtl8127_rx_vlan_skb(struct rtl8127_private *tp, + struct RxDesc *desc, + struct sk_buff *skb) +{ + u32 opts2 = le32_to_cpu(rtl8127_rx_desc_opts2(tp, desc)); + int ret = -1; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + if (tp->vlgrp && (opts2 & RxVlanTag)) { + rtl8127_rx_hwaccel_skb(skb, tp->vlgrp, + swab16(opts2 & 0xffff)); + ret = 0; + } +#elif LINUX_VERSION_CODE < KERNEL_VERSION(3,10,0) + if (opts2 & RxVlanTag) + __vlan_hwaccel_put_tag(skb, swab16(opts2 & 0xffff)); +#else + if (opts2 & RxVlanTag) + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), swab16(opts2 & 0xffff)); +#endif + + rtl8127_clear_rx_desc_opts2(tp, desc); + return ret; +} + +#else /* !CONFIG_R8127_VLAN */ + +static inline u32 +rtl8127_tx_vlan_tag(struct rtl8127_private *tp, + struct sk_buff *skb) +{ + return 0; +} + +static int +rtl8127_rx_vlan_skb(struct rtl8127_private *tp, + struct RxDesc *desc, + struct sk_buff *skb) +{ + return -1; +} + +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) + +static netdev_features_t rtl8127_fix_features(struct net_device *dev, + netdev_features_t features) +{ + if (dev->mtu > MSS_MAX) + features &= ~NETIF_F_ALL_TSO; + if (dev->mtu > ETH_DATA_LEN) { + features &= ~NETIF_F_ALL_TSO; + features &= ~NETIF_F_ALL_CSUM; + } +#ifndef CONFIG_R8127_VLAN + features &= ~NETIF_F_ALL_CSUM; +#endif + + return features; +} + +static int rtl8127_hw_set_features(struct net_device *dev, + netdev_features_t features) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u32 rx_config; + + rx_config = RTL_R32(tp, RxConfig); + if (features & NETIF_F_RXALL) { + tp->rtl8127_rx_config |= (AcceptErr | AcceptRunt); + rx_config |= (AcceptErr | AcceptRunt); + } else { + tp->rtl8127_rx_config &= ~(AcceptErr | AcceptRunt); + rx_config &= ~(AcceptErr | AcceptRunt); + } + + if (features & NETIF_F_HW_VLAN_RX) { + tp->rtl8127_rx_config |= (EnableInnerVlan | EnableOuterVlan); + rx_config |= (EnableInnerVlan | EnableOuterVlan); + } else { + tp->rtl8127_rx_config &= ~(EnableInnerVlan | EnableOuterVlan); + rx_config &= ~(EnableInnerVlan | EnableOuterVlan); + } + + RTL_W32(tp, RxConfig, rx_config); + + if (features & NETIF_F_RXCSUM) + tp->cp_cmd |= RxChkSum; + else + tp->cp_cmd &= ~RxChkSum; + + RTL_W16(tp, CPlusCmd, tp->cp_cmd); + RTL_R16(tp, CPlusCmd); + + return 0; +} + +static int rtl8127_set_features(struct net_device *dev, + netdev_features_t features) +{ + features &= NETIF_F_RXALL | NETIF_F_RXCSUM | NETIF_F_HW_VLAN_RX; + + rtl8127_hw_set_features(dev, features); + + return 0; +} + +#endif + +static u8 rtl8127_get_mdi_status(struct rtl8127_private *tp) +{ + if (!tp->link_ok(tp->dev)) + return ETH_TP_MDI_INVALID; + + if (rtl8127_mdio_direct_read_phy_ocp(tp, 0xA444) & BIT_1) + return ETH_TP_MDI; + else + return ETH_TP_MDI_X; +} + +static void rtl8127_gset_xmii(struct net_device *dev, +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + struct ethtool_cmd *cmd +#else + struct ethtool_link_ksettings *cmd +#endif + ) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u16 aner = tp->phy_reg_aner; + u16 anlpar = tp->phy_reg_anlpar; + u16 gbsr = tp->phy_reg_gbsr; + u16 status_2500 = tp->phy_reg_status_2500; + unsigned long flags; + u64 lpa_adv = 0; + u32 status; + u8 autoneg, duplex; + u32 speed = 0; + u16 bmcr; + u64 supported, advertising; + u8 report_lpa = 0; + + supported = SUPPORTED_10baseT_Half | + SUPPORTED_10baseT_Full | + SUPPORTED_100baseT_Half | + SUPPORTED_100baseT_Full | + SUPPORTED_1000baseT_Full | + SUPPORTED_2500baseX_Full | + RTK_SUPPORTED_5000baseX_Full | + SUPPORTED_10000baseT_Full | + SUPPORTED_Autoneg | + SUPPORTED_TP | + SUPPORTED_Pause | + SUPPORTED_Asym_Pause; + + if (!HW_SUPP_PHY_LINK_SPEED_2500M(tp)) + supported &= ~SUPPORTED_2500baseX_Full; + + if (!HW_SUPP_PHY_LINK_SPEED_5000M(tp)) + supported &= ~RTK_SUPPORTED_5000baseX_Full; + + if (!HW_SUPP_PHY_LINK_SPEED_10000M(tp)) + supported &= ~SUPPORTED_10000baseT_Full; + + advertising = tp->advertising; + if (tp->phy_auto_nego_reg || tp->phy_1000_ctrl_reg || + tp->phy_2500_ctrl_reg) { + advertising = 0; + if (tp->phy_auto_nego_reg & ADVERTISE_10HALF) + advertising |= ADVERTISED_10baseT_Half; + if (tp->phy_auto_nego_reg & ADVERTISE_10FULL) + advertising |= ADVERTISED_10baseT_Full; + if (tp->phy_auto_nego_reg & ADVERTISE_100HALF) + advertising |= ADVERTISED_100baseT_Half; + if (tp->phy_auto_nego_reg & ADVERTISE_100FULL) + advertising |= ADVERTISED_100baseT_Full; + if (tp->phy_1000_ctrl_reg & ADVERTISE_1000FULL) + advertising |= ADVERTISED_1000baseT_Full; + if (tp->phy_2500_ctrl_reg & RTK_ADVERTISE_2500FULL) + advertising |= ADVERTISED_2500baseX_Full; + if (tp->phy_2500_ctrl_reg & RTK_ADVERTISE_5000FULL) + advertising |= RTK_ADVERTISED_5000baseX_Full; + if (tp->phy_2500_ctrl_reg & RTK_ADVERTISE_10000FULL) + advertising |= ADVERTISED_10000baseT_Full; + } + + spin_lock_irqsave(&tp->phy_lock, flags); + rtl8127_mdio_write(tp, 0x1F, 0x0000); + bmcr = rtl8127_mdio_read(tp, MII_BMCR); + spin_unlock_irqrestore(&tp->phy_lock, flags); + if (bmcr & BMCR_ANENABLE) { + autoneg = AUTONEG_ENABLE; + advertising |= ADVERTISED_Autoneg; + } else { + autoneg = AUTONEG_DISABLE; + } + + advertising |= ADVERTISED_TP; + + status = RTL_R32(tp, PHYstatus); + if (netif_running(dev) && (status & LinkStatus)) + report_lpa = 1; + + if (report_lpa) { + /*link on*/ + speed = rtl8127_convert_link_speed(status); + + if (status & TxFlowCtrl) + advertising |= ADVERTISED_Asym_Pause; + + if (status & RxFlowCtrl) + advertising |= ADVERTISED_Pause; + + duplex = ((status & (_1000bpsF | _2500bpsF | _5000bpsF | _10000bpsF)) || + (status & FullDup)) ? + DUPLEX_FULL : DUPLEX_HALF; + + /*link partner*/ + if (aner & EXPANSION_NWAY) + lpa_adv |= ADVERTISED_Autoneg; + if (anlpar & LPA_10HALF) + lpa_adv |= ADVERTISED_10baseT_Half; + if (anlpar & LPA_10FULL) + lpa_adv |= ADVERTISED_10baseT_Full; + if (anlpar & LPA_100HALF) + lpa_adv |= ADVERTISED_100baseT_Half; + if (anlpar & LPA_100FULL) + lpa_adv |= ADVERTISED_100baseT_Full; + if (anlpar & LPA_PAUSE_CAP) + lpa_adv |= ADVERTISED_Pause; + if (anlpar & LPA_PAUSE_ASYM) + lpa_adv |= ADVERTISED_Asym_Pause; + if (gbsr & LPA_1000HALF) + lpa_adv |= ADVERTISED_1000baseT_Half; + if (gbsr & LPA_1000FULL) + lpa_adv |= ADVERTISED_1000baseT_Full; + if (status_2500 & RTK_LPA_ADVERTISE_2500FULL) + lpa_adv |= ADVERTISED_2500baseX_Full; + if (status_2500 & RTK_LPA_ADVERTISE_5000FULL) + lpa_adv |= RTK_ADVERTISED_5000baseX_Full; + if (status_2500 & RTK_LPA_ADVERTISE_10000FULL) + lpa_adv |= ADVERTISED_10000baseT_Full; + } else { + /*link down*/ + speed = SPEED_UNKNOWN; + duplex = DUPLEX_UNKNOWN; + lpa_adv = 0; + } + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + cmd->supported = (u32)supported; + cmd->advertising = (u32)advertising; + cmd->autoneg = autoneg; + cmd->speed = speed; + cmd->duplex = duplex; + cmd->port = PORT_TP; + cmd->lp_advertising = (u32)lpa_adv; + cmd->eth_tp_mdix = rtl8127_get_mdi_status(tp); +#else + ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported, + supported); + ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising, + advertising); + ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.lp_advertising, + lpa_adv); + + if (supported & SUPPORTED_2500baseX_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, + cmd->link_modes.supported, 1); + } + if (advertising & ADVERTISED_2500baseX_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, + cmd->link_modes.advertising, 1); + } + if (supported & RTK_SUPPORTED_5000baseX_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, + cmd->link_modes.supported, 1); + } + if (advertising & RTK_ADVERTISED_5000baseX_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, + cmd->link_modes.advertising, 1); + } + if (supported & SUPPORTED_10000baseT_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, + cmd->link_modes.supported, 1); + } + if (advertising & ADVERTISED_10000baseT_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, + cmd->link_modes.advertising, 1); + } + if (report_lpa) { + if (lpa_adv & ADVERTISED_2500baseX_Full) { + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, + cmd->link_modes.lp_advertising, 1); + } + if (lpa_adv & RTK_ADVERTISED_5000baseX_Full) + linkmode_mod_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, + cmd->link_modes.lp_advertising, 1); + if (lpa_adv & ADVERTISED_10000baseT_Full) + linkmode_mod_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, + cmd->link_modes.lp_advertising, 1); + } + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) + /* Use ETHTOOL_LINK_MODE_2500baseT_Full_BIT instead of + ETHTOOL_LINK_MODE_2500baseX_Full_BIT. */ + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseX_Full_BIT, + cmd->link_modes.supported, 0); + + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseX_Full_BIT, + cmd->link_modes.advertising, 0); + + linkmode_mod_bit(ETHTOOL_LINK_MODE_2500baseX_Full_BIT, + cmd->link_modes.lp_advertising, 0); +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) */ + + cmd->base.autoneg = autoneg; + cmd->base.speed = speed; + cmd->base.duplex = duplex; + cmd->base.port = PORT_TP; + cmd->base.eth_tp_mdix = rtl8127_get_mdi_status(tp); +#endif +} + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static int +rtl8127_get_settings(struct net_device *dev, +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + struct ethtool_cmd *cmd +#else + struct ethtool_link_ksettings *cmd +#endif + ) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + tp->get_settings(dev, cmd); + + return 0; +} + +static void rtl8127_get_regs(struct net_device *dev, struct ethtool_regs *regs, + void *p) +{ + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + unsigned int i; + u8 *data = p; + + if (regs->len < R8127_REGS_DUMP_SIZE) + return /* -EINVAL */; + + memset(p, 0, regs->len); + + for (i = 0; i < R8127_MAC_REGS_SIZE; i++) + *data++ = readb(ioaddr + i); + data = (u8*)p + 256; + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + for (i = 0; i < R8127_PHY_REGS_SIZE/2; i++) { + *(u16*)data = rtl8127_mdio_read(tp, i); + data += 2; + } + data = (u8*)p + 256 * 2; + + for (i = 0; i < R8127_EPHY_REGS_SIZE/2; i++) { + *(u16*)data = rtl8127_ephy_read(tp, i); + data += 2; + } + data = (u8*)p + 256 * 3; + + for (i = 0; i < R8127_ERI_REGS_SIZE; i+=4) { + *(u32*)data = rtl8127_eri_read(tp, i , 4, ERIAR_ExGMAC); + data += 4; + } +} + +static void rtl8127_get_pauseparam(struct net_device *dev, + struct ethtool_pauseparam *pause) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + pause->autoneg = (tp->autoneg ? AUTONEG_ENABLE : AUTONEG_DISABLE); + if (tp->fcpause == rtl8127_fc_rx_pause) + pause->rx_pause = 1; + else if (tp->fcpause == rtl8127_fc_tx_pause) + pause->tx_pause = 1; + else if (tp->fcpause == rtl8127_fc_full) { + pause->rx_pause = 1; + pause->tx_pause = 1; + } +} + +static int rtl8127_set_pauseparam(struct net_device *dev, + struct ethtool_pauseparam *pause) +{ + struct rtl8127_private *tp = netdev_priv(dev); + enum rtl8127_fc_mode newfc; + + if (pause->tx_pause || pause->rx_pause) + newfc = rtl8127_fc_full; + else + newfc = rtl8127_fc_none; + + if (tp->fcpause != newfc) { + tp->fcpause = newfc; + + rtl8127_set_speed(dev, tp->autoneg, tp->speed, tp->duplex, tp->advertising); + } + + return 0; + +} + +static u32 +rtl8127_get_msglevel(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + return tp->msg_enable; +} + +static void +rtl8127_set_msglevel(struct net_device *dev, + u32 value) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + tp->msg_enable = value; +} + +static const char rtl8127_gstrings[][ETH_GSTRING_LEN] = { + /* legacy */ + "tx_packets", + "rx_packets", + "tx_errors", + "rx_errors", + "rx_missed", + "align_errors", + "tx_single_collisions", + "tx_multi_collisions", + "unicast", + "broadcast", + "multicast", + "tx_aborted", + "tx_underrun", + + /* extended */ + "tx_octets", + "rx_octets", + "rx_multicast64", + "tx_unicast64", + "tx_broadcast64", + "tx_multicast64", + "tx_pause_on", + "tx_pause_off", + "tx_pause_all", + "tx_deferred", + "tx_late_collision", + "tx_all_collision", + "tx_aborted32", + "align_errors32", + "rx_frame_too_long", + "rx_runt", + "rx_pause_on", + "rx_pause_off", + "rx_pause_all", + "rx_unknown_opcode", + "rx_mac_error", + "tx_underrun32", + "rx_mac_missed", + "rx_tcam_dropped", + "tdu", + "rdu", +}; +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,33) +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static int rtl8127_get_stats_count(struct net_device *dev) +{ + return ARRAY_SIZE(rtl8127_gstrings); +} +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +#else +static int rtl8127_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return ARRAY_SIZE(rtl8127_gstrings); + default: + return -EOPNOTSUPP; + } +} +#endif + +static void +rtl8127_set_ring_size(struct rtl8127_private *tp, u32 rx, u32 tx) +{ + int i; + + for (i = 0; i < R8127_MAX_RX_QUEUES; i++) + tp->rx_ring[i].num_rx_desc = rx; + + for (i = 0; i < R8127_MAX_TX_QUEUES; i++) + tp->tx_ring[i].num_tx_desc = tx; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) +static void rtl8127_get_ringparam(struct net_device *dev, + struct ethtool_ringparam *ring, + struct kernel_ethtool_ringparam *kernel_ring, + struct netlink_ext_ack *extack) +#else +static void rtl8127_get_ringparam(struct net_device *dev, + struct ethtool_ringparam *ring) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + ring->rx_max_pending = MAX_NUM_TX_DESC; + ring->tx_max_pending = MAX_NUM_RX_DESC; + ring->rx_pending = tp->rx_ring[0].num_rx_desc; + ring->tx_pending = tp->tx_ring[0].num_tx_desc; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) +static int rtl8127_set_ringparam(struct net_device *dev, + struct ethtool_ringparam *ring, + struct kernel_ethtool_ringparam *kernel_ring, + struct netlink_ext_ack *extack) +#else +static int rtl8127_set_ringparam(struct net_device *dev, + struct ethtool_ringparam *ring) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u32 new_rx_count, new_tx_count; + int rc = 0; + + if ((ring->rx_mini_pending) || (ring->rx_jumbo_pending)) + return -EINVAL; + + new_tx_count = clamp_t(u32, ring->tx_pending, + MIN_NUM_TX_DESC, MAX_NUM_TX_DESC); + + new_rx_count = clamp_t(u32, ring->rx_pending, + MIN_NUM_RX_DESC, MAX_NUM_RX_DESC); + + if ((new_rx_count == tp->rx_ring[0].num_rx_desc) && + (new_tx_count == tp->tx_ring[0].num_tx_desc)) { + /* nothing to do */ + return 0; + } + + if (netif_running(dev)) { + rtl8127_wait_for_quiescence(dev); + rtl8127_close(dev); + } + + rtl8127_set_ring_size(tp, new_rx_count, new_tx_count); + + if (netif_running(dev)) + rc = rtl8127_open(dev); + + return rc; +} +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static void +rtl8127_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, + u64 *data) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct rtl8127_counters *counters; + dma_addr_t paddr; + + ASSERT_RTNL(); + + counters = tp->tally_vaddr; + paddr = tp->tally_paddr; + if (!counters) + return; + + rtl8127_dump_tally_counter(tp, paddr); + + data[0] = le64_to_cpu(counters->tx_packets); + data[1] = le64_to_cpu(counters->rx_packets); + data[2] = le64_to_cpu(counters->tx_errors); + data[3] = le32_to_cpu(counters->rx_errors); + data[4] = le16_to_cpu(counters->rx_missed); + data[5] = le16_to_cpu(counters->align_errors); + data[6] = le32_to_cpu(counters->tx_one_collision); + data[7] = le32_to_cpu(counters->tx_multi_collision); + data[8] = le64_to_cpu(counters->rx_unicast); + data[9] = le64_to_cpu(counters->rx_broadcast); + data[10] = le32_to_cpu(counters->rx_multicast); + data[11] = le16_to_cpu(counters->tx_aborted); + data[12] = le16_to_cpu(counters->tx_underrun); + + data[13] = le64_to_cpu(counters->tx_octets); + data[14] = le64_to_cpu(counters->rx_octets); + data[15] = le64_to_cpu(counters->rx_multicast64); + data[16] = le64_to_cpu(counters->tx_unicast64); + data[17] = le64_to_cpu(counters->tx_broadcast64); + data[18] = le64_to_cpu(counters->tx_multicast64); + data[19] = le32_to_cpu(counters->tx_pause_on); + data[20] = le32_to_cpu(counters->tx_pause_off); + data[21] = le32_to_cpu(counters->tx_pause_all); + data[22] = le32_to_cpu(counters->tx_deferred); + data[23] = le32_to_cpu(counters->tx_late_collision); + data[24] = le32_to_cpu(counters->tx_all_collision); + data[25] = le32_to_cpu(counters->tx_aborted32); + data[26] = le32_to_cpu(counters->align_errors32); + data[27] = le32_to_cpu(counters->rx_frame_too_long); + data[28] = le32_to_cpu(counters->rx_runt); + data[29] = le32_to_cpu(counters->rx_pause_on); + data[30] = le32_to_cpu(counters->rx_pause_off); + data[31] = le32_to_cpu(counters->rx_pause_all); + data[32] = le32_to_cpu(counters->rx_unknown_opcode); + data[33] = le32_to_cpu(counters->rx_mac_error); + data[34] = le32_to_cpu(counters->tx_underrun32); + data[35] = le32_to_cpu(counters->rx_mac_missed); + data[36] = le32_to_cpu(counters->rx_tcam_dropped); + data[37] = le32_to_cpu(counters->tdu); + data[38] = le32_to_cpu(counters->rdu); +} + +static void +rtl8127_get_strings(struct net_device *dev, + u32 stringset, + u8 *data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, rtl8127_gstrings, sizeof(rtl8127_gstrings)); + break; + } +} +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + +static int rtl_get_eeprom_len(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + return tp->eeprom_len; +} + +static int rtl_get_eeprom(struct net_device *dev, struct ethtool_eeprom *eeprom, u8 *buf) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i,j,ret; + int start_w, end_w; + int VPD_addr, VPD_data; + u32 *eeprom_buff; + u16 tmp; + + if (tp->eeprom_type == EEPROM_TYPE_NONE) { + dev_printk(KERN_DEBUG, tp_to_dev(tp), "Detect none EEPROM\n"); + return -EOPNOTSUPP; + } else if (eeprom->len == 0 || (eeprom->offset+eeprom->len) > tp->eeprom_len) { + dev_printk(KERN_DEBUG, tp_to_dev(tp), "Invalid parameter\n"); + return -EINVAL; + } + + VPD_addr = 0xD2; + VPD_data = 0xD4; + + start_w = eeprom->offset >> 2; + end_w = (eeprom->offset + eeprom->len - 1) >> 2; + + eeprom_buff = kmalloc(sizeof(u32)*(end_w - start_w + 1), GFP_KERNEL); + if (!eeprom_buff) + return -ENOMEM; + + rtl8127_enable_cfg9346_write(tp); + ret = -EFAULT; + for (i=start_w; i<=end_w; i++) { + pci_write_config_word(tp->pci_dev, VPD_addr, (u16)i*4); + ret = -EFAULT; + for (j = 0; j < 10; j++) { + fsleep(400); + pci_read_config_word(tp->pci_dev, VPD_addr, &tmp); + if (tmp&0x8000) { + ret = 0; + break; + } + } + + if (ret) + break; + + pci_read_config_dword(tp->pci_dev, VPD_data, &eeprom_buff[i-start_w]); + } + rtl8127_disable_cfg9346_write(tp); + + if (!ret) + memcpy(buf, (u8 *)eeprom_buff + (eeprom->offset & 3), eeprom->len); + + kfree(eeprom_buff); + + return ret; +} + +#undef ethtool_op_get_link +#define ethtool_op_get_link _kc_ethtool_op_get_link +static u32 _kc_ethtool_op_get_link(struct net_device *dev) +{ + return netif_carrier_ok(dev) ? 1 : 0; +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) +#undef ethtool_op_get_sg +#define ethtool_op_get_sg _kc_ethtool_op_get_sg +static u32 _kc_ethtool_op_get_sg(struct net_device *dev) +{ +#ifdef NETIF_F_SG + return (dev->features & NETIF_F_SG) != 0; +#else + return 0; +#endif +} + +#undef ethtool_op_set_sg +#define ethtool_op_set_sg _kc_ethtool_op_set_sg +static int _kc_ethtool_op_set_sg(struct net_device *dev, u32 data) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->mcfg == CFG_METHOD_DEFAULT) + return -EOPNOTSUPP; + +#ifdef NETIF_F_SG + if (data) + dev->features |= NETIF_F_SG; + else + dev->features &= ~NETIF_F_SG; +#endif + + return 0; +} +#endif + +static void +rtl8127_set_eee_lpi_timer(struct rtl8127_private *tp) +{ + u16 dev_lpi_timer; + + dev_lpi_timer = tp->eee.tx_lpi_timer; + + RTL_W16(tp, EEE_TXIDLE_TIMER_8125, dev_lpi_timer); +} + +static bool rtl8127_is_adv_eee_enabled(struct rtl8127_private *tp) +{ + if (rtl8127_mdio_direct_read_phy_ocp(tp, 0xA430) & BIT_15) + return true; + else + return false; +} + +static void rtl8127_disable_adv_eee(struct rtl8127_private *tp) +{ + bool lock; + + if (rtl8127_is_adv_eee_enabled(tp)) + lock = true; + else + lock = false; + + if (lock) + rtl8127_set_phy_mcu_patch_request(tp); + + rtl8127_clear_mac_ocp_bit(tp, 0xE052, BIT_0); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA442, BIT_12 | BIT_13); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA430, BIT_15); + + if (lock) + rtl8127_clear_phy_mcu_patch_request(tp); +} + +static int rtl8127_enable_eee(struct rtl8127_private *tp) +{ + struct ethtool_keee *eee = &tp->eee; + u16 eee_adv_cap1_t = rtl8127_ethtool_adv_to_mmd_eee_adv_cap1_t(eee->advertised); + u16 eee_adv_cap2_t = rtl8127_ethtool_adv_to_mmd_eee_adv_cap2_t(eee->advertised); + + rtl8127_set_mac_ocp_bit(tp, 0xE040, (BIT_1|BIT_0)); + + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA5D0, + MDIO_EEE_100TX | MDIO_EEE_1000T | MDIO_EEE_10GT, + eee_adv_cap1_t); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA6D4, + MDIO_EEE_2_5GT | MDIO_EEE_5GT, + eee_adv_cap2_t); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA6D8, BIT_4); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA428, BIT_7); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA4A2, BIT_9); + + /*Advanced EEE*/ + rtl8127_disable_adv_eee(tp); + + return 0; +} + +static int rtl8127_disable_eee(struct rtl8127_private *tp) +{ + rtl8127_clear_mac_ocp_bit(tp, 0xE040, (BIT_1|BIT_0)); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5D0, + (MDIO_EEE_100TX | MDIO_EEE_1000T | MDIO_EEE_10GT)); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA6D4, + (MDIO_EEE_2_5GT | MDIO_EEE_5GT)); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA6D8, BIT_4); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA428, BIT_7); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA4A2, BIT_9); + + /*Advanced EEE*/ + rtl8127_disable_adv_eee(tp); + + return 0; +} + +static int rtl_nway_reset(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret, bmcr; + + if (unlikely(tp->rtk_enable_diag)) + return -EBUSY; + + /* if autoneg is off, it's an error */ + rtl8127_mdio_write(tp, 0x1F, 0x0000); + bmcr = rtl8127_mdio_read(tp, MII_BMCR); + + if (bmcr & BMCR_ANENABLE) { + bmcr |= BMCR_ANRESTART; + rtl8127_mdio_write(tp, MII_BMCR, bmcr); + ret = 0; + } else { + ret = -EINVAL; + } + + return ret; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,6,0) +static u32 +rtl8127_device_lpi_t_to_ethtool_lpi_t(struct rtl8127_private *tp , u32 lpi_timer) +{ + u32 to_us; + u16 status; + + to_us = lpi_timer * 80; + status = RTL_R16(tp, PHYstatus); + if (status & LinkStatus) { + /*link on*/ + if (HW_SUPP_PHY_LINK_SPEED_10000M(tp)) { + //5G : lpi_timer * 12.8ns + //2.5G : lpi_timer * 25.6ns + //Giga: lpi_timer * 8ns + //100M : lpi_timer * 80ns + if (status & (_10000bpsF)) + to_us = (lpi_timer * 128) / 10; + else if (status & (_5000bpsF)) + to_us = (lpi_timer * 128) / 10; + else if (status & _2500bpsF) + to_us = (lpi_timer * 256) / 10; + else if (status & _1000bpsF) + to_us = lpi_timer * 8; + } else if (HW_SUPP_PHY_LINK_SPEED_5000M(tp)) { + //5G : lpi_timer * 12.8ns + //2.5G : lpi_timer * 25.6ns + //Giga: lpi_timer * 8ns + //100M : lpi_timer * 80ns + if (status & (_5000bpsF)) + to_us = (lpi_timer * 128) / 10; + else if (status & _2500bpsF) + to_us = (lpi_timer * 256) / 10; + else if (status & _1000bpsF) + to_us = lpi_timer * 8; + } else { + //2.5G : lpi_timer * 3.2ns + //Giga: lpi_timer * 8ns + //100M : lpi_timer * 80ns + if (status & _2500bpsF) + to_us = (lpi_timer * 32) / 10; + else if (status & _1000bpsF) + to_us = lpi_timer * 8; + } + } + + //ns to us + to_us /= 1000; + + return to_us; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,9,0) +static void +rtl8127_adv_to_linkmode(unsigned long *mode, u64 adv) +{ + linkmode_zero(mode); + + if (adv & ADVERTISED_10baseT_Half) + linkmode_set_bit(ETHTOOL_LINK_MODE_10baseT_Half_BIT, mode); + if (adv & ADVERTISED_10baseT_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_10baseT_Full_BIT, mode); + if (adv & ADVERTISED_100baseT_Half) + linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Half_BIT, mode); + if (adv & ADVERTISED_100baseT_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Full_BIT, mode); + if (adv & ADVERTISED_1000baseT_Half) + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Half_BIT, mode); + if (adv & ADVERTISED_1000baseT_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, mode); + if (adv & ADVERTISED_2500baseX_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, mode); + if (adv & RTK_ADVERTISED_5000baseX_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, mode); + if (adv & ADVERTISED_10000baseT_Full) + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, mode); +} + +static int +rtl_ethtool_get_eee(struct net_device *net, struct ethtool_keee *edata) +{ + __ETHTOOL_DECLARE_LINK_MODE_MASK(common); + struct rtl8127_private *tp = netdev_priv(net); + struct ethtool_keee *eee = &tp->eee; + u32 tx_lpi_timer; + u16 val; + + if (unlikely(tp->rtk_enable_diag)) + return -EBUSY; + + /* Get LP advertisement EEE */ + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D2); + mii_eee_cap1_mod_linkmode_t(edata->lp_advertised, val); + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA6D0); + mii_eee_cap2_mod_linkmode_sup_t(edata->lp_advertised, val); + + /* Get EEE Tx LPI timer*/ + tx_lpi_timer = rtl8127_device_lpi_t_to_ethtool_lpi_t(tp, eee->tx_lpi_timer); + + val = rtl8127_mac_ocp_read(tp, 0xE040); + val &= BIT_1 | BIT_0; + + edata->eee_enabled = !!val; + linkmode_copy(edata->supported, eee->supported); + linkmode_copy(edata->advertised, eee->advertised); + edata->tx_lpi_enabled = edata->eee_enabled; + edata->tx_lpi_timer = tx_lpi_timer; + linkmode_and(common, edata->advertised, edata->lp_advertised); + edata->eee_active = !linkmode_empty(common); + + return 0; +} + +static int +rtl_ethtool_set_eee(struct net_device *net, struct ethtool_keee *edata) +{ + __ETHTOOL_DECLARE_LINK_MODE_MASK(advertising); + __ETHTOOL_DECLARE_LINK_MODE_MASK(tmp); + struct rtl8127_private *tp = netdev_priv(net); + struct ethtool_keee *eee = &tp->eee; + int rc = 0; + + if (!HW_HAS_WRITE_PHY_MCU_RAM_CODE(tp) || + tp->DASH) + return -EOPNOTSUPP; + + if (unlikely(tp->rtk_enable_diag)) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "Diag Enabled\n"); + rc = -EBUSY; + goto out; + } + + if (tp->autoneg != AUTONEG_ENABLE) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE requires autoneg\n"); + rc = -EINVAL; + goto out; + } + + /* + if (edata->tx_lpi_enabled) { + if (edata->tx_lpi_timer > tp->max_jumbo_frame_size || + edata->tx_lpi_timer < ETH_MIN_MTU) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "Valid LPI timer range is %d to %d. \n", + ETH_MIN_MTU, tp->max_jumbo_frame_size); + rc = -EINVAL; + goto out; + } + } + */ + + rtl8127_adv_to_linkmode(advertising, tp->advertising); + if (linkmode_empty(edata->advertised)) { + linkmode_and(edata->advertised, advertising, eee->supported); + } else if (linkmode_andnot(tmp, edata->advertised, advertising)) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE advertised must be a subset of autoneg advertised speeds\n"); + rc = -EINVAL; + goto out; + } + + if (linkmode_andnot(tmp, edata->advertised, eee->supported)) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE advertised must be a subset of support \n"); + rc = -EINVAL; + goto out; + } + + //tp->eee.eee_enabled = edata->eee_enabled; + //tp->eee_adv_t = rtl8127_ethtool_adv_to_mmd_eee_adv_cap1_t(edata->advertised); + + linkmode_copy(eee->advertised, edata->advertised); + //eee->tx_lpi_enabled = edata->tx_lpi_enabled; + //eee->tx_lpi_timer = edata->tx_lpi_timer; + eee->eee_enabled = edata->eee_enabled; + + if (eee->eee_enabled) + rtl8127_enable_eee(tp); + else + rtl8127_disable_eee(tp); + + rtl_nway_reset(net); + +out: + return rc; +} +#else +static int +rtl_ethtool_get_eee(struct net_device *net, struct ethtool_eee *edata) +{ + struct rtl8127_private *tp = netdev_priv(net); + struct ethtool_eee *eee = &tp->eee; + u32 lp, adv, tx_lpi_timer, supported = 0; + u16 val; + + if (unlikely(tp->rtk_enable_diag)) + return -EBUSY; + + /* Get Supported EEE */ + //val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5C4); + //supported = mmd_eee_cap_to_ethtool_sup_t(val); + supported = eee->supported; + + /* Get advertisement EEE */ + adv = eee->advertised; + + /* Get LP advertisement EEE */ + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA5D2); + lp = mmd_eee_adv_to_ethtool_adv_t(val); + val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA6D0); + if (val & RTK_LPA_EEE_ADVERTISE_2500FULL) + lp |= ADVERTISED_2500baseX_Full; + + /* Get EEE Tx LPI timer*/ + tx_lpi_timer = rtl8127_device_lpi_t_to_ethtool_lpi_t(tp, eee->tx_lpi_timer); + + val = rtl8127_mac_ocp_read(tp, 0xE040); + val &= BIT_1 | BIT_0; + + edata->eee_enabled = !!val; + edata->eee_active = !!(supported & adv & lp); + edata->supported = supported; + edata->advertised = adv; + edata->lp_advertised = lp; + edata->tx_lpi_enabled = edata->eee_enabled; + edata->tx_lpi_timer = tx_lpi_timer; + + return 0; +} + +static int +rtl_ethtool_set_eee(struct net_device *net, struct ethtool_eee *edata) +{ + struct rtl8127_private *tp = netdev_priv(net); + struct ethtool_eee *eee = &tp->eee; + u32 advertising; + int rc = 0; + + if (!HW_HAS_WRITE_PHY_MCU_RAM_CODE(tp) || + tp->DASH) + return -EOPNOTSUPP; + + if (unlikely(tp->rtk_enable_diag)) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "Diag Enabled\n"); + rc = -EBUSY; + goto out; + } + + if (tp->autoneg != AUTONEG_ENABLE) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE requires autoneg\n"); + rc = -EINVAL; + goto out; + } + + /* + if (edata->tx_lpi_enabled) { + if (edata->tx_lpi_timer > tp->max_jumbo_frame_size || + edata->tx_lpi_timer < ETH_MIN_MTU) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "Valid LPI timer range is %d to %d. \n", + ETH_MIN_MTU, tp->max_jumbo_frame_size); + rc = -EINVAL; + goto out; + } + } + */ + + advertising = tp->advertising; + if (!edata->advertised) { + edata->advertised = advertising & eee->supported; + } else if (edata->advertised & ~advertising) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE advertised %x must be a subset of autoneg advertised speeds %x\n", + edata->advertised, advertising); + rc = -EINVAL; + goto out; + } + + if (edata->advertised & ~eee->supported) { + dev_printk(KERN_WARNING, tp_to_dev(tp), "EEE advertised %x must be a subset of support %x\n", + edata->advertised, eee->supported); + rc = -EINVAL; + goto out; + } + + //tp->eee.eee_enabled = edata->eee_enabled; + //tp->eee_adv_t = ethtool_adv_to_mmd_eee_adv_t(edata->advertised); + + eee->advertised = edata->advertised; + //eee->tx_lpi_enabled = edata->tx_lpi_enabled; + //eee->tx_lpi_timer = edata->tx_lpi_timer; + eee->eee_enabled = edata->eee_enabled; + + if (eee->eee_enabled) + rtl8127_enable_eee(tp); + else + rtl8127_disable_eee(tp); + + rtl_nway_reset(net); + +out: + return rc; +} +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,9,0) */ +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,6,0) */ + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) +static void rtl8127_get_channels(struct net_device *dev, + struct ethtool_channels *channel) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + channel->max_rx = tp->HwSuppNumRxQueues; + channel->max_tx = tp->HwSuppNumTxQueues; + channel->rx_count = tp->num_rx_rings; + channel->tx_count = tp->num_tx_rings; +} +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) */ + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) +static const struct ethtool_ops rtl8127_ethtool_ops = { + .get_drvinfo = rtl8127_get_drvinfo, + .get_regs_len = rtl8127_get_regs_len, + .get_link = ethtool_op_get_link, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) + .get_ringparam = rtl8127_get_ringparam, + .set_ringparam = rtl8127_set_ringparam, +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) + .get_settings = rtl8127_get_settings, + .set_settings = rtl8127_set_settings, +#else + .get_link_ksettings = rtl8127_get_settings, + .set_link_ksettings = rtl8127_set_settings, +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) + .get_pauseparam = rtl8127_get_pauseparam, + .set_pauseparam = rtl8127_set_pauseparam, +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,0) + .get_msglevel = rtl8127_get_msglevel, + .set_msglevel = rtl8127_set_msglevel, +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) + .get_rx_csum = rtl8127_get_rx_csum, + .set_rx_csum = rtl8127_set_rx_csum, + .get_tx_csum = rtl8127_get_tx_csum, + .set_tx_csum = rtl8127_set_tx_csum, + .get_sg = ethtool_op_get_sg, + .set_sg = ethtool_op_set_sg, +#ifdef NETIF_F_TSO + .get_tso = ethtool_op_get_tso, + .set_tso = ethtool_op_set_tso, +#endif //NETIF_F_TSO +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,3,0) + .get_regs = rtl8127_get_regs, + .get_wol = rtl8127_get_wol, + .set_wol = rtl8127_set_wol, + .get_strings = rtl8127_get_strings, +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,33) + .get_stats_count = rtl8127_get_stats_count, +#else + .get_sset_count = rtl8127_get_sset_count, +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,33) + .get_ethtool_stats = rtl8127_get_ethtool_stats, +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,23) +#ifdef ETHTOOL_GPERMADDR + .get_perm_addr = ethtool_op_get_perm_addr, +#endif //ETHTOOL_GPERMADDR +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,23) + .get_eeprom = rtl_get_eeprom, + .get_eeprom_len = rtl_get_eeprom_len, +#ifdef ENABLE_RSS_SUPPORT + .get_rxnfc = rtl8127_get_rxnfc, + .set_rxnfc = rtl8127_set_rxnfc, + .get_rxfh_indir_size = rtl8127_rss_indir_size, + .get_rxfh_key_size = rtl8127_get_rxfh_key_size, + .get_rxfh = rtl8127_get_rxfh, + .set_rxfh = rtl8127_set_rxfh, +#endif //ENABLE_RSS_SUPPORT +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) +#ifdef ENABLE_PTP_SUPPORT + .get_ts_info = rtl8127_get_ts_info, +#else + .get_ts_info = ethtool_op_get_ts_info, +#endif //ENABLE_PTP_SUPPORT +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,5,0) +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,6,0) + .get_eee = rtl_ethtool_get_eee, + .set_eee = rtl_ethtool_set_eee, +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,6,0) */ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) + .get_channels = rtl8127_get_channels, +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) */ + .nway_reset = rtl_nway_reset, + +}; +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + +static void rtl8127_get_mac_version(struct rtl8127_private *tp) +{ + u32 reg,val32; + u32 ICVerID; + + val32 = RTL_R32(tp, TxConfig); + reg = val32 & 0x7c800000; + ICVerID = val32 & 0x00700000; + + switch (reg) { + case 0x6C800000: + if (ICVerID == 0x00000000) { + tp->mcfg = CFG_METHOD_1; + } else if (ICVerID == 0x100000) { + tp->mcfg = CFG_METHOD_2; + } else { + tp->mcfg = CFG_METHOD_2; + tp->HwIcVerUnknown = TRUE; + } + + tp->efuse_ver = EFUSE_SUPPORT_V4; + break; + default: + printk("unknown chip version (%x)\n",reg); + tp->mcfg = CFG_METHOD_DEFAULT; + tp->HwIcVerUnknown = TRUE; + tp->efuse_ver = EFUSE_NOT_SUPPORT; + break; + } +} + +static void +rtl8127_print_mac_version(struct rtl8127_private *tp) +{ + int i; + for (i = ARRAY_SIZE(rtl_chip_info) - 1; i >= 0; i--) { + if (tp->mcfg == rtl_chip_info[i].mcfg) { + dprintk("Realtek %s Ethernet controller mcfg = %04d\n", + MODULENAME, rtl_chip_info[i].mcfg); + return; + } + } + + dprintk("mac_version == Unknown\n"); +} + +static void +rtl8127_tally_counter_addr_fill(struct rtl8127_private *tp) +{ + if (!tp->tally_paddr) + return; + + RTL_W32(tp, CounterAddrHigh, (u64)tp->tally_paddr >> 32); + RTL_W32(tp, CounterAddrLow, (u64)tp->tally_paddr & (DMA_BIT_MASK(32))); +} + +static void +rtl8127_tally_counter_clear(struct rtl8127_private *tp) +{ + if (!tp->tally_paddr) + return; + + RTL_W32(tp, CounterAddrHigh, (u64)tp->tally_paddr >> 32); + RTL_W32(tp, CounterAddrLow, ((u64)tp->tally_paddr & (DMA_BIT_MASK(32))) | CounterReset); +} + +static void +rtl8127_clear_phy_ups_reg(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA466, BIT_0); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA468, BIT_3 | BIT_1); +} + +static int +rtl8127_is_ups_resume(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + return (rtl8127_mac_ocp_read(tp, 0xD42C) & BIT_8); +} + +static void +rtl8127_clear_ups_resume_bit(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_clear_mac_ocp_bit(tp, 0xD42C, BIT_8); +} + +static u8 +rtl8127_get_phy_state(struct rtl8127_private *tp) +{ + return (rtl8127_mdio_direct_read_phy_ocp(tp, 0xA420) & 0x7); +} + +static void +rtl8127_wait_phy_ups_resume(struct net_device *dev, u16 PhyState) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + for (i=0; i< 100; i++) { + if (rtl8127_get_phy_state(tp) == PhyState) + break; + else + mdelay(1); + } + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,18) + WARN_ON_ONCE(i == 100); +#endif +} + +void +rtl8127_enable_now_is_oob(struct rtl8127_private *tp) +{ + if (tp->HwSuppNowIsOobVer == 1) + RTL_W8(tp, MCUCmd_reg, RTL_R8(tp, MCUCmd_reg) | Now_is_oob); +} + +void +rtl8127_disable_now_is_oob(struct rtl8127_private *tp) +{ + if (tp->HwSuppNowIsOobVer == 1) + RTL_W8(tp, MCUCmd_reg, RTL_R8(tp, MCUCmd_reg) & ~Now_is_oob); +} + +static void +rtl8127_exit_oob(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u16 data16; + + rtl8127_disable_rx_packet_filter(tp); + + if (HW_DASH_SUPPORT_DASH(tp)) { + rtl8127_driver_start(tp); + rtl8127_dash2_disable_txrx(dev); +#ifdef ENABLE_DASH_SUPPORT + DashHwInit(dev); +#endif + } + +#ifdef ENABLE_REALWOW_SUPPORT + rtl8127_realwow_hw_init(dev); +#else + rtl8127_mac_ocp_write(tp, 0xC0BC, 0x00FF); +#endif //ENABLE_REALWOW_SUPPORT + + rtl8127_nic_reset(dev); + + rtl8127_disable_now_is_oob(tp); + + data16 = rtl8127_mac_ocp_read(tp, 0xE8DE) & ~BIT_14; + rtl8127_mac_ocp_write(tp, 0xE8DE, data16); + rtl8127_wait_ll_share_fifo_ready(dev); + + rtl8127_mac_ocp_write(tp, 0xC0AA, 0x07D0); +#ifdef ENABLE_LIB_SUPPORT + rtl8127_mac_ocp_write(tp, 0xC0A6, 0x04E2); +#else + rtl8127_mac_ocp_write(tp, 0xC0A6, 0x01B5); +#endif + rtl8127_mac_ocp_write(tp, 0xC01E, 0x5555); + + rtl8127_wait_ll_share_fifo_ready(dev); + + //wait ups resume (phy state 2) + if (rtl8127_is_ups_resume(dev)) { + rtl8127_wait_phy_ups_resume(dev, 2); + rtl8127_clear_ups_resume_bit(dev); + rtl8127_clear_phy_ups_reg(dev); + } +} + +void +rtl8127_hw_disable_mac_mcu_bps(struct net_device *dev) +{ + u16 regAddr; + + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_enable_aspm_clkreq_lock(tp, 0); + + rtl8127_mac_ocp_write(tp, 0xFC48, 0x0000); + + for (regAddr = 0xFC28; regAddr < 0xFC48; regAddr += 2) { + rtl8127_mac_ocp_write(tp, regAddr, 0x0000); + } + + fsleep(3000); + + rtl8127_mac_ocp_write(tp, 0xFC26, 0x0000); +} + +#ifndef ENABLE_USE_FIRMWARE_FILE +static void +rtl8127_switch_mac_mcu_ram_code_page(struct rtl8127_private *tp, u16 page) +{ + u16 tmpUshort; + + page &= (BIT_1 | BIT_0); + tmpUshort = rtl8127_mac_ocp_read(tp, 0xE446); + tmpUshort &= ~(BIT_1 | BIT_0); + tmpUshort |= page; + rtl8127_mac_ocp_write(tp, 0xE446, tmpUshort); +} + +static void +_rtl8127_set_hw_mcu_patch_code_ver(struct rtl8127_private *tp, u64 ver) +{ + int i; + + /* Switch to page 2 */ + rtl8127_switch_mac_mcu_ram_code_page(tp, 2); + + for (i = 0; i < 8; i += 2) { + rtl8127_mac_ocp_write(tp, 0xF9F8 + 6 - i, (u16)ver); + ver >>= 16; + } + + /* Switch back to page 0 */ + rtl8127_switch_mac_mcu_ram_code_page(tp, 0); +} + +static void +rtl8127_set_hw_mcu_patch_code_ver(struct rtl8127_private *tp, u64 ver) +{ + _rtl8127_set_hw_mcu_patch_code_ver(tp, ver); + + tp->hw_mcu_patch_code_ver = ver; +} + +static u64 +rtl8127_get_hw_mcu_patch_code_ver(struct rtl8127_private *tp) +{ + u64 ver; + int i; + + /* Switch to page 2 */ + rtl8127_switch_mac_mcu_ram_code_page(tp, 2); + + ver = 0; + for (i = 0; i < 8; i += 2) { + ver <<= 16; + ver |= rtl8127_mac_ocp_read(tp, 0xF9F8 + i); + } + + /* Switch back to page 0 */ + rtl8127_switch_mac_mcu_ram_code_page(tp, 0); + + return ver; +} + +static u64 +rtl8127_get_bin_mcu_patch_code_ver(const u16 *entry, u16 entry_cnt) +{ + u64 ver; + int i; + + if (entry == NULL || entry_cnt == 0 || entry_cnt < 4) + return 0; + + ver = 0; + for (i = 0; i < 4; i++) { + ver <<= 16; + ver |= entry[entry_cnt - 4 + i]; + } + + return ver; +} + +static void +_rtl8127_write_mac_mcu_ram_code(struct rtl8127_private *tp, const u16 *entry, u16 entry_cnt) +{ + u16 i; + + for (i = 0; i < entry_cnt; i++) + rtl8127_mac_ocp_write(tp, 0xF800 + i * 2, entry[i]); +} + +static void +_rtl8127_write_mac_mcu_ram_code_with_page(struct rtl8127_private *tp, const u16 *entry, u16 entry_cnt, u16 page_size) +{ + u16 i; + u16 offset; + + if (page_size == 0) + return; + + for (i = 0; i < entry_cnt; i++) { + offset = i % page_size; + if (offset == 0) { + u16 page = (i / page_size); + rtl8127_switch_mac_mcu_ram_code_page(tp, page); + } + rtl8127_mac_ocp_write(tp, 0xF800 + offset * 2, entry[i]); + } +} + +static void +rtl8127_write_mac_mcu_ram_code(struct rtl8127_private *tp, const u16 *entry, u16 entry_cnt) +{ + if (FALSE == HW_SUPPORT_MAC_MCU(tp)) + return; + + if (entry == NULL || entry_cnt == 0) + return; + + if (tp->MacMcuPageSize > 0) + _rtl8127_write_mac_mcu_ram_code_with_page(tp, entry, entry_cnt, tp->MacMcuPageSize); + else + _rtl8127_write_mac_mcu_ram_code(tp, entry, entry_cnt); + + if (tp->bin_mcu_patch_code_ver > 0) + rtl8127_set_hw_mcu_patch_code_ver(tp, tp->bin_mcu_patch_code_ver); +} + +static void +rtl8127_set_mac_mcu_8127a_tc(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + static const u16 mcu_patch_code[] = { + 0xE010, 0xE019, 0xE01B, 0xE01D, 0xE029, 0xE02C, 0xE0E1, 0xE192, 0xE194, + 0xE196, 0xE198, 0xE19A, 0xE19C, 0xE19E, 0xE1A0, 0xE1A2, 0xC008, 0x7100, + 0x4897, 0x9900, 0xC005, 0xC602, 0xBE00, 0x3D8E, 0xD428, 0xD400, 0xC602, + 0xBE00, 0x0000, 0xC502, 0xBD00, 0x0000, 0x48C3, 0x4847, 0x48C1, 0x8CF8, + 0x74F8, 0x74F8, 0x74F8, 0x4842, 0x8CF8, 0x1E10, 0xC502, 0xBD00, 0x14BA, + 0x1E10, 0xC502, 0xBD00, 0x14EE, 0xC643, 0x76C0, 0x49E1, 0xF13F, 0xC140, + 0x7720, 0x49E0, 0xF003, 0x1B00, 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, + 0x49E4, 0xF003, 0x1B08, 0xE002, 0x1B0C, 0x21B8, 0x1A0E, 0x44DA, 0xE893, + 0x481C, 0xE884, 0xE001, 0x49E0, 0xF003, 0x1B00, 0xE00A, 0x49E2, 0xF003, + 0x1B04, 0xE006, 0x49E4, 0xF003, 0x1B08, 0xE002, 0x1B0C, 0x21B8, 0x1A12, + 0x44DA, 0xE87F, 0x481F, 0xE870, 0xE001, 0x49E0, 0xF003, 0x1B00, 0xE00A, + 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E4, 0xF003, 0x1B08, 0xE002, 0x1B0C, + 0x21B8, 0x1A1C, 0x44DA, 0xE86B, 0x481F, 0xE85C, 0xE004, 0xE04F, 0xDD98, + 0xD450, 0x49E0, 0xF003, 0x1B00, 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, + 0x49E4, 0xF003, 0x1B08, 0xE002, 0x1B0C, 0x21B8, 0x1A0E, 0x44DA, 0xE854, + 0x489E, 0x481F, 0xE844, 0xE001, 0x1908, 0xE83E, 0x49E0, 0xF003, 0x1B00, + 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E4, 0xF003, 0x1B08, 0xE002, + 0x1B0C, 0x21B8, 0x1A8A, 0x44DA, 0xE83D, 0x4813, 0xE82E, 0x49F9, 0xF106, + 0x4838, 0xE837, 0x4813, 0xE828, 0xE001, 0x49E0, 0xF003, 0x1B00, 0xE00A, + 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E4, 0xF003, 0x1B08, 0xE002, 0x1B0C, + 0x21B8, 0x1A84, 0x44DA, 0xE823, 0x4890, 0x4811, 0xE813, 0x49F9, 0xF106, + 0x4838, 0xE81C, 0x4890, 0x4811, 0xE80C, 0xC207, 0x7440, 0xC602, 0xBE00, + 0x14CC, 0x0FFE, 0xDE20, 0xE092, 0xC3FD, 0xE802, 0xFF80, 0xC0FB, 0x7202, + 0x49AE, 0xF1FE, 0x9900, 0x44D3, 0x4413, 0x482F, 0x9A02, 0x7202, 0x49AE, + 0xF1FE, 0xFF80, 0xC0EE, 0x7202, 0x49AE, 0xF1FE, 0x44D3, 0x4413, 0x48AF, + 0x9A02, 0x7202, 0x49AE, 0xF1FE, 0x7100, 0xFF80, 0xB401, 0xB402, 0xB404, + 0xB407, 0xC61F, 0x76C0, 0x49E1, 0xF164, 0xC11C, 0x7720, 0x1906, 0xE88A, + 0x1B0C, 0x21B8, 0x1A40, 0x44DA, 0xE895, 0x4810, 0xE886, 0x190C, 0xE881, + 0x1B08, 0x21B8, 0x1A26, 0x44DA, 0xE88C, 0x4890, 0x4891, 0xE87C, 0x49F9, + 0xF107, 0x4898, 0x4899, 0xE877, 0xE003, 0xDD98, 0xD450, 0x1908, 0xE86F, + 0x49E0, 0xF003, 0x1B00, 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E2, + 0xF003, 0x1B08, 0xE002, 0x1B0C, 0x21B8, 0x1A5C, 0x44DA, 0xE86E, 0x4897, + 0x4898, 0x4819, 0x481A, 0xE85C, 0x49F9, 0xF109, 0x4838, 0xE865, 0x4897, + 0x4898, 0x4819, 0x481A, 0xE853, 0xE001, 0x190A, 0xE84D, 0x1B00, 0xE85B, + 0x44E1, 0x4838, 0xE858, 0x44E9, 0x1908, 0xE845, 0x49E0, 0xF003, 0x1B00, + 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E4, 0xF003, 0x1B08, 0xE002, + 0x1B0C, 0x21B8, 0x1A86, 0x44DA, 0xE844, 0x44CC, 0xE835, 0x49F9, 0xF108, + 0x4838, 0xE83E, 0x44CD, 0xE82F, 0xE003, 0xE021, 0xFFC0, 0x190A, 0xE827, + 0x1B00, 0x4839, 0xE834, 0x249A, 0x1C00, 0x44E1, 0x1909, 0xE81F, 0x49E0, + 0xF003, 0x1B00, 0xE00A, 0x49E2, 0xF003, 0x1B04, 0xE006, 0x49E4, 0xF003, + 0x1B08, 0xE002, 0x1B0C, 0x21B8, 0x1A1A, 0x44DA, 0xE81E, 0xC5E4, 0x414D, + 0x418C, 0xE80D, 0xB007, 0xB004, 0xB002, 0xB001, 0xC602, 0xBE00, 0x14B2, + 0x0FFE, 0xDE20, 0xC3FE, 0xE802, 0xFF80, 0xC0FC, 0x7202, 0x49AE, 0xF1FE, + 0x9900, 0x44D3, 0x4413, 0x482F, 0x9A02, 0x7202, 0x49AE, 0xF1FE, 0xFF80, + 0xC0EF, 0x7202, 0x49AE, 0xF1FE, 0x44D3, 0x4413, 0x48AF, 0x9A02, 0x7202, + 0x49AE, 0xF1FE, 0x7100, 0xFF80, 0xC502, 0xBD00, 0x0000, 0xC502, 0xBD00, + 0x0000, 0xC502, 0xBD00, 0x0000, 0xC302, 0xBB00, 0x0000, 0xC602, 0xBE00, + 0x0000, 0xC102, 0xB900, 0x0000, 0xC102, 0xB900, 0x0000, 0xC602, 0xBE00, + 0x0000, 0xC602, 0xBE00, 0x0000, 0x1332, 0x0018, 0x0C05, 0x140D + }; + + /* Get BIN mac mcu patch code version */ + tp->bin_mcu_patch_code_ver = rtl8127_get_bin_mcu_patch_code_ver(mcu_patch_code, ARRAY_SIZE(mcu_patch_code)); + + if (tp->hw_mcu_patch_code_ver != tp->bin_mcu_patch_code_ver) + rtl8127_write_mac_mcu_ram_code(tp, mcu_patch_code, ARRAY_SIZE(mcu_patch_code)); + + rtl8127_mac_ocp_write(tp, 0xFC26, 0x8000); + + rtl8127_mac_ocp_write(tp, 0xFC2E, 0x14B8); + rtl8127_mac_ocp_write(tp, 0xFC30, 0x14EC); + + rtl8127_mac_ocp_write(tp, 0xFC48, 0x0018); +} + +static void +_rtl8127_set_mac_mcu_8127a_1(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + static const u16 mcu_patch_code[] = { + 0xE010, 0xE014, 0xE018, 0xE01C, 0xE020, 0xE033, 0xE035, 0xE037, 0xE039, + 0xE03B, 0xE03D, 0xE03F, 0xE041, 0xE043, 0xE045, 0xE047, 0x7020, 0x4809, + 0xC502, 0xBD00, 0x1522, 0x7760, 0x4879, 0xC002, 0xB800, 0x41E2, 0x7160, + 0x4819, 0xC302, 0xBB00, 0x508E, 0x7720, 0x4879, 0xC102, 0xB900, 0x50F8, + 0x9F86, 0xB400, 0xB401, 0xB402, 0xB403, 0xC00D, 0x7100, 0xC20C, 0x7340, + 0x418B, 0x9900, 0xB003, 0xB002, 0xB001, 0xB000, 0xC702, 0xBF00, 0x3550, + 0xFC48, 0xD482, 0xC602, 0xBE00, 0x0000, 0xC602, 0xBE00, 0x0000, 0xC102, + 0xB900, 0x0000, 0xC302, 0xBB00, 0x0000, 0xC002, 0xB800, 0x0000, 0xC002, + 0xB800, 0x0000, 0xC502, 0xBD00, 0x0000, 0xC102, 0xB900, 0x0000, 0xC102, + 0xB900, 0x0000, 0xC602, 0xBE00, 0x0000, 0xC602, 0xBE00, 0x0000, 0x6961, + 0x0019, 0x0311, 0x1431 + }; + + /* Get BIN mac mcu patch code version */ + tp->bin_mcu_patch_code_ver = rtl8127_get_bin_mcu_patch_code_ver(mcu_patch_code, ARRAY_SIZE(mcu_patch_code)); + + if (tp->hw_mcu_patch_code_ver != tp->bin_mcu_patch_code_ver) + rtl8127_write_mac_mcu_ram_code(tp, mcu_patch_code, ARRAY_SIZE(mcu_patch_code)); + + rtl8127_mac_ocp_write(tp, 0xFC26, 0x8000); + + rtl8127_mac_ocp_write(tp, 0xFC28, 0x1520); + rtl8127_mac_ocp_write(tp, 0xFC2A, 0x41E0); + rtl8127_mac_ocp_write(tp, 0xFC2C, 0x508C); + rtl8127_mac_ocp_write(tp, 0xFC2E, 0x50F6); + rtl8127_mac_ocp_write(tp, 0xFC30, 0x354E); + + rtl8127_mac_ocp_write(tp, 0xFC48, 0x001F); +} + +static void +rtl8127_set_mac_mcu_8127a_1(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u8 tmp = (u8)rtl8127_mac_ocp_read(tp, 0xD006); + + if (tmp != 0x04) + return; + + _rtl8127_set_mac_mcu_8127a_1(dev); +} + +static void +rtl8127_hw_mac_mcu_config(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (tp->NotWrMcuPatchCode == TRUE) + return; + + rtl8127_hw_disable_mac_mcu_bps(dev); + + /* Get H/W mac mcu patch code version */ + tp->hw_mcu_patch_code_ver = rtl8127_get_hw_mcu_patch_code_ver(tp); + + switch (tp->mcfg) { + case CFG_METHOD_1: + rtl8127_set_mac_mcu_8127a_tc(dev); + break; + case CFG_METHOD_2: + rtl8127_set_mac_mcu_8127a_1(dev); + break; + default: + break; + } +} +#endif + +#ifdef ENABLE_USE_FIRMWARE_FILE +static void rtl8127_release_firmware(struct rtl8127_private *tp) +{ + if (tp->rtl_fw) { + rtl8127_fw_release_firmware(tp->rtl_fw); + kfree(tp->rtl_fw); + tp->rtl_fw = NULL; + } +} + +static void rtl8127_apply_firmware(struct rtl8127_private *tp) +{ + /* TODO: release firmware if rtl_fw_write_firmware signals failure. */ + if (tp->rtl_fw) { + rtl8127_fw_write_firmware(tp, tp->rtl_fw); + /* At least one firmware doesn't reset tp->ocp_base. */ + tp->ocp_base = OCP_STD_PHY_BASE; + + /* PHY soft reset may still be in progress */ + //phy_read_poll_timeout(tp->phydev, MII_BMCR, val, + // !(val & BMCR_RESET), + // 50000, 600000, true); + rtl8127_wait_phy_reset_complete(tp); + + tp->hw_ram_code_ver = rtl8127_get_hw_phy_mcu_code_ver(tp); + tp->sw_ram_code_ver = tp->hw_ram_code_ver; + tp->HwHasWrRamCodeToMicroP = TRUE; + } +} +#endif + +static void +rtl8127_hw_init(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u32 csi_tmp; + + rtl8127_enable_aspm_clkreq_lock(tp, 0); + rtl8127_enable_force_clkreq(tp, 0); + + //Disable UPS + rtl8127_mac_ocp_write(tp, 0xD40A, rtl8127_mac_ocp_read(tp, 0xD40A) & ~(BIT_4)); + +#ifndef ENABLE_USE_FIRMWARE_FILE + if (!tp->rtl_fw) + rtl8127_hw_mac_mcu_config(dev); +#endif + + //Set PCIE uncorrectable error status mask pcie 0x108 + csi_tmp = rtl8127_csi_read(tp, 0x108); + csi_tmp |= BIT_20; + rtl8127_csi_write(tp, 0x108, csi_tmp); + + rtl8127_enable_cfg9346_write(tp); + rtl8127_disable_linkchg_wakeup(dev); + rtl8127_disable_cfg9346_write(tp); + rtl8127_disable_magic_packet(dev); + rtl8127_disable_d0_speedup(tp); + rtl8127_set_pci_pme(tp, 0); + if (s0_magic_packet == 1) + rtl8127_enable_magic_packet(dev); + +#ifdef ENABLE_USE_FIRMWARE_FILE + if (tp->rtl_fw && + !tp->resume_not_chg_speed && + !(HW_DASH_SUPPORT_TYPE_3(tp) && + tp->HwPkgDet == 0x06)) + rtl8127_apply_firmware(tp); +#endif +} + +static void +rtl8127_clear_ephy_ext_addr(struct rtl8127_private *tp) +{ + rtl8127_set_ephy_ext_addr(tp, 0x0000); +} + +static void +rtl8127_hw_ephy_config_8127_1(struct rtl8127_private *tp) +{ + rtl8127_ephy_write(tp, 0x8088, 0x0064); + rtl8127_ephy_write(tp, 0x8488, 0x0064); + rtl8127_ephy_write(tp, 0x8888, 0x0064); + rtl8127_ephy_write(tp, 0x8C88, 0x0064); + rtl8127_ephy_write(tp, 0x8188, 0x0064); + rtl8127_ephy_write(tp, 0x8588, 0x0064); + rtl8127_ephy_write(tp, 0x8988, 0x0064); + rtl8127_ephy_write(tp, 0x8D88, 0x0064); + rtl8127_ephy_write(tp, 0x808C, 0x09B0); + rtl8127_ephy_write(tp, 0x848C, 0x09B0); + rtl8127_ephy_write(tp, 0x888C, 0x0F90); + rtl8127_ephy_write(tp, 0x8C8C, 0x0F90); + rtl8127_ephy_write(tp, 0x818C, 0x09B0); + rtl8127_ephy_write(tp, 0x858C, 0x09B0); + rtl8127_ephy_write(tp, 0x898C, 0x0F90); + rtl8127_ephy_write(tp, 0x8D8C, 0x0F90); + rtl8127_ephy_write(tp, 0x808A, 0x09B8); + rtl8127_ephy_write(tp, 0x848A, 0x09B8); + rtl8127_ephy_write(tp, 0x888A, 0x0F98); + rtl8127_ephy_write(tp, 0x8C8A, 0x0F98); + rtl8127_ephy_write(tp, 0x818A, 0x09B8); + rtl8127_ephy_write(tp, 0x858A, 0x09B8); + rtl8127_ephy_write(tp, 0x898A, 0x0F98); + rtl8127_ephy_write(tp, 0x8D8A, 0x0F98); + rtl8127_ephy_write(tp, 0x9020, 0x0080); + rtl8127_ephy_write(tp, 0x9420, 0x0080); + rtl8127_ephy_write(tp, 0x9820, 0x0080); + rtl8127_ephy_write(tp, 0x9C20, 0x0080); + rtl8127_ephy_write(tp, 0x901E, 0x0190); + rtl8127_ephy_write(tp, 0x941E, 0x0190); + rtl8127_ephy_write(tp, 0x981E, 0x0140); + rtl8127_ephy_write(tp, 0x9C1E, 0x0140); + rtl8127_ephy_write(tp, 0x901C, 0x0190); + rtl8127_ephy_write(tp, 0x941C, 0x0190); + rtl8127_ephy_write(tp, 0x981C, 0x0140); + rtl8127_ephy_write(tp, 0x9C1C, 0x0140); + + /* Clear extended address */ + rtl8127_clear_ephy_ext_addr(tp); +} + +static void +rtl8127_hw_ephy_config(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->mcfg) { + case CFG_METHOD_2: + rtl8127_hw_ephy_config_8127_1(tp); + break; + default: + /* nothing to do */ + break; + } +} + +static u16 +rtl8127_get_hw_phy_mcu_code_ver(struct rtl8127_private *tp) +{ + u16 hw_ram_code_ver; + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x801E); + hw_ram_code_ver = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA438); + + return hw_ram_code_ver; +} + +static int +rtl8127_check_hw_phy_mcu_code_ver(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ram_code_ver_match = 0; + + tp->hw_ram_code_ver = rtl8127_get_hw_phy_mcu_code_ver(tp); + + if (tp->hw_ram_code_ver == tp->sw_ram_code_ver) { + ram_code_ver_match = 1; + tp->HwHasWrRamCodeToMicroP = TRUE; + } + + return ram_code_ver_match; +} + +bool +rtl8127_set_phy_mcu_patch_request(struct rtl8127_private *tp) +{ + u16 gphy_val; + u16 WaitCount; + bool bSuccess = TRUE; + + rtl8127_set_eth_phy_ocp_bit(tp, 0xB820, BIT_4); + + WaitCount = 0; + do { + gphy_val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xB800); + udelay(100); + WaitCount++; + } while (!(gphy_val & BIT_6) && (WaitCount < 1000)); + + if (!(gphy_val & BIT_6) && (WaitCount == 1000)) + bSuccess = FALSE; + + if (!bSuccess) + dprintk("rtl8127_set_phy_mcu_patch_request fail.\n"); + + return bSuccess; +} + +bool +rtl8127_clear_phy_mcu_patch_request(struct rtl8127_private *tp) +{ + u16 gphy_val; + u16 WaitCount; + bool bSuccess = TRUE; + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB820, BIT_4); + + WaitCount = 0; + do { + gphy_val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xB800); + udelay(100); + WaitCount++; + } while ((gphy_val & BIT_6) && (WaitCount < 1000)); + + if ((gphy_val & BIT_6) && (WaitCount == 1000)) + bSuccess = FALSE; + + if (!bSuccess) + dprintk("rtl8127_clear_phy_mcu_patch_request fail.\n"); + + return bSuccess; +} + +#ifndef ENABLE_USE_FIRMWARE_FILE +static void +rtl8127_write_hw_phy_mcu_code_ver(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x801E); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, tp->sw_ram_code_ver); + tp->hw_ram_code_ver = tp->sw_ram_code_ver; +} + +static void +rtl8127_set_phy_mcu_ram_code(struct net_device *dev, const u16 *ramcode, u16 codesize) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u16 i; + u16 addr; + u16 val; + + if (ramcode == NULL || codesize % 2) + goto out; + + for (i = 0; i < codesize; i += 2) { + addr = ramcode[i]; + val = ramcode[i + 1]; + if (addr == 0xFFFF && val == 0xFFFF) + break; + rtl8127_mdio_direct_write_phy_ocp(tp, addr, val); + } + +out: + return; +} + +static void +rtl8127_enable_phy_disable_mode(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppCheckPhyDisableModeVer) { + case 3: + RTL_W8(tp, 0xF2, RTL_R8(tp, 0xF2) | BIT_5); + break; + } + + dprintk("enable phy disable mode.\n"); +} + +static void +rtl8127_disable_phy_disable_mode(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + switch (tp->HwSuppCheckPhyDisableModeVer) { + case 3: + RTL_W8(tp, 0xF2, RTL_R8(tp, 0xF2) & ~BIT_5); + break; + } + + mdelay(1); + + dprintk("disable phy disable mode.\n"); +} + +static const u16 phy_mcu_ram_code_8127a_tc_1[] = { + 0xa436, 0x8023, 0xa438, 0x3200, 0xa436, 0xB82E, 0xa438, 0x0001, + 0xBF8E, 0x1410, 0xBF8E, 0x1410, 0xBF90, 0xC20E, 0xBF9E, 0xBFFC, + 0xBFAA, 0x1924, 0xBFB4, 0x0B1E, 0xBFB6, 0x3740, 0xBFB8, 0x460E, + 0xBFBE, 0x000D, 0xBF8A, 0x3FF7, 0xBF9A, 0x0007, 0xBF1E, 0x01FF, + 0xBF1E, 0x01FF, 0xBF1E, 0x01FF, 0xBF2E, 0x454D, 0xbc10, 0xD50C, + 0xbc10, 0x950C, 0xbc10, 0xD50C, 0xbc10, 0x550C, 0xbddE, 0xEF00, + 0xbd32, 0xF000, 0xbd2C, 0x0800, 0xbdc8, 0x04B0, 0xbdc8, 0x0CB0, + 0xBD92, 0x0003, 0xBD94, 0x8000, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0016, 0xBD94, 0x1000, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0016, 0xBD94, 0x1200, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0000, 0xBD94, 0x1F1F, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0001, 0xBD94, 0x1F00, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0003, 0xBD94, 0x0000, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0016, 0xBD94, 0x0000, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xBD92, 0x0000, 0xBD94, 0x0000, 0xBD96, 0x000f, 0xBD96, 0x0000, + 0xbdc8, 0x08B0, 0xbdc8, 0x00B0, 0xbd32, 0x0000, 0xbd2C, 0x0000, + 0xbc10, 0x750C, 0xbc10, 0x650C, 0xbc10, 0x750C, 0xbc10, 0x550C, + 0xb820, 0x0090, 0xa436, 0xA016, 0xa438, 0x0000, 0xa436, 0xA012, + 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, 0xa438, 0x8010, + 0xa438, 0x1800, 0xa438, 0x809b, 0xa438, 0x1800, 0xa438, 0x8145, + 0xa438, 0x1800, 0xa438, 0x8197, 0xa438, 0x1800, 0xa438, 0x81d4, + 0xa438, 0x1800, 0xa438, 0x8214, 0xa438, 0x1800, 0xa438, 0x8226, + 0xa438, 0x1800, 0xa438, 0x8232, 0xa438, 0xd707, 0xa438, 0x4141, + 0xa438, 0xd70a, 0xa438, 0x4115, 0xa438, 0xd705, 0xa438, 0x40da, + 0xa438, 0xb808, 0xa438, 0xd028, 0xa438, 0xd1c1, 0xa438, 0x1800, + 0xa438, 0x801e, 0xa438, 0x9808, 0xa438, 0xd07b, 0xa438, 0xd1c5, + 0xa438, 0xbe10, 0xa438, 0xd503, 0xa438, 0xa108, 0xa438, 0xd505, + 0xa438, 0x8103, 0xa438, 0xd504, 0xa438, 0xa002, 0xa438, 0xa302, + 0xa438, 0xd707, 0xa438, 0x4061, 0xa438, 0xd503, 0xa438, 0x8b01, + 0xa438, 0xd500, 0xa438, 0xc48a, 0xa438, 0xd503, 0xa438, 0xcc09, + 0xa438, 0xcd58, 0xa438, 0xaf01, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x1764, 0xa438, 0xd719, 0xa438, 0x606c, 0xa438, 0xd704, + 0xa438, 0x645c, 0xa438, 0xd75e, 0xa438, 0x604d, 0xa438, 0xfff8, + 0xa438, 0x9e10, 0xa438, 0x1000, 0xa438, 0x1764, 0xa438, 0xd719, + 0xa438, 0x606c, 0xa438, 0xd704, 0xa438, 0x631c, 0xa438, 0xd75e, + 0xa438, 0x404d, 0xa438, 0xfff8, 0xa438, 0xd504, 0xa438, 0xaa18, + 0xa438, 0xa001, 0xa438, 0xa1e0, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x1764, 0xa438, 0xd719, 0xa438, 0x7fac, 0xa438, 0xd504, + 0xa438, 0xa001, 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x1764, + 0xa438, 0xd704, 0xa438, 0x5f5c, 0xa438, 0xd719, 0xa438, 0x3aaf, + 0xa438, 0x8058, 0xa438, 0xf016, 0xa438, 0xd707, 0xa438, 0x6121, + 0xa438, 0x1000, 0xa438, 0x16f8, 0xa438, 0xd503, 0xa438, 0xcd59, + 0xa438, 0xaf01, 0xa438, 0xd500, 0xa438, 0x1800, 0xa438, 0x0e49, + 0xa438, 0xd503, 0xa438, 0x8040, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x16f8, 0xa438, 0xd503, 0xa438, 0xcd5a, 0xa438, 0xaf01, + 0xa438, 0xd500, 0xa438, 0x1800, 0xa438, 0x0e2f, 0xa438, 0xd504, + 0xa438, 0xa008, 0xa438, 0xa204, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x1764, 0xa438, 0xd701, 0xa438, 0x5fa0, 0xa438, 0xd503, + 0xa438, 0xa082, 0xa438, 0xd500, 0xa438, 0xd71e, 0xa438, 0x4097, + 0xa438, 0xd078, 0xa438, 0xd1aa, 0xa438, 0xf003, 0xa438, 0xd078, + 0xa438, 0xd1aa, 0xa438, 0xd707, 0xa438, 0x4081, 0xa438, 0xd70a, + 0xa438, 0x4055, 0xa438, 0xf014, 0xa438, 0xd706, 0xa438, 0x6065, + 0xa438, 0xcc89, 0xa438, 0xf002, 0xa438, 0xcc8b, 0xa438, 0x1000, + 0xa438, 0x0bb2, 0xa438, 0xd705, 0xa438, 0x2ad0, 0xa438, 0x808f, + 0xa438, 0xf003, 0xa438, 0x1000, 0xa438, 0x0bb8, 0xa438, 0x1000, + 0xa438, 0x0bbe, 0xa438, 0x607a, 0xa438, 0x9c01, 0xa438, 0xf002, + 0xa438, 0xbc01, 0xa438, 0x1000, 0xa438, 0x0cc9, 0xa438, 0x1800, + 0xa438, 0x132f, 0xa438, 0x9a10, 0xa438, 0x9d02, 0xa438, 0xd706, + 0xa438, 0x629a, 0xa438, 0x61bb, 0xa438, 0xd707, 0xa438, 0x60d7, + 0xa438, 0xd70a, 0xa438, 0x6196, 0xa438, 0xd0e5, 0xa438, 0xd1e7, + 0xa438, 0xf012, 0xa438, 0xd70a, 0xa438, 0x5f55, 0xa438, 0xd060, + 0xa438, 0xd1e8, 0xa438, 0xf00d, 0xa438, 0xd056, 0xa438, 0xd1e8, + 0xa438, 0xf00a, 0xa438, 0xd043, 0xa438, 0xd1e8, 0xa438, 0xf007, + 0xa438, 0x609b, 0xa438, 0xd078, 0xa438, 0xd1e9, 0xa438, 0xf003, + 0xa438, 0xd07f, 0xa438, 0xd1e9, 0xa438, 0xd503, 0xa438, 0xab01, + 0xa438, 0xd500, 0xa438, 0xd706, 0xa438, 0x6139, 0xa438, 0xd503, + 0xa438, 0x6065, 0xa438, 0xa7f0, 0xa438, 0xf003, 0xa438, 0x0cf0, + 0xa438, 0x0750, 0xa438, 0x8908, 0xa438, 0xf004, 0xa438, 0xd503, + 0xa438, 0xa908, 0xa438, 0x87f0, 0xa438, 0xd503, 0xa438, 0xa040, + 0xa438, 0xd500, 0xa438, 0xd705, 0xa438, 0x407b, 0xa438, 0x1000, + 0xa438, 0x1ac8, 0xa438, 0xd503, 0xa438, 0x0c07, 0xa438, 0x0902, + 0xa438, 0xa008, 0xa438, 0xd500, 0xa438, 0x9d80, 0xa438, 0xc48c, + 0xa438, 0xd73e, 0xa438, 0x6000, 0xa438, 0xd706, 0xa438, 0x419b, + 0xa438, 0xd503, 0xa438, 0x0c87, 0xa438, 0x0981, 0xa438, 0xd500, + 0xa438, 0xd073, 0xa438, 0xd1b7, 0xa438, 0xc490, 0xa438, 0x1000, + 0xa438, 0x1764, 0xa438, 0xd704, 0xa438, 0x5fbb, 0xa438, 0xd503, + 0xa438, 0xd706, 0xa438, 0x607a, 0xa438, 0x617c, 0xa438, 0xf013, + 0xa438, 0xd70b, 0xa438, 0x40b4, 0xa438, 0xd706, 0xa438, 0x6309, + 0xa438, 0x6348, 0xa438, 0xf025, 0xa438, 0xd706, 0xa438, 0x6408, + 0xa438, 0xf022, 0xa438, 0xd70b, 0xa438, 0x40b4, 0xa438, 0xd706, + 0xa438, 0x6429, 0xa438, 0x6468, 0xa438, 0xf028, 0xa438, 0xd706, + 0xa438, 0x6468, 0xa438, 0xf025, 0xa438, 0xd70b, 0xa438, 0x67d4, + 0xa438, 0xd706, 0xa438, 0x6488, 0xa438, 0x6589, 0xa438, 0x2c69, + 0xa438, 0x8135, 0xa438, 0x66ab, 0xa438, 0xf037, 0xa438, 0xc320, + 0xa438, 0xc420, 0xa438, 0xf03c, 0xa438, 0xd707, 0xa438, 0x6077, + 0xa438, 0xc310, 0xa438, 0xf004, 0xa438, 0xd70a, 0xa438, 0x5fb5, + 0xa438, 0xc340, 0xa438, 0xc410, 0xa438, 0xf033, 0xa438, 0xc340, + 0xa438, 0xc420, 0xa438, 0xf030, 0xa438, 0xc340, 0xa438, 0xc440, + 0xa438, 0xf02d, 0xa438, 0xc304, 0xa438, 0xc408, 0xa438, 0xf02a, + 0xa438, 0xc302, 0xa438, 0xc404, 0xa438, 0xf027, 0xa438, 0xc308, + 0xa438, 0xc408, 0xa438, 0xf024, 0xa438, 0xc308, 0xa438, 0xc410, + 0xa438, 0xf021, 0xa438, 0xd707, 0xa438, 0x6077, 0xa438, 0xc304, + 0xa438, 0xf004, 0xa438, 0xd70a, 0xa438, 0x5fb5, 0xa438, 0xc308, + 0xa438, 0xc408, 0xa438, 0xf018, 0xa438, 0xd707, 0xa438, 0x6077, + 0xa438, 0xc308, 0xa438, 0xf004, 0xa438, 0xd70a, 0xa438, 0x5fb5, + 0xa438, 0xc310, 0xa438, 0xc410, 0xa438, 0xf00f, 0xa438, 0xc304, + 0xa438, 0xc408, 0xa438, 0xf00c, 0xa438, 0xc302, 0xa438, 0xc404, + 0xa438, 0xf009, 0xa438, 0xd707, 0xa438, 0x6077, 0xa438, 0xc308, + 0xa438, 0xf004, 0xa438, 0xd70a, 0xa438, 0x5fb5, 0xa438, 0xc310, + 0xa438, 0xc410, 0xa438, 0x1800, 0xa438, 0x14e7, 0xa438, 0xd07b, + 0xa438, 0xd1c5, 0xa438, 0xd503, 0xa438, 0xa028, 0xa438, 0x8970, + 0xa438, 0x880f, 0xa438, 0x0c0f, 0xa438, 0x0909, 0xa438, 0x87f0, + 0xa438, 0xc600, 0xa438, 0xa521, 0xa438, 0xd501, 0xa438, 0xce01, + 0xa438, 0xa202, 0xa438, 0xa201, 0xa438, 0x8201, 0xa438, 0xce00, + 0xa438, 0xd500, 0xa438, 0xd706, 0xa438, 0x4425, 0xa438, 0xd503, + 0xa438, 0xab80, 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x0c1d, + 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0xaac0, 0xa438, 0xd00d, + 0xa438, 0xd1a2, 0xa438, 0xd700, 0xa438, 0x401a, 0xa438, 0xa60c, + 0xa438, 0xd010, 0xa438, 0xd1a2, 0xa438, 0xd700, 0xa438, 0x401a, + 0xa438, 0xa70c, 0xa438, 0xd09e, 0xa438, 0xd1a2, 0xa438, 0xd700, + 0xa438, 0x401a, 0xa438, 0xce00, 0xa438, 0xd500, 0xa438, 0xd505, + 0xa438, 0xab01, 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x0c30, + 0xa438, 0xd505, 0xa438, 0x8b01, 0xa438, 0xd500, 0xa438, 0xbc02, + 0xa438, 0xc482, 0xa438, 0xd503, 0xa438, 0xcc01, 0xa438, 0xcd0c, + 0xa438, 0xaf01, 0xa438, 0xd500, 0xa438, 0xd75e, 0xa438, 0x4000, + 0xa438, 0xd706, 0xa438, 0x4245, 0xa438, 0x1000, 0xa438, 0x0c1d, + 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0x8ac0, 0xa438, 0x860c, + 0xa438, 0x870c, 0xa438, 0xce00, 0xa438, 0xd500, 0xa438, 0xd505, + 0xa438, 0xab01, 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x0c30, + 0xa438, 0xd505, 0xa438, 0x8b01, 0xa438, 0xd500, 0xa438, 0x1800, + 0xa438, 0x0e74, 0xa438, 0xd707, 0xa438, 0x43cf, 0xa438, 0x1000, + 0xa438, 0x0c1d, 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0xaac0, + 0xa438, 0xd00d, 0xa438, 0xd1a2, 0xa438, 0xd700, 0xa438, 0x401a, + 0xa438, 0xa60c, 0xa438, 0xd010, 0xa438, 0xd1a2, 0xa438, 0xd700, + 0xa438, 0x401a, 0xa438, 0xa70c, 0xa438, 0xd09e, 0xa438, 0xd1a2, + 0xa438, 0xd700, 0xa438, 0x401a, 0xa438, 0xce00, 0xa438, 0xd500, + 0xa438, 0xd505, 0xa438, 0xab01, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x0c30, 0xa438, 0xd505, 0xa438, 0x8b01, 0xa438, 0xd500, + 0xa438, 0xd501, 0xa438, 0xce00, 0xa438, 0xab10, 0xa438, 0xbb10, + 0xa438, 0x9c02, 0xa438, 0xd75e, 0xa438, 0x6000, 0xa438, 0xd707, + 0xa438, 0x424f, 0xa438, 0x1000, 0xa438, 0x0c1d, 0xa438, 0xd501, + 0xa438, 0xce01, 0xa438, 0x8ac0, 0xa438, 0x860c, 0xa438, 0x870c, + 0xa438, 0xce00, 0xa438, 0xd500, 0xa438, 0xd505, 0xa438, 0xab01, + 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x0c30, 0xa438, 0xd505, + 0xa438, 0x8b01, 0xa438, 0xd500, 0xa438, 0x9808, 0xa438, 0x9c01, + 0xa438, 0x1800, 0xa438, 0x172a, 0xa438, 0xd707, 0xa438, 0x43cf, + 0xa438, 0x1000, 0xa438, 0x0c1d, 0xa438, 0xd501, 0xa438, 0xce01, + 0xa438, 0xaac0, 0xa438, 0xd00d, 0xa438, 0xd1a2, 0xa438, 0xd700, + 0xa438, 0x401a, 0xa438, 0xa60c, 0xa438, 0xd010, 0xa438, 0xd1a2, + 0xa438, 0xd700, 0xa438, 0x401a, 0xa438, 0xa70c, 0xa438, 0xd09e, + 0xa438, 0xd1a2, 0xa438, 0xd700, 0xa438, 0x401a, 0xa438, 0xce00, + 0xa438, 0xd500, 0xa438, 0xd505, 0xa438, 0xab01, 0xa438, 0xd500, + 0xa438, 0x1000, 0xa438, 0x0c30, 0xa438, 0xd505, 0xa438, 0x8b01, + 0xa438, 0xd500, 0xa438, 0x1000, 0xa438, 0x1a6b, 0xa438, 0x9808, + 0xa438, 0x9c01, 0xa438, 0xd707, 0xa438, 0x424f, 0xa438, 0x1000, + 0xa438, 0x0c1d, 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0x8ac0, + 0xa438, 0x860c, 0xa438, 0x870c, 0xa438, 0xce00, 0xa438, 0xd500, + 0xa438, 0xd505, 0xa438, 0xab01, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x0c30, 0xa438, 0xd505, 0xa438, 0x8b01, 0xa438, 0xd500, + 0xa438, 0xd503, 0xa438, 0x8b80, 0xa438, 0xd500, 0xa438, 0x1000, + 0xa438, 0x1aa5, 0xa438, 0xd503, 0xa438, 0xcda3, 0xa438, 0xaf01, + 0xa438, 0x1800, 0xa438, 0x181c, 0xa438, 0xd700, 0xa438, 0x41f6, + 0xa438, 0xd703, 0xa438, 0x41a3, 0xa438, 0x1000, 0xa438, 0x0c1d, + 0xa438, 0xd501, 0xa438, 0xa580, 0xa438, 0xa701, 0xa438, 0xd500, + 0xa438, 0xd014, 0xa438, 0xd1c3, 0xa438, 0xd703, 0xa438, 0x401c, + 0xa438, 0x1000, 0xa438, 0x0c30, 0xa438, 0x1800, 0xa438, 0x069b, + 0xa438, 0xba10, 0xa438, 0xd70c, 0xa438, 0x4107, 0xa438, 0xd702, + 0xa438, 0x40d0, 0xa438, 0xd504, 0xa438, 0xa110, 0xa438, 0xd500, + 0xa438, 0x1800, 0xa438, 0x1447, 0xa438, 0x1800, 0xa438, 0x1420, + 0xa438, 0xd70c, 0xa438, 0x60a6, 0xa438, 0xd501, 0xa438, 0xce01, + 0xa438, 0x840f, 0xa438, 0xce00, 0xa438, 0xd503, 0xa438, 0x8008, + 0xa438, 0xd500, 0xa438, 0x1800, 0xa438, 0x1360, 0xa436, 0xA026, + 0xa438, 0x1359, 0xa436, 0xA024, 0xa438, 0x141f, 0xa436, 0xA022, + 0xa438, 0x068f, 0xa436, 0xA020, 0xa438, 0x1815, 0xa436, 0xA006, + 0xa438, 0x1723, 0xa436, 0xA004, 0xa438, 0x0e59, 0xa436, 0xA002, + 0xa438, 0x1452, 0xa436, 0xA000, 0xa438, 0x12bb, 0xa436, 0xA008, + 0xa438, 0xff00, 0xa436, 0xA016, 0xa438, 0x0010, 0xa436, 0xA012, + 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, 0xa438, 0x8010, + 0xa438, 0x1800, 0xa438, 0x8010, 0xa438, 0x1800, 0xa438, 0x8019, + 0xa438, 0x1800, 0xa438, 0x8019, 0xa438, 0x1800, 0xa438, 0x8019, + 0xa438, 0x1800, 0xa438, 0x8019, 0xa438, 0x1800, 0xa438, 0x8019, + 0xa438, 0x1800, 0xa438, 0x8019, 0xa438, 0xd700, 0xa438, 0x2a59, + 0xa438, 0x0101, 0xa438, 0x2841, 0xa438, 0x0122, 0xa438, 0x2d69, + 0xa438, 0x00e6, 0xa438, 0x1800, 0xa438, 0x00e0, 0xa436, 0xA08E, + 0xa438, 0x0000, 0xa436, 0xA08C, 0xa438, 0x0000, 0xa436, 0xA08A, + 0xa438, 0x0000, 0xa436, 0xA088, 0xa438, 0x0000, 0xa436, 0xA086, + 0xa438, 0x0000, 0xa436, 0xA084, 0xa438, 0x0000, 0xa436, 0xA082, + 0xa438, 0x00e2, 0xa436, 0xA080, 0xa438, 0x0000, 0xa436, 0xA090, + 0xa438, 0x0002, 0xa436, 0xA016, 0xa438, 0x0020, 0xa436, 0xA012, + 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, 0xa438, 0x8010, + 0xa438, 0x1800, 0xa438, 0x9dfb, 0xa438, 0x1800, 0xa438, 0x9e02, + 0xa438, 0x1800, 0xa438, 0x9e02, 0xa438, 0x1800, 0xa438, 0x9e02, + 0xa438, 0x1800, 0xa438, 0x9e02, 0xa438, 0x1800, 0xa438, 0x9e02, + 0xa438, 0x1800, 0xa438, 0x9e02, 0xa438, 0xd71f, 0xa438, 0x626d, + 0xa438, 0xd71e, 0xa438, 0x4103, 0xa438, 0xa70c, 0xa438, 0xb801, + 0xa438, 0xba04, 0xa438, 0xd71f, 0xa438, 0x6001, 0xa438, 0x1800, + 0xa438, 0x001c, 0xa438, 0xb801, 0xa438, 0xd704, 0xa438, 0x2211, + 0xa438, 0x9d54, 0xa438, 0xd700, 0xa438, 0x2739, 0xa438, 0x837a, + 0xa438, 0x1800, 0xa438, 0x814b, 0xa438, 0xcd70, 0xa438, 0xb801, + 0xa438, 0xa708, 0xa438, 0xd700, 0xa438, 0x60d7, 0xa438, 0x6073, + 0xa438, 0xce02, 0xa438, 0xf004, 0xa438, 0xce01, 0xa438, 0xf002, + 0xa438, 0xce01, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0ccf, 0xa438, 0x0b04, 0xa438, 0xcc22, + 0xa438, 0xcd01, 0xa438, 0xa702, 0xa438, 0x9503, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0x1000, 0xa438, 0x9c02, 0xa438, 0xa610, + 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xcd71, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0xd707, 0xa438, 0x61cf, 0xa438, 0xa502, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd702, 0xa438, 0x5fbe, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, + 0xa438, 0xd403, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x8502, + 0xa438, 0xa340, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0xa440, 0xa438, 0xa8c0, + 0xa438, 0xa2fc, 0xa438, 0xa120, 0xa438, 0x0ca0, 0xa438, 0x0480, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa370, 0xa438, 0x9503, + 0xa438, 0xcd72, 0xa438, 0xd1f5, 0xa438, 0xd057, 0xa438, 0xd1c4, + 0xa438, 0xd066, 0xa438, 0xd1c4, 0xa438, 0xd077, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x607c, 0xa438, 0x613d, + 0xa438, 0xfffb, 0xa438, 0xa310, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fbd, 0xa438, 0xa607, 0xa438, 0xf007, + 0xa438, 0xa607, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fbc, 0xa438, 0xa310, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fbb, 0xa438, 0x8840, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0cf8, 0xa438, 0x0d48, 0xa438, 0x8320, + 0xa438, 0xa180, 0xa438, 0x9503, 0xa438, 0xd1c4, 0xa438, 0xd055, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fbb, + 0xa438, 0xd706, 0xa438, 0x6227, 0xa438, 0xd700, 0xa438, 0x5f3a, + 0xa438, 0x88c0, 0xa438, 0x82fc, 0xa438, 0x8120, 0xa438, 0x8350, + 0xa438, 0x84a0, 0xa438, 0x8607, 0xa438, 0xa510, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8df8, 0xa438, 0x8370, 0xa438, 0x8180, + 0xa438, 0x9503, 0xa438, 0xff96, 0xa438, 0x8510, 0xa438, 0xa508, + 0xa438, 0x8508, 0xa438, 0xcd73, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x9920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x6065, + 0xa438, 0x5f94, 0xa438, 0xffdf, 0xa438, 0xb820, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, + 0xa438, 0x8120, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0x8880, 0xa438, 0x8480, + 0xa438, 0x8606, 0xa438, 0xcd74, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8b0f, 0xa438, 0x8c3f, 0xa438, 0x9503, 0xa438, 0xa810, + 0xa438, 0xa120, 0xa438, 0xa310, 0xa438, 0xa4a0, 0xa438, 0xa606, + 0xa438, 0xd700, 0xa438, 0x37cd, 0xa438, 0x80e5, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8310, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x80f0, 0xa438, 0x33a9, 0xa438, 0x80ed, + 0xa438, 0xd17a, 0xa438, 0xd04c, 0xa438, 0xf006, 0xa438, 0xd199, + 0xa438, 0xd04c, 0xa438, 0xf003, 0xa438, 0xd1d6, 0xa438, 0xd04c, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd706, 0xa438, 0x5f69, 0xa438, 0xd700, 0xa438, 0x60d7, + 0xa438, 0x6073, 0xa438, 0xce05, 0xa438, 0xf004, 0xa438, 0xce04, + 0xa438, 0xf002, 0xa438, 0xce04, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, 0xa438, 0x0b02, + 0xa438, 0x0c38, 0xa438, 0x0c10, 0xa438, 0x9503, 0xa438, 0xa180, + 0xa438, 0xa680, 0xa438, 0xcd75, 0xa438, 0xd199, 0xa438, 0xd04b, + 0xa438, 0xd13b, 0xa438, 0xd055, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fbb, 0xa438, 0xa302, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, + 0xa438, 0x5f6a, 0xa438, 0xbb50, 0xa438, 0xcd76, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb5, 0xa438, 0x9b10, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb5, + 0xa438, 0xa120, 0xa438, 0x0c12, 0xa438, 0x0310, 0xa438, 0x8480, + 0xa438, 0x0c84, 0xa438, 0x0604, 0xa438, 0xcd77, 0xa438, 0xd1a0, + 0xa438, 0xd04b, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x422a, 0xa438, 0xa1a0, + 0xa438, 0xa312, 0xa438, 0xa480, 0xa438, 0xa684, 0xa438, 0xcd78, + 0xa438, 0xd148, 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xcd79, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x7f8a, 0xa438, 0x9b40, + 0xa438, 0xcd7f, 0xa438, 0xd71f, 0xa438, 0x7fe1, 0xa438, 0x1800, + 0xa438, 0x001c, 0xa438, 0xa708, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce02, 0xa438, 0xf00a, 0xa438, 0xce02, + 0xa438, 0xf008, 0xa438, 0xce01, 0xa438, 0xf006, 0xa438, 0xce01, + 0xa438, 0xf004, 0xa438, 0xce01, 0xa438, 0xf002, 0xa438, 0xce01, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0ccf, 0xa438, 0x0b04, 0xa438, 0xcc22, 0xa438, 0xcd01, + 0xa438, 0xa702, 0xa438, 0x9503, 0xa438, 0xcd15, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0xba20, 0xa438, 0xd701, 0xa438, 0x4154, + 0xa438, 0xd115, 0xa438, 0xd04f, 0xa438, 0x1000, 0xa438, 0x9c02, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xf003, 0xa438, 0x1000, 0xa438, 0x9c02, 0xa438, 0xa610, + 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xcd16, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0xd707, 0xa438, 0x61cf, 0xa438, 0xa502, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd702, 0xa438, 0x5fbe, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, + 0xa438, 0xd403, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x8502, + 0xa438, 0xa340, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0xa440, 0xa438, 0xa8c0, + 0xa438, 0xd705, 0xa438, 0x40d3, 0xa438, 0xd707, 0xa438, 0x4082, + 0xa438, 0x0cfc, 0xa438, 0x02bc, 0xa438, 0xf002, 0xa438, 0xa2fc, + 0xa438, 0xa120, 0xa438, 0x0ca0, 0xa438, 0x0480, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa370, 0xa438, 0x9503, 0xa438, 0xcd17, + 0xa438, 0xd1f5, 0xa438, 0xd057, 0xa438, 0xd1c4, 0xa438, 0xd066, + 0xa438, 0xd1c4, 0xa438, 0xd077, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x607c, 0xa438, 0x613d, 0xa438, 0xfffb, + 0xa438, 0xa310, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fbd, 0xa438, 0xa607, 0xa438, 0xf007, 0xa438, 0xa607, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fbc, + 0xa438, 0xa310, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fbb, 0xa438, 0x8840, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0cf8, 0xa438, 0x0d48, 0xa438, 0x8320, 0xa438, 0xa180, + 0xa438, 0x9503, 0xa438, 0xd1c4, 0xa438, 0xd055, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fbb, 0xa438, 0xd706, + 0xa438, 0x6227, 0xa438, 0xd700, 0xa438, 0x5f3a, 0xa438, 0x88c0, + 0xa438, 0x82fc, 0xa438, 0x8120, 0xa438, 0x8350, 0xa438, 0x84a0, + 0xa438, 0x8607, 0xa438, 0xa510, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8df8, 0xa438, 0x8370, 0xa438, 0x8180, 0xa438, 0x9503, + 0xa438, 0xff8f, 0xa438, 0x8510, 0xa438, 0xa508, 0xa438, 0x8508, + 0xa438, 0xcd18, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5fb4, 0xa438, 0xb920, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x9920, 0xa438, 0x9a20, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x6065, + 0xa438, 0x5f94, 0xa438, 0xffde, 0xa438, 0xb820, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, + 0xa438, 0x8120, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0x8880, 0xa438, 0x8480, + 0xa438, 0x8606, 0xa438, 0xcd19, 0xa438, 0xd705, 0xa438, 0x40d3, + 0xa438, 0xd707, 0xa438, 0x4082, 0xa438, 0x8310, 0xa438, 0x1000, + 0xa438, 0x9c0c, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8b0f, + 0xa438, 0x8c3f, 0xa438, 0x9503, 0xa438, 0xa810, 0xa438, 0xa120, + 0xa438, 0xd705, 0xa438, 0x40b3, 0xa438, 0xd707, 0xa438, 0x4062, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa4a0, + 0xa438, 0xa606, 0xa438, 0xd700, 0xa438, 0x37cd, 0xa438, 0x8236, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8310, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8241, 0xa438, 0x33a9, + 0xa438, 0x823e, 0xa438, 0xd17a, 0xa438, 0xd04c, 0xa438, 0xf006, + 0xa438, 0xd199, 0xa438, 0xd04c, 0xa438, 0xf003, 0xa438, 0xd1d6, + 0xa438, 0xd04c, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, + 0xa438, 0x5f29, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce05, 0xa438, 0xf00a, 0xa438, 0xce05, 0xa438, 0xf008, + 0xa438, 0xce04, 0xa438, 0xf006, 0xa438, 0xce04, 0xa438, 0xf004, + 0xa438, 0xce04, 0xa438, 0xf002, 0xa438, 0xce04, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b02, 0xa438, 0x0c38, 0xa438, 0x0c10, 0xa438, 0x9503, + 0xa438, 0xa180, 0xa438, 0xa680, 0xa438, 0xcd1a, 0xa438, 0xd199, + 0xa438, 0xd04b, 0xa438, 0xd13b, 0xa438, 0xd055, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7b, 0xa438, 0xa302, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2a, 0xa438, 0xbb10, 0xa438, 0xcd1b, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd71f, 0xa438, 0x5f75, 0xa438, 0xa704, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce03, 0xa438, 0xf00a, + 0xa438, 0xce03, 0xa438, 0xf008, 0xa438, 0xce03, 0xa438, 0xf006, + 0xa438, 0xce03, 0xa438, 0xf004, 0xa438, 0xce03, 0xa438, 0xf002, + 0xa438, 0xce03, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa00a, + 0xa438, 0x81a0, 0xa438, 0x8312, 0xa438, 0x8480, 0xa438, 0xa686, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, + 0xa438, 0x9b10, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb5, 0xa438, 0xcd2a, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fa4, 0xa438, 0xd17b, 0xa438, 0xd04a, + 0xa438, 0xa980, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f68, 0xa438, 0x800a, + 0xa438, 0x8604, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8801, + 0xa438, 0x9503, 0xa438, 0xd40c, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, + 0xa438, 0xd417, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xa00a, + 0xa438, 0xa604, 0xa438, 0xd700, 0xa438, 0x39b9, 0xa438, 0x82e4, + 0xa438, 0xd707, 0xa438, 0x432f, 0xa438, 0xd700, 0xa438, 0x608f, + 0xa438, 0x60b1, 0xa438, 0x60d3, 0xa438, 0x60f5, 0xa438, 0xce08, + 0xa438, 0xf007, 0xa438, 0xce08, 0xa438, 0xf005, 0xa438, 0xce08, + 0xa438, 0xf003, 0xa438, 0xce08, 0xa438, 0xf001, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa1a0, 0xa438, 0xa302, 0xa438, 0xa480, + 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4062, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xd17b, 0xa438, 0xd049, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd706, 0xa438, 0x5f68, 0xa438, 0xd700, 0xa438, 0x37cd, + 0xa438, 0x8302, 0xa438, 0x800a, 0xa438, 0x8604, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd417, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xa00a, 0xa438, 0xa604, + 0xa438, 0xd17b, 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f68, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, + 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, + 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, + 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xa1a0, 0xa438, 0xd707, 0xa438, 0x40df, 0xa438, 0x40a2, + 0xa438, 0x0cfc, 0xa438, 0x02bc, 0xa438, 0x8310, 0xa438, 0xf004, + 0xa438, 0x0cfc, 0xa438, 0x02fc, 0xa438, 0xa310, 0xa438, 0xa302, + 0xa438, 0xa480, 0xa438, 0xa686, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa340, 0xa438, 0x9503, 0xa438, 0xcd2b, 0xa438, 0xd199, + 0xa438, 0xd04a, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f68, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, + 0xa438, 0xcd2c, 0xa438, 0xd199, 0xa438, 0xd04b, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, + 0xa438, 0x5f6a, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8312, + 0xa438, 0x8604, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, + 0xa438, 0x9503, 0xa438, 0xd417, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xd707, 0xa438, 0x409f, + 0xa438, 0x4062, 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, + 0xa438, 0xa302, 0xa438, 0xa604, 0xa438, 0xd409, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5fb4, 0xa438, 0xb920, 0xa438, 0xcd2d, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x9920, + 0xa438, 0xbb10, 0xa438, 0xcd2e, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fb5, 0xa438, 0x800a, 0xa438, 0x81a0, + 0xa438, 0x8312, 0xa438, 0x8480, 0xa438, 0x0c86, 0xa438, 0x0680, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, 0xa438, 0xa140, + 0xa438, 0x9503, 0xa438, 0x9b10, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb5, 0xa438, 0x1800, 0xa438, 0x8740, + 0xa438, 0xa70c, 0xa438, 0x8510, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce03, 0xa438, 0xf00a, 0xa438, 0xce03, + 0xa438, 0xf008, 0xa438, 0xce03, 0xa438, 0xf006, 0xa438, 0xce03, + 0xa438, 0xf004, 0xa438, 0xce03, 0xa438, 0xf002, 0xa438, 0xce03, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0ccf, 0xa438, 0x0b04, 0xa438, 0xcc21, 0xa438, 0xcd10, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x406d, 0xa438, 0x0cc0, + 0xa438, 0x0080, 0xa438, 0xd198, 0xa438, 0xd07f, 0xa438, 0xcd11, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fa4, + 0xa438, 0xd102, 0xa438, 0xd040, 0xa438, 0xcd12, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x1000, + 0xa438, 0x9c02, 0xa438, 0xa340, 0xa438, 0x1000, 0xa438, 0x9c52, + 0xa438, 0xa110, 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8110, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0xa224, + 0xa438, 0xa00a, 0xa438, 0xa802, 0xa438, 0xa980, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa370, 0xa438, 0x9503, 0xa438, 0xcd13, + 0xa438, 0xd700, 0xa438, 0x2469, 0xa438, 0x83db, 0xa438, 0x634b, + 0xa438, 0x39b9, 0xa438, 0x83c5, 0xa438, 0xf017, 0xa438, 0xd17a, + 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f64, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, + 0xa438, 0xcd14, 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, + 0xa438, 0x5f65, 0xa438, 0x800a, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8340, 0xa438, 0xd700, 0xa438, 0x37cd, 0xa438, 0x83e3, + 0xa438, 0xa180, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x2469, + 0xa438, 0x8409, 0xa438, 0xd700, 0xa438, 0x65eb, 0xa438, 0xd700, + 0xa438, 0x6609, 0xa438, 0xd705, 0xa438, 0x6051, 0xa438, 0xf009, + 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa00a, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd705, 0xa438, 0x7fb1, + 0xa438, 0xd700, 0xa438, 0x39b9, 0xa438, 0x83fa, 0xa438, 0xf01e, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa801, 0xa438, 0x9503, + 0xa438, 0xd40c, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd417, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xf00f, 0xa438, 0xd700, + 0xa438, 0x39b9, 0xa438, 0x840d, 0xa438, 0xf00e, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa801, 0xa438, 0x9503, 0xa438, 0xd40c, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd700, 0xa438, 0x6044, + 0xa438, 0xf004, 0xa438, 0xd407, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xa508, 0xa438, 0x8508, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8d07, 0xa438, 0x9503, 0xa438, 0xa00a, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, 0xa438, 0xd704, + 0xa438, 0x6091, 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf003, + 0xa438, 0xd1c4, 0xa438, 0xd045, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x6a8b, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f24, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, 0xa438, 0xd704, + 0xa438, 0x6091, 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf003, + 0xa438, 0xd1c4, 0xa438, 0xd045, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f65, + 0xa438, 0x800a, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, + 0xa438, 0xa180, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x2d49, + 0xa438, 0x848d, 0xa438, 0x37c9, 0xa438, 0x8459, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd417, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xf023, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd417, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8d07, 0xa438, 0x9503, 0xa438, 0xa00a, 0xa438, 0xd17a, + 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f64, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, + 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x5f65, + 0xa438, 0x800a, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, + 0xa438, 0x9503, 0xa438, 0xd417, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xf00b, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5fab, 0xa438, 0xba08, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7f8b, 0xa438, 0x9a08, 0xa438, 0xa00a, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, + 0xa438, 0x800a, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, + 0xa438, 0xa180, 0xa438, 0x9503, 0xa438, 0xb920, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x9920, + 0xa438, 0xd700, 0xa438, 0x296d, 0xa438, 0x84bc, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0xa004, 0xa438, 0x1000, 0xa438, 0x9be6, + 0xa438, 0x8004, 0xa438, 0xa001, 0xa438, 0x1000, 0xa438, 0x9be6, + 0xa438, 0x8001, 0xa438, 0xa020, 0xa438, 0x1000, 0xa438, 0x9be6, + 0xa438, 0x8020, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xd120, + 0xa438, 0xd040, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x8704, 0xa438, 0xcd21, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fad, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0xba20, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x84d5, 0xa438, 0xd700, 0xa438, 0x612b, 0xa438, 0xd701, + 0xa438, 0x40f4, 0xa438, 0xd196, 0xa438, 0xd04d, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x8802, + 0xa438, 0xcd22, 0xa438, 0xd17a, 0xa438, 0xd05a, 0xa438, 0xa501, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fbd, + 0xa438, 0x8501, 0xa438, 0xa502, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd702, 0xa438, 0x5fbe, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd403, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0x8502, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8801, 0xa438, 0x9503, 0xa438, 0xd40c, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xd707, 0xa438, 0x428f, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce06, 0xa438, 0xf01d, + 0xa438, 0xce06, 0xa438, 0xf01b, 0xa438, 0xce06, 0xa438, 0xf019, + 0xa438, 0xce06, 0xa438, 0xf017, 0xa438, 0xce06, 0xa438, 0xf015, + 0xa438, 0xce06, 0xa438, 0xf013, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce02, 0xa438, 0xf00a, 0xa438, 0xce02, + 0xa438, 0xf008, 0xa438, 0xce01, 0xa438, 0xf006, 0xa438, 0xce01, + 0xa438, 0xf004, 0xa438, 0xce01, 0xa438, 0xf002, 0xa438, 0xce01, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa340, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0xa110, 0xa438, 0x1000, 0xa438, 0x9be6, + 0xa438, 0x8110, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xa304, + 0xa438, 0xa440, 0xa438, 0xa8c0, 0xa438, 0xd707, 0xa438, 0x40cf, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8d07, 0xa438, 0x9503, + 0xa438, 0xa00a, 0xa438, 0xa120, 0xa438, 0xa310, 0xa438, 0x0ca0, + 0xa438, 0x0480, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x853b, + 0xa438, 0x60a4, 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0xa308, + 0xa438, 0xf002, 0xa438, 0x8308, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c60, 0xa438, 0x0340, 0xa438, 0x9503, 0xa438, 0xcd23, + 0xa438, 0xd162, 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x8840, 0xa438, 0xd1c4, + 0xa438, 0xd045, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x6327, 0xa438, 0xd700, + 0xa438, 0x5f3b, 0xa438, 0x88c0, 0xa438, 0x800a, 0xa438, 0x8120, + 0xa438, 0x8358, 0xa438, 0x8308, 0xa438, 0x84a0, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8360, 0xa438, 0x9503, 0xa438, 0xd707, + 0xa438, 0x2f7d, 0xa438, 0x84d7, 0xa438, 0xd17a, 0xa438, 0xd05a, + 0xa438, 0xa501, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, + 0xa438, 0x5fbd, 0xa438, 0x8501, 0xa438, 0xff8b, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, + 0xa438, 0xcd24, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb4, 0xa438, 0x9920, 0xa438, 0x9a20, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x6065, 0xa438, 0x5f94, + 0xa438, 0xffd9, 0xa438, 0xb820, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0x800a, + 0xa438, 0x8120, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8598, + 0xa438, 0x6144, 0xa438, 0xd704, 0xa438, 0x60bd, 0xa438, 0xd707, + 0xa438, 0x417f, 0xa438, 0x4140, 0xa438, 0xf006, 0xa438, 0x82fc, + 0xa438, 0xa201, 0xa438, 0xf007, 0xa438, 0xa2fc, 0xa438, 0xf005, + 0xa438, 0x0cfc, 0xa438, 0x02bc, 0xa438, 0xf002, 0xa438, 0xa2fc, + 0xa438, 0xa304, 0xa438, 0x8880, 0xa438, 0x0cc0, 0xa438, 0x0440, + 0xa438, 0xcd25, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x85b7, + 0xa438, 0x6224, 0xa438, 0xd704, 0xa438, 0x605d, 0xa438, 0xf004, + 0xa438, 0xd704, 0xa438, 0x613b, 0xa438, 0xf00b, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x85b7, 0xa438, 0x60eb, 0xa438, 0xd707, + 0xa438, 0x40bf, 0xa438, 0x4080, 0xa438, 0x8310, 0xa438, 0x1000, + 0xa438, 0x9c0c, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, + 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, + 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8c07, + 0xa438, 0x0c07, 0xa438, 0x0d03, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x85d9, 0xa438, 0x60c4, 0xa438, 0xd704, 0xa438, 0x409d, + 0xa438, 0x8b0f, 0xa438, 0x8c38, 0xa438, 0xf003, 0xa438, 0xab07, + 0xa438, 0xac38, 0xa438, 0x0c07, 0xa438, 0x0d03, 0xa438, 0x9503, + 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa120, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x85e7, 0xa438, 0x6064, 0xa438, 0xd704, + 0xa438, 0x607d, 0xa438, 0xa4a0, 0xa438, 0xa605, 0xa438, 0xd17a, + 0xa438, 0xd049, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x85f9, + 0xa438, 0x6164, 0xa438, 0xd704, 0xa438, 0x413d, 0xa438, 0xa00a, + 0xa438, 0xa120, 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x409a, 0xa438, 0xd706, 0xa438, 0x4046, 0xa438, 0xf014, + 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff5, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x8120, 0xa438, 0x8310, + 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdd, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8b0f, 0xa438, 0x8c38, 0xa438, 0x8d38, + 0xa438, 0x9503, 0xa438, 0xcd26, 0xa438, 0xd704, 0xa438, 0x6291, + 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x862e, 0xa438, 0x37c9, + 0xa438, 0x862b, 0xa438, 0x33a9, 0xa438, 0x8628, 0xa438, 0xd18a, + 0xa438, 0xd04b, 0xa438, 0xf00c, 0xa438, 0xd17a, 0xa438, 0xd04b, + 0xa438, 0xf009, 0xa438, 0xd1c6, 0xa438, 0xd04b, 0xa438, 0xf006, + 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd1c4, + 0xa438, 0xd046, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8641, + 0xa438, 0x6164, 0xa438, 0xd704, 0xa438, 0x413d, 0xa438, 0xa00a, + 0xa438, 0xa120, 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x409a, 0xa438, 0xd706, 0xa438, 0x4048, 0xa438, 0xf014, + 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff5, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x8120, 0xa438, 0x8310, + 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdd, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d01, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, + 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, + 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, + 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xa180, 0xa438, 0xcd27, 0xa438, 0xd704, 0xa438, 0x61f1, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8685, 0xa438, 0x33a9, + 0xa438, 0x8682, 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf009, + 0xa438, 0xd1b7, 0xa438, 0xd049, 0xa438, 0xf006, 0xa438, 0xd17a, + 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd128, 0xa438, 0xd044, + 0xa438, 0xd13b, 0xa438, 0xd055, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x869a, 0xa438, 0x6164, 0xa438, 0xd704, 0xa438, 0x413d, + 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xd704, 0xa438, 0x407b, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd700, 0xa438, 0x405b, 0xa438, 0xf014, 0xa438, 0xd704, + 0xa438, 0x6055, 0xa438, 0xfff7, 0xa438, 0x1000, 0xa438, 0x9c52, + 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8310, 0xa438, 0xa380, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xffdf, 0xa438, 0xa302, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x86c4, 0xa438, 0x6184, 0xa438, 0xd704, + 0xa438, 0x415d, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xd704, + 0xa438, 0x407b, 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, + 0xa438, 0xa302, 0xa438, 0xa403, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x409a, + 0xa438, 0xd706, 0xa438, 0x4048, 0xa438, 0xf014, 0xa438, 0xd704, + 0xa438, 0x6055, 0xa438, 0xfff5, 0xa438, 0x1000, 0xa438, 0x9c52, + 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8312, 0xa438, 0xa380, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xffdc, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b02, 0xa438, 0x0c38, 0xa438, 0x0c10, + 0xa438, 0x9503, 0xa438, 0xcd28, 0xa438, 0xd704, 0xa438, 0x61f1, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x86f5, 0xa438, 0x33a9, + 0xa438, 0x86f2, 0xa438, 0xd199, 0xa438, 0xd04a, 0xa438, 0xf009, + 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf006, 0xa438, 0xd199, + 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd128, 0xa438, 0xd044, + 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8709, 0xa438, 0x6184, + 0xa438, 0xd704, 0xa438, 0x415d, 0xa438, 0xa00a, 0xa438, 0xa1a0, + 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, 0xa438, 0xf002, + 0xa438, 0xa310, 0xa438, 0xa302, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x409a, 0xa438, 0xd706, 0xa438, 0x404a, 0xa438, 0xf014, + 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff5, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8312, + 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdc, 0xa438, 0x8403, + 0xa438, 0xd409, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xbb10, + 0xa438, 0xcd29, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd71f, 0xa438, 0x5f75, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0x8312, 0xa438, 0x8480, 0xa438, 0x8604, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, 0xa438, 0xa140, + 0xa438, 0x9503, 0xa438, 0x9b10, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb5, 0xa438, 0xcd30, 0xa438, 0xaa80, + 0xa438, 0xd704, 0xa438, 0x2319, 0xa438, 0x9d9a, 0xa438, 0xd700, + 0xa438, 0x40bd, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xaf01, + 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5fb4, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8798, + 0xa438, 0xd700, 0xa438, 0x273d, 0xa438, 0x8794, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x876a, 0xa438, 0x33a9, 0xa438, 0x8762, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd1b7, 0xa438, 0xd05c, + 0xa438, 0xf013, 0xa438, 0xd1b7, 0xa438, 0xd05c, 0xa438, 0xf010, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd1b7, 0xa438, 0xd05c, + 0xa438, 0xf00b, 0xa438, 0xd1b7, 0xa438, 0xd05c, 0xa438, 0xf008, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd199, 0xa438, 0xd05d, + 0xa438, 0xf003, 0xa438, 0xd199, 0xa438, 0xd05d, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x8786, 0xa438, 0x33a9, 0xa438, 0x877e, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd1bf, 0xa438, 0xd06d, + 0xa438, 0xf013, 0xa438, 0xd1de, 0xa438, 0xd06d, 0xa438, 0xf010, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd1bf, 0xa438, 0xd06d, + 0xa438, 0xf00b, 0xa438, 0xd1bf, 0xa438, 0xd06d, 0xa438, 0xf008, + 0xa438, 0xd702, 0xa438, 0x6099, 0xa438, 0xd199, 0xa438, 0xd06e, + 0xa438, 0xf003, 0xa438, 0xd199, 0xa438, 0xd06e, 0xa438, 0xd703, + 0xa438, 0x60d0, 0xa438, 0x1000, 0xa438, 0x9c20, 0xa438, 0xd41a, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd408, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xcd31, 0xa438, 0xd700, 0xa438, 0x2fa9, + 0xa438, 0x879f, 0xa438, 0x33c9, 0xa438, 0x87a2, 0xa438, 0x6117, + 0xa438, 0xf00a, 0xa438, 0xd141, 0xa438, 0xd043, 0xa438, 0xf009, + 0xa438, 0xd121, 0xa438, 0xd043, 0xa438, 0xf006, 0xa438, 0xd122, + 0xa438, 0xd042, 0xa438, 0xf003, 0xa438, 0xd181, 0xa438, 0xd043, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8817, 0xa438, 0xd700, + 0xa438, 0x6cab, 0xa438, 0xd700, 0xa438, 0x6d47, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0x8312, 0xa438, 0x8480, 0xa438, 0x0c86, + 0xa438, 0x0680, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8702, + 0xa438, 0x9503, 0xa438, 0xd407, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xd703, 0xa438, 0x60d0, 0xa438, 0x1000, 0xa438, 0x9c20, + 0xa438, 0xd41a, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd406, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xcd39, 0xa438, 0xd404, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce03, 0xa438, 0xf00a, 0xa438, 0xce03, + 0xa438, 0xf008, 0xa438, 0xce03, 0xa438, 0xf006, 0xa438, 0xce03, + 0xa438, 0xf004, 0xa438, 0xce03, 0xa438, 0xf002, 0xa438, 0xce03, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c07, 0xa438, 0x0c01, 0xa438, 0x8d07, 0xa438, 0x9503, + 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa302, 0xa438, 0x0ca0, + 0xa438, 0x0480, 0xa438, 0xa684, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xb920, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x8810, + 0xa438, 0x9920, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x60e5, 0xa438, 0x5f94, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xfff0, 0xa438, 0xb820, + 0xa438, 0xa810, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd706, 0xa438, 0x5f69, + 0xa438, 0xf010, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xae80, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x2b6d, 0xa438, 0x88c0, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, + 0xa438, 0xd406, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xcd32, + 0xa438, 0xd701, 0xa438, 0x6191, 0xa438, 0xa504, 0xa438, 0xcd3a, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x6067, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd701, 0xa438, 0x5f3a, + 0xa438, 0x8504, 0xa438, 0xd700, 0xa438, 0x2739, 0xa438, 0x88b6, + 0xa438, 0xd707, 0xa438, 0x6061, 0xa438, 0x1800, 0xa438, 0x88a8, + 0xa438, 0xd193, 0xa438, 0xd047, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f29, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, + 0xa438, 0xf008, 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, + 0xa438, 0xf004, 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b01, 0xa438, 0x0c3f, 0xa438, 0x0c08, + 0xa438, 0x9503, 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4062, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa120, + 0xa438, 0xa420, 0xa438, 0xd193, 0xa438, 0xd048, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f29, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce03, 0xa438, 0xf00a, + 0xa438, 0xce03, 0xa438, 0xf008, 0xa438, 0xce03, 0xa438, 0xf006, + 0xa438, 0xce03, 0xa438, 0xf004, 0xa438, 0xce03, 0xa438, 0xf002, + 0xa438, 0xce03, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0c01, 0xa438, 0x9503, + 0xa438, 0x8420, 0xa438, 0x800a, 0xa438, 0x8120, 0xa438, 0x8312, + 0xa438, 0x8480, 0xa438, 0x8604, 0xa438, 0xd419, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xd702, 0xa438, 0x4080, 0xa438, 0xbb20, + 0xa438, 0x1800, 0xa438, 0x8c32, 0xa438, 0xa00a, 0xa438, 0xa302, + 0xa438, 0xa480, 0xa438, 0xa604, 0xa438, 0xd193, 0xa438, 0xd047, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f29, + 0xa438, 0x1800, 0xa438, 0x888b, 0xa438, 0x800a, 0xa438, 0x8302, + 0xa438, 0x8480, 0xa438, 0x8604, 0xa438, 0xd405, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xbb20, 0xa438, 0xa00a, 0xa438, 0xa302, + 0xa438, 0xa480, 0xa438, 0xa604, 0xa438, 0x1800, 0xa438, 0x8c32, + 0xa438, 0xd405, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd404, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd700, 0xa438, 0x2b69, + 0xa438, 0x88c0, 0xa438, 0xf06d, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce03, 0xa438, 0xf00a, 0xa438, 0xce03, + 0xa438, 0xf008, 0xa438, 0xce03, 0xa438, 0xf006, 0xa438, 0xce03, + 0xa438, 0xf004, 0xa438, 0xce03, 0xa438, 0xf002, 0xa438, 0xce03, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c07, 0xa438, 0x0c01, 0xa438, 0x0c3f, 0xa438, 0x0d08, + 0xa438, 0x9503, 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa302, + 0xa438, 0x0ca0, 0xa438, 0x0480, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x88e8, 0xa438, 0x60a4, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x8604, 0xa438, 0xf002, 0xa438, 0xa604, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0xd700, 0xa438, 0x37cd, + 0xa438, 0x88f0, 0xa438, 0x8310, 0xa438, 0x9503, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xb920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, + 0xa438, 0x8810, 0xa438, 0x9920, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x60e5, 0xa438, 0x5f94, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xfff0, + 0xa438, 0xb820, 0xa438, 0xa810, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0xd700, + 0xa438, 0x2d59, 0xa438, 0x8c32, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8c07, 0xa438, 0x9503, 0xa438, 0xa120, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x891f, 0xa438, 0x60e4, 0xa438, 0xd704, + 0xa438, 0x40bd, 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x8928, 0xa438, 0x60a4, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0xa202, 0xa438, 0xf002, 0xa438, 0x8202, 0xa438, 0xa420, + 0xa438, 0x1800, 0xa438, 0x8d48, 0xa438, 0x8810, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fbb, 0xa438, 0xd17a, + 0xa438, 0xd05a, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, + 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, + 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b04, 0xa438, 0x0c3f, 0xa438, 0x0c21, 0xa438, 0x8d07, + 0xa438, 0x9503, 0xa438, 0xa340, 0xa438, 0x1000, 0xa438, 0x9c52, + 0xa438, 0xa110, 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8110, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xa304, 0xa438, 0xa440, + 0xa438, 0xa8c0, 0xa438, 0x8810, 0xa438, 0xa00a, 0xa438, 0xa120, + 0xa438, 0xa310, 0xa438, 0xd704, 0xa438, 0x405d, 0xa438, 0xa308, + 0xa438, 0x0cfc, 0xa438, 0x0224, 0xa438, 0x0ca0, 0xa438, 0x0480, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0xd700, + 0xa438, 0x37cd, 0xa438, 0x896d, 0xa438, 0x8310, 0xa438, 0x9503, + 0xa438, 0xd162, 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x8840, 0xa438, 0xd1c4, + 0xa438, 0xd045, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd706, 0xa438, 0x6127, 0xa438, 0xd700, + 0xa438, 0x5f3b, 0xa438, 0x88c0, 0xa438, 0x800a, 0xa438, 0x8120, + 0xa438, 0x8350, 0xa438, 0x84a0, 0xa438, 0xffad, 0xa438, 0xb920, + 0xa438, 0xcd33, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb4, 0xa438, 0x9920, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x6065, 0xa438, 0x5f94, 0xa438, 0xffee, + 0xa438, 0xb820, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0x800a, 0xa438, 0x8120, + 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, 0xa438, 0x9c5b, + 0xa438, 0xd704, 0xa438, 0x60bd, 0xa438, 0xd707, 0xa438, 0x413f, + 0xa438, 0x4100, 0xa438, 0xf004, 0xa438, 0x0cfd, 0xa438, 0x0201, + 0xa438, 0xf005, 0xa438, 0x0cfc, 0xa438, 0x02bc, 0xa438, 0xf002, + 0xa438, 0xa2fc, 0xa438, 0xa304, 0xa438, 0x8880, 0xa438, 0x0cc0, + 0xa438, 0x0440, 0xa438, 0xcd34, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x405b, 0xa438, 0xf006, 0xa438, 0xd704, 0xa438, 0x60fd, + 0xa438, 0xd707, 0xa438, 0x40bf, 0xa438, 0x4080, 0xa438, 0x8310, + 0xa438, 0x1000, 0xa438, 0x9c0c, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8c07, 0xa438, 0xd704, 0xa438, 0x409d, 0xa438, 0x8b0f, + 0xa438, 0x8c38, 0xa438, 0xf004, 0xa438, 0x0c0f, 0xa438, 0x0b07, + 0xa438, 0xac38, 0xa438, 0x0c38, 0xa438, 0x0d10, 0xa438, 0x9503, + 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa120, 0xa438, 0xd704, + 0xa438, 0x607d, 0xa438, 0xa4a0, 0xa438, 0xa604, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x89db, 0xa438, 0xd17a, 0xa438, 0xd049, + 0xa438, 0xf003, 0xa438, 0xd19f, 0xa438, 0xd049, 0xa438, 0xd704, + 0xa438, 0x413d, 0xa438, 0xa00a, 0xa438, 0xa120, 0xa438, 0xd704, + 0xa438, 0x407b, 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, + 0xa438, 0xa403, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x6a14, 0xa438, 0xd700, 0xa438, 0x409a, 0xa438, 0xd706, + 0xa438, 0x4988, 0xa438, 0xf014, 0xa438, 0xd704, 0xa438, 0x6055, + 0xa438, 0xfff5, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, + 0xa438, 0x8120, 0xa438, 0x8310, 0xa438, 0xa380, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, + 0xa438, 0x5f77, 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, + 0xa438, 0xffdd, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8b0f, + 0xa438, 0x8c38, 0xa438, 0x8d38, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x8a14, 0xa438, 0x33a9, 0xa438, 0x8a11, + 0xa438, 0xd1f4, 0xa438, 0xd04b, 0xa438, 0xf006, 0xa438, 0xd1b7, + 0xa438, 0xd04b, 0xa438, 0xf003, 0xa438, 0xd1c6, 0xa438, 0xd04c, + 0xa438, 0xd704, 0xa438, 0x413d, 0xa438, 0xa00a, 0xa438, 0xa120, + 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, 0xa438, 0xf002, + 0xa438, 0xa310, 0xa438, 0xa403, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x62f4, 0xa438, 0xd700, 0xa438, 0x405a, + 0xa438, 0xf01a, 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff7, + 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x8120, + 0xa438, 0x8310, 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, + 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdf, + 0xa438, 0x8403, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x629c, 0xa438, 0xfffb, 0xa438, 0x8403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x6096, 0xa438, 0xd700, + 0xa438, 0x619c, 0xa438, 0xfffa, 0xa438, 0xd706, 0xa438, 0x4128, + 0xa438, 0xd702, 0xa438, 0x60b0, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xae80, 0xa438, 0x9503, 0xa438, 0x1800, 0xa438, 0x8b62, + 0xa438, 0xd17a, 0xa438, 0xd05a, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, + 0xa438, 0xf008, 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, + 0xa438, 0xf004, 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b04, 0xa438, 0x0c3f, 0xa438, 0x0c21, + 0xa438, 0x8d07, 0xa438, 0x9503, 0xa438, 0xa340, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0xa110, 0xa438, 0x1000, 0xa438, 0x9be6, + 0xa438, 0x8110, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xa304, + 0xa438, 0xa440, 0xa438, 0xa8c0, 0xa438, 0x8810, 0xa438, 0xa00a, + 0xa438, 0xa120, 0xa438, 0xa310, 0xa438, 0xd704, 0xa438, 0x405d, + 0xa438, 0xa308, 0xa438, 0x0cfc, 0xa438, 0x0224, 0xa438, 0x0ca0, + 0xa438, 0x0480, 0xa438, 0x8604, 0xa438, 0xcd35, 0xa438, 0xd162, + 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x8840, 0xa438, 0xd1c4, 0xa438, 0xd045, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd706, 0xa438, 0x6127, 0xa438, 0xd700, 0xa438, 0x5f3b, + 0xa438, 0x88c0, 0xa438, 0x800a, 0xa438, 0x8120, 0xa438, 0x8350, + 0xa438, 0x84a0, 0xa438, 0xffb3, 0xa438, 0xbb80, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, + 0xa438, 0xcd36, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb4, 0xa438, 0x9920, 0xa438, 0x9b80, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x6065, 0xa438, 0x5f94, + 0xa438, 0xffe8, 0xa438, 0xb820, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0x800a, + 0xa438, 0x8120, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0xa108, + 0xa438, 0x1000, 0xa438, 0x9be6, 0xa438, 0x8108, 0xa438, 0x1000, + 0xa438, 0x9c5b, 0xa438, 0xd704, 0xa438, 0x60bd, 0xa438, 0xd707, + 0xa438, 0x413f, 0xa438, 0x4100, 0xa438, 0xf004, 0xa438, 0x0cfd, + 0xa438, 0x0201, 0xa438, 0xf005, 0xa438, 0x0cfc, 0xa438, 0x02bc, + 0xa438, 0xf002, 0xa438, 0xa2fc, 0xa438, 0xa304, 0xa438, 0x8880, + 0xa438, 0x0cc0, 0xa438, 0x0440, 0xa438, 0xcd37, 0xa438, 0xd704, + 0xa438, 0x407d, 0xa438, 0x405b, 0xa438, 0xf006, 0xa438, 0xd704, + 0xa438, 0x60fd, 0xa438, 0xd707, 0xa438, 0x40bf, 0xa438, 0x4080, + 0xa438, 0x8310, 0xa438, 0x1000, 0xa438, 0x9c0c, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8c07, 0xa438, 0xd704, 0xa438, 0x409d, + 0xa438, 0x8b0f, 0xa438, 0x8c38, 0xa438, 0xf004, 0xa438, 0x0c0f, + 0xa438, 0x0b07, 0xa438, 0xac38, 0xa438, 0x0c38, 0xa438, 0x0d10, + 0xa438, 0x9503, 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa120, + 0xa438, 0xd704, 0xa438, 0x607d, 0xa438, 0xa4a0, 0xa438, 0xa604, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8b01, 0xa438, 0x33a9, + 0xa438, 0x8afe, 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0xf006, + 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0xf003, 0xa438, 0xd17a, + 0xa438, 0xd048, 0xa438, 0xd704, 0xa438, 0x413d, 0xa438, 0xa00a, + 0xa438, 0xa120, 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x409a, 0xa438, 0xd706, 0xa438, 0x4048, 0xa438, 0xf014, + 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff5, 0xa438, 0x1000, + 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x8120, 0xa438, 0x8310, + 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, 0xa438, 0x8380, + 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdd, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8b0f, 0xa438, 0x8c38, 0xa438, 0x8d38, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8b3a, + 0xa438, 0x33a9, 0xa438, 0x8b37, 0xa438, 0xd199, 0xa438, 0xd04b, + 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf003, + 0xa438, 0xd189, 0xa438, 0xd04c, 0xa438, 0xd704, 0xa438, 0x413d, + 0xa438, 0xa00a, 0xa438, 0xa120, 0xa438, 0xd704, 0xa438, 0x407b, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd700, 0xa438, 0x409a, 0xa438, 0xd706, 0xa438, 0x4048, + 0xa438, 0xf014, 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff5, + 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x8120, + 0xa438, 0x8310, 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, + 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdd, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d01, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, + 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, + 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa180, 0xa438, 0xcd38, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x8b92, 0xa438, 0x33a9, 0xa438, 0x8b8a, + 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, 0xa438, 0xd04a, + 0xa438, 0xf013, 0xa438, 0xd100, 0xa438, 0xd049, 0xa438, 0xf010, + 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd1b7, 0xa438, 0xd049, + 0xa438, 0xf00b, 0xa438, 0xd100, 0xa438, 0xd048, 0xa438, 0xf008, + 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, 0xa438, 0xd04b, + 0xa438, 0xf003, 0xa438, 0xd199, 0xa438, 0xd04a, 0xa438, 0xd13b, + 0xa438, 0xd055, 0xa438, 0xd704, 0xa438, 0x413d, 0xa438, 0xa00a, + 0xa438, 0xa1a0, 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x405b, 0xa438, 0xf014, 0xa438, 0xd704, 0xa438, 0x6055, + 0xa438, 0xfff7, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0x8310, 0xa438, 0xa380, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, + 0xa438, 0x5f77, 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, + 0xa438, 0xffdf, 0xa438, 0xa302, 0xa438, 0xd704, 0xa438, 0x415d, + 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xd704, 0xa438, 0x407b, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xa302, + 0xa438, 0xa403, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x409a, 0xa438, 0xd706, + 0xa438, 0x4048, 0xa438, 0xf014, 0xa438, 0xd704, 0xa438, 0x6055, + 0xa438, 0xfff5, 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0x8312, 0xa438, 0xa380, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, + 0xa438, 0x5f77, 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, + 0xa438, 0xffdc, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b02, 0xa438, 0x0c38, 0xa438, 0x0c10, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8c03, 0xa438, 0x33a9, + 0xa438, 0x8bfb, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd199, + 0xa438, 0xd04a, 0xa438, 0xf013, 0xa438, 0xd100, 0xa438, 0xd04a, + 0xa438, 0xf010, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, + 0xa438, 0xd04a, 0xa438, 0xf00b, 0xa438, 0xd100, 0xa438, 0xd048, + 0xa438, 0xf008, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd199, + 0xa438, 0xd04b, 0xa438, 0xf003, 0xa438, 0xd16b, 0xa438, 0xd04b, + 0xa438, 0xd704, 0xa438, 0x415d, 0xa438, 0xa00a, 0xa438, 0xa1a0, + 0xa438, 0xd704, 0xa438, 0x407b, 0xa438, 0x8310, 0xa438, 0xf002, + 0xa438, 0xa310, 0xa438, 0xa302, 0xa438, 0xa403, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x40ba, 0xa438, 0xd706, 0xa438, 0x406a, 0xa438, 0x8403, + 0xa438, 0xf014, 0xa438, 0xd704, 0xa438, 0x6055, 0xa438, 0xfff4, + 0xa438, 0x1000, 0xa438, 0x9c52, 0xa438, 0x800a, 0xa438, 0x81a0, + 0xa438, 0x8312, 0xa438, 0xa380, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd705, 0xa438, 0x5f77, + 0xa438, 0x8380, 0xa438, 0x1000, 0xa438, 0x9c5b, 0xa438, 0xffdb, + 0xa438, 0x81a0, 0xa438, 0x8310, 0xa438, 0xa302, 0xa438, 0xa00a, + 0xa438, 0xa480, 0xa438, 0x8420, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x8c42, 0xa438, 0x60e4, 0xa438, 0xd700, 0xa438, 0x40a7, + 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0x8604, 0xa438, 0xf002, + 0xa438, 0xa604, 0xa438, 0xd700, 0xa438, 0x60c7, 0xa438, 0xa602, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa702, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce03, + 0xa438, 0xf00a, 0xa438, 0xce03, 0xa438, 0xf008, 0xa438, 0xce03, + 0xa438, 0xf006, 0xa438, 0xce03, 0xa438, 0xf004, 0xa438, 0xce03, + 0xa438, 0xf002, 0xa438, 0xce03, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0c01, + 0xa438, 0x0c3f, 0xa438, 0x0d08, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x646d, 0xa438, 0x37c9, 0xa438, 0x8c83, 0xa438, 0x33a9, + 0xa438, 0x8c79, 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, + 0xa438, 0x6098, 0xa438, 0xd100, 0xa438, 0xd048, 0xa438, 0xf01a, + 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0xf017, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd100, + 0xa438, 0xd049, 0xa438, 0xf010, 0xa438, 0xd17a, 0xa438, 0xd04c, + 0xa438, 0xf00d, 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, + 0xa438, 0x6098, 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf006, + 0xa438, 0xd17a, 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd17a, + 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, + 0xa438, 0x5f29, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, + 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, + 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b01, 0xa438, 0x0c3f, 0xa438, 0x0c08, 0xa438, 0x9503, + 0xa438, 0xa120, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8cc7, + 0xa438, 0x6224, 0xa438, 0xd700, 0xa438, 0x4147, 0xa438, 0xd704, + 0xa438, 0x409d, 0xa438, 0xd704, 0xa438, 0x613b, 0xa438, 0xf00a, + 0xa438, 0xd707, 0xa438, 0x411f, 0xa438, 0x40e0, 0xa438, 0xf004, + 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4062, 0xa438, 0x8310, + 0xa438, 0xf002, 0xa438, 0xa310, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x8cd2, 0xa438, 0x60e4, 0xa438, 0xd700, 0xa438, 0x40a7, + 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0xa202, 0xa438, 0xf002, + 0xa438, 0x8202, 0xa438, 0xa420, 0xa438, 0xcd3b, 0xa438, 0xd700, + 0xa438, 0x65ad, 0xa438, 0x43c7, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x8ced, 0xa438, 0x33a9, 0xa438, 0x8ce5, 0xa438, 0xd702, + 0xa438, 0x4098, 0xa438, 0xd199, 0xa438, 0xd04c, 0xa438, 0xf024, + 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0xf021, 0xa438, 0xd702, + 0xa438, 0x4098, 0xa438, 0xd1c6, 0xa438, 0xd04c, 0xa438, 0xf01c, + 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0xf019, 0xa438, 0xd702, + 0xa438, 0x4098, 0xa438, 0xd199, 0xa438, 0xd04c, 0xa438, 0xf014, + 0xa438, 0xd17a, 0xa438, 0xd04c, 0xa438, 0xf011, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x8d00, 0xa438, 0x33a9, 0xa438, 0x8cfd, + 0xa438, 0xd1e5, 0xa438, 0xd04c, 0xa438, 0xf009, 0xa438, 0xd191, + 0xa438, 0xd04d, 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd04d, + 0xa438, 0xf003, 0xa438, 0xd16b, 0xa438, 0xd04c, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2c, 0xa438, 0xd700, + 0xa438, 0x40e7, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c07, + 0xa438, 0x0d01, 0xa438, 0x9503, 0xa438, 0xf006, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, + 0xa438, 0xcd3c, 0xa438, 0xd700, 0xa438, 0x644d, 0xa438, 0x43c7, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8d33, 0xa438, 0x33a9, + 0xa438, 0x8d2b, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, + 0xa438, 0xd04a, 0xa438, 0xf019, 0xa438, 0xd17a, 0xa438, 0xd048, + 0xa438, 0xf016, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, + 0xa438, 0xd04b, 0xa438, 0xf011, 0xa438, 0xd17a, 0xa438, 0xd048, + 0xa438, 0xf00e, 0xa438, 0xd702, 0xa438, 0x4098, 0xa438, 0xd17a, + 0xa438, 0xd04a, 0xa438, 0xf009, 0xa438, 0xd17a, 0xa438, 0xd049, + 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd04b, 0xa438, 0xf003, + 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2d, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, + 0xa438, 0xf008, 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, + 0xa438, 0xf004, 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b02, 0xa438, 0x0c38, 0xa438, 0x0c10, + 0xa438, 0x9503, 0xa438, 0xa180, 0xa438, 0xcd3d, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x8d8a, 0xa438, 0x37c9, 0xa438, 0x8d80, + 0xa438, 0x33a9, 0xa438, 0x8d76, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd15c, 0xa438, 0xd04a, + 0xa438, 0xf01a, 0xa438, 0xd199, 0xa438, 0xd04b, 0xa438, 0xf017, + 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, + 0xa438, 0xd13e, 0xa438, 0xd04a, 0xa438, 0xf010, 0xa438, 0xd199, + 0xa438, 0xd04b, 0xa438, 0xf00d, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd17a, 0xa438, 0xd04c, + 0xa438, 0xf006, 0xa438, 0xd199, 0xa438, 0xd04c, 0xa438, 0xf003, + 0xa438, 0xd17a, 0xa438, 0xd04b, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x8da5, 0xa438, 0x33a9, 0xa438, 0x8d9b, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd100, + 0xa438, 0xd050, 0xa438, 0xf017, 0xa438, 0xd17a, 0xa438, 0xd05a, + 0xa438, 0xf014, 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, + 0xa438, 0x6098, 0xa438, 0xd100, 0xa438, 0xd050, 0xa438, 0xf00d, + 0xa438, 0xd17a, 0xa438, 0xd05a, 0xa438, 0xf00a, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd17a, + 0xa438, 0xd05a, 0xa438, 0xf003, 0xa438, 0xd17a, 0xa438, 0xd05a, + 0xa438, 0xd707, 0xa438, 0x3ad1, 0xa438, 0x8e73, 0xa438, 0xd705, + 0xa438, 0x36b1, 0xa438, 0x8ddd, 0xa438, 0xd700, 0xa438, 0x2969, + 0xa438, 0x8dd5, 0xa438, 0x63c4, 0xa438, 0xd700, 0xa438, 0x40e7, + 0xa438, 0xd704, 0xa438, 0x40bd, 0xa438, 0xd704, 0xa438, 0x60dc, + 0xa438, 0x63fb, 0xa438, 0xf016, 0xa438, 0xd707, 0xa438, 0x429f, + 0xa438, 0xf001, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7b, 0xa438, 0xd700, + 0xa438, 0x40e7, 0xa438, 0xd704, 0xa438, 0x40bd, 0xa438, 0xd704, + 0xa438, 0x407c, 0xa438, 0x1800, 0xa438, 0x8e73, 0xa438, 0xd706, + 0xa438, 0x2b59, 0xa438, 0x8e73, 0xa438, 0xf009, 0xa438, 0xaa20, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd701, 0xa438, 0x5f72, 0xa438, 0x8a20, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd700, + 0xa438, 0x4287, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf01e, 0xa438, 0xce08, 0xa438, 0xf01c, + 0xa438, 0xce08, 0xa438, 0xf01a, 0xa438, 0xce08, 0xa438, 0xf018, + 0xa438, 0xce08, 0xa438, 0xf016, 0xa438, 0xce08, 0xa438, 0xf014, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, + 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf009, 0xa438, 0xce08, + 0xa438, 0xf007, 0xa438, 0xce08, 0xa438, 0xf005, 0xa438, 0xce08, + 0xa438, 0xf003, 0xa438, 0xce08, 0xa438, 0xf001, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8e23, + 0xa438, 0x6224, 0xa438, 0xd700, 0xa438, 0x4147, 0xa438, 0xd704, + 0xa438, 0x409d, 0xa438, 0xd704, 0xa438, 0x417b, 0xa438, 0xf008, + 0xa438, 0xd707, 0xa438, 0x411f, 0xa438, 0x40e0, 0xa438, 0xf004, + 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4062, 0xa438, 0x8320, + 0xa438, 0xf002, 0xa438, 0xa320, 0xa438, 0x8310, 0xa438, 0xcd3e, + 0xa438, 0xd700, 0xa438, 0x2969, 0xa438, 0x8e4b, 0xa438, 0x37c9, + 0xa438, 0x8e41, 0xa438, 0x33a9, 0xa438, 0x8e37, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd15d, + 0xa438, 0xd04b, 0xa438, 0xf01a, 0xa438, 0xd17a, 0xa438, 0xd04b, + 0xa438, 0xf017, 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, + 0xa438, 0x6098, 0xa438, 0xd16b, 0xa438, 0xd04b, 0xa438, 0xf010, + 0xa438, 0xd17a, 0xa438, 0xd04b, 0xa438, 0xf00d, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd17a, + 0xa438, 0xd04b, 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd04b, + 0xa438, 0xf003, 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x8e6b, 0xa438, 0x6264, 0xa438, 0xd700, + 0xa438, 0x4187, 0xa438, 0xd704, 0xa438, 0x40bd, 0xa438, 0xd704, + 0xa438, 0x41bb, 0xa438, 0x1800, 0xa438, 0x8f25, 0xa438, 0xd707, + 0xa438, 0x413f, 0xa438, 0x4100, 0xa438, 0x1800, 0xa438, 0x8f25, + 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4060, 0xa438, 0x1800, + 0xa438, 0x8f25, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd701, 0xa438, 0x5f77, 0xa438, 0x1800, + 0xa438, 0x8f25, 0xa438, 0xd700, 0xa438, 0x4287, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf01e, + 0xa438, 0xce08, 0xa438, 0xf01c, 0xa438, 0xce08, 0xa438, 0xf01a, + 0xa438, 0xce08, 0xa438, 0xf018, 0xa438, 0xce08, 0xa438, 0xf016, + 0xa438, 0xce08, 0xa438, 0xf014, 0xa438, 0xd700, 0xa438, 0x60cf, + 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, + 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00b, 0xa438, 0xce08, + 0xa438, 0xf009, 0xa438, 0xce08, 0xa438, 0xf007, 0xa438, 0xce08, + 0xa438, 0xf005, 0xa438, 0xce08, 0xa438, 0xf003, 0xa438, 0xce08, + 0xa438, 0xf001, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa102, 0xa438, 0xa802, 0xa438, 0x9503, + 0xa438, 0xaa02, 0xa438, 0xd700, 0xa438, 0x40e7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c0f, 0xa438, 0x0b05, 0xa438, 0xac38, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd704, + 0xa438, 0x60fd, 0xa438, 0xd707, 0xa438, 0x40a0, 0xa438, 0xf003, + 0xa438, 0xd707, 0xa438, 0x4042, 0xa438, 0xa308, 0xa438, 0xa310, + 0xa438, 0xcd3f, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8ed0, + 0xa438, 0x33a9, 0xa438, 0x8ec6, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd1d6, 0xa438, 0xd04a, + 0xa438, 0xf017, 0xa438, 0xd1d6, 0xa438, 0xd04a, 0xa438, 0xf014, + 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, + 0xa438, 0xd1d6, 0xa438, 0xd04a, 0xa438, 0xf00d, 0xa438, 0xd1d6, + 0xa438, 0xd04a, 0xa438, 0xf00a, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd199, 0xa438, 0xd04a, + 0xa438, 0xf003, 0xa438, 0xd199, 0xa438, 0xd04a, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0x0c30, 0xa438, 0x0320, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x8efa, 0xa438, 0x33a9, 0xa438, 0x8ef0, + 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, + 0xa438, 0xd199, 0xa438, 0xd04a, 0xa438, 0xf017, 0xa438, 0xd17a, + 0xa438, 0xd04b, 0xa438, 0xf014, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd702, 0xa438, 0x6098, 0xa438, 0xd199, 0xa438, 0xd04a, + 0xa438, 0xf00d, 0xa438, 0xd17a, 0xa438, 0xd04b, 0xa438, 0xf00a, + 0xa438, 0xd700, 0xa438, 0x40c7, 0xa438, 0xd702, 0xa438, 0x6098, + 0xa438, 0xd17a, 0xa438, 0xd04b, 0xa438, 0xf003, 0xa438, 0xd17a, + 0xa438, 0xd04b, 0xa438, 0xd700, 0xa438, 0x4287, 0xa438, 0xd704, + 0xa438, 0x425d, 0xa438, 0xd704, 0xa438, 0x421c, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd701, + 0xa438, 0x5f77, 0xa438, 0x1000, 0xa438, 0x9c16, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xf009, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd701, 0xa438, 0x5f37, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8102, 0xa438, 0x9503, 0xa438, 0x8a02, 0xa438, 0xcd40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8380, 0xa438, 0x9503, + 0xa438, 0xd40a, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xbb10, + 0xa438, 0xcd41, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd71f, 0xa438, 0x5f75, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0x8302, 0xa438, 0x8480, 0xa438, 0xd700, + 0xa438, 0x6047, 0xa438, 0xa680, 0xa438, 0x8606, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8340, 0xa438, 0xd700, 0xa438, 0x6060, + 0xa438, 0xa120, 0xa438, 0xa302, 0xa438, 0x9503, 0xa438, 0x8c40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8750, 0xa438, 0x8702, + 0xa438, 0x9503, 0xa438, 0x9b30, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8011, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb5, 0xa438, 0xcd42, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5f94, 0xa438, 0xd700, + 0xa438, 0x2969, 0xa438, 0x8f67, 0xa438, 0xd700, 0xa438, 0x6140, + 0xa438, 0xd705, 0xa438, 0x611e, 0xa438, 0xd703, 0xa438, 0x40d0, + 0xa438, 0x1000, 0xa438, 0x9c20, 0xa438, 0xd41a, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0x8340, 0xa438, 0xa801, 0xa438, 0xd700, + 0xa438, 0x2fa9, 0xa438, 0x8f70, 0xa438, 0x33c9, 0xa438, 0x8f73, + 0xa438, 0x6117, 0xa438, 0xf00a, 0xa438, 0xd141, 0xa438, 0xd043, + 0xa438, 0xf009, 0xa438, 0xd121, 0xa438, 0xd043, 0xa438, 0xf006, + 0xa438, 0xd122, 0xa438, 0xd042, 0xa438, 0xf003, 0xa438, 0xd181, + 0xa438, 0xd043, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, + 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, + 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd705, 0xa438, 0x611e, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0ccf, 0xa438, 0x0b02, 0xa438, 0x8cc7, + 0xa438, 0x9503, 0xa438, 0xf008, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0ccf, 0xa438, 0x0b46, 0xa438, 0x0cc7, 0xa438, 0x0c03, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x40e7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d09, 0xa438, 0x9503, + 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, + 0xa438, 0x0d0a, 0xa438, 0x9503, 0xa438, 0xd705, 0xa438, 0x607e, + 0xa438, 0xa302, 0xa438, 0xf00b, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa011, 0xa438, 0x9503, 0xa438, 0xd14f, 0xa438, 0xd043, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xa810, 0xa438, 0x8240, 0xa438, 0xa00a, 0xa438, 0xa1a0, + 0xa438, 0xa480, 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, + 0xa438, 0x407d, 0xa438, 0x8604, 0xa438, 0xf002, 0xa438, 0xa604, + 0xa438, 0xd700, 0xa438, 0x60c7, 0xa438, 0xa682, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa702, 0xa438, 0x9503, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xb920, + 0xa438, 0xcd43, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb4, 0xa438, 0x8810, 0xa438, 0x9920, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x60e5, 0xa438, 0x5f94, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, + 0xa438, 0xffef, 0xa438, 0xb820, 0xa438, 0xa810, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, + 0xa438, 0xbb20, 0xa438, 0xcd44, 0xa438, 0xd700, 0xa438, 0x4060, + 0xa438, 0x1800, 0xa438, 0x9072, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x9003, 0xa438, 0x33a9, 0xa438, 0x9000, 0xa438, 0xd17a, + 0xa438, 0xd047, 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd048, + 0xa438, 0xf003, 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd700, + 0xa438, 0x2d49, 0xa438, 0x9014, 0xa438, 0xd701, 0xa438, 0x60b0, + 0xa438, 0x1800, 0xa438, 0x90c4, 0xa438, 0x1800, 0xa438, 0x911b, + 0xa438, 0x0c06, 0xa438, 0x0a06, 0xa438, 0x0cc0, 0xa438, 0x0cc0, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa7a0, 0xa438, 0x9503, + 0xa438, 0xcd45, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x902a, + 0xa438, 0x33a9, 0xa438, 0x9027, 0xa438, 0xd17a, 0xa438, 0xd047, + 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0xf003, + 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0x8c40, 0xa438, 0xd418, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xac40, 0xa438, 0xaa01, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd706, 0xa438, 0x5f6f, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x904b, 0xa438, 0x33a9, 0xa438, 0x9048, 0xa438, 0xd17a, + 0xa438, 0xd047, 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd048, + 0xa438, 0xf003, 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xab20, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, + 0xa438, 0xcd46, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x9066, + 0xa438, 0x33a9, 0xa438, 0x9063, 0xa438, 0xd1b7, 0xa438, 0xd04d, + 0xa438, 0xf006, 0xa438, 0xd1b7, 0xa438, 0xd04d, 0xa438, 0xf003, + 0xa438, 0xd1b7, 0xa438, 0xd04d, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0x1800, 0xa438, 0x90c4, + 0xa438, 0xd701, 0xa438, 0x4830, 0xa438, 0xd700, 0xa438, 0x47e7, + 0xa438, 0xd704, 0xa438, 0x47bd, 0xa438, 0xd704, 0xa438, 0x477c, + 0xa438, 0x0c06, 0xa438, 0x0a06, 0xa438, 0x0cc0, 0xa438, 0x0cc0, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x9089, 0xa438, 0x33a9, + 0xa438, 0x9086, 0xa438, 0xd17a, 0xa438, 0xd047, 0xa438, 0xf006, + 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0xf003, 0xa438, 0xd17a, + 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, + 0xa438, 0x5f2f, 0xa438, 0x8c40, 0xa438, 0xd418, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xac40, 0xa438, 0xaa01, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd706, + 0xa438, 0x5f6f, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x90aa, + 0xa438, 0x33a9, 0xa438, 0x90a7, 0xa438, 0xd17a, 0xa438, 0xd047, + 0xa438, 0xf006, 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0xf003, + 0xa438, 0xd17a, 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd701, 0xa438, 0x40f0, + 0xa438, 0xac40, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa750, + 0xa438, 0x9503, 0xa438, 0xab20, 0xa438, 0xd1c4, 0xa438, 0xd046, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x424f, + 0xa438, 0x8c40, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8750, + 0xa438, 0x8380, 0xa438, 0x9503, 0xa438, 0xd417, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xd701, 0xa438, 0x40d0, 0xa438, 0xac40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, + 0xa438, 0xcd47, 0xa438, 0xd700, 0xa438, 0x686b, 0xa438, 0x6060, + 0xa438, 0x1800, 0xa438, 0x90f3, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x90e7, 0xa438, 0x33a9, 0xa438, 0x90e4, 0xa438, 0xd15c, + 0xa438, 0xd04c, 0xa438, 0xf006, 0xa438, 0xd182, 0xa438, 0xd04c, + 0xa438, 0xf003, 0xa438, 0xd191, 0xa438, 0xd04c, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0x1800, + 0xa438, 0x911b, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0ccf, + 0xa438, 0x0b03, 0xa438, 0x8cc7, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x60e7, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, + 0xa438, 0x0d1a, 0xa438, 0x9503, 0xa438, 0xf006, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d19, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x9111, 0xa438, 0x33a9, + 0xa438, 0x910e, 0xa438, 0xd19f, 0xa438, 0xd049, 0xa438, 0xf006, + 0xa438, 0xd199, 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd199, + 0xa438, 0xd04b, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, + 0xa438, 0x5f2f, 0xa438, 0xd416, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xbb10, 0xa438, 0xcd4f, 0xa438, 0xcd50, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd71f, + 0xa438, 0x5f75, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8302, + 0xa438, 0x8480, 0xa438, 0xd700, 0xa438, 0x6047, 0xa438, 0xa682, + 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x8604, 0xa438, 0xf002, 0xa438, 0xa604, 0xa438, 0x8818, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, 0xa438, 0xa120, + 0xa438, 0xa302, 0xa438, 0x9503, 0xa438, 0x8c40, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8750, 0xa438, 0x9503, 0xa438, 0x9b30, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8011, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb5, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, + 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, + 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, + 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xcd51, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5f94, 0xa438, 0xd71e, 0xa438, 0x6103, 0xa438, 0xd700, + 0xa438, 0x2fa9, 0xa438, 0x916f, 0xa438, 0x33c9, 0xa438, 0x9172, + 0xa438, 0x6177, 0xa438, 0x61b1, 0xa438, 0xd101, 0xa438, 0xd040, + 0xa438, 0xf00c, 0xa438, 0xd141, 0xa438, 0xd043, 0xa438, 0xf009, + 0xa438, 0xd121, 0xa438, 0xd043, 0xa438, 0xf006, 0xa438, 0xd122, + 0xa438, 0xd042, 0xa438, 0xf003, 0xa438, 0xd181, 0xa438, 0xd043, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd705, 0xa438, 0x60fe, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0ccf, 0xa438, 0x0b03, 0xa438, 0x8c07, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8cc0, 0xa438, 0x9503, + 0xa438, 0x8106, 0xa438, 0xd700, 0xa438, 0x60e7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d1a, 0xa438, 0x9503, + 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, + 0xa438, 0x0d19, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x2f99, + 0xa438, 0x919c, 0xa438, 0xa804, 0xa438, 0xd705, 0xa438, 0x607e, + 0xa438, 0xa302, 0xa438, 0xf00b, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa017, 0xa438, 0x9503, 0xa438, 0xd14f, 0xa438, 0xd043, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa480, 0xa438, 0xd700, + 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0x8644, + 0xa438, 0xf003, 0xa438, 0x0c44, 0xa438, 0x0604, 0xa438, 0xd701, + 0xa438, 0x40d0, 0xa438, 0xac40, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa750, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x6047, + 0xa438, 0xa682, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, + 0xa438, 0xc5aa, 0xa438, 0x9503, 0xa438, 0xab80, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5faf, 0xa438, 0xb920, + 0xa438, 0xcd52, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7fb4, 0xa438, 0x9920, 0xa438, 0xa00a, 0xa438, 0xa1a0, + 0xa438, 0xd705, 0xa438, 0x605e, 0xa438, 0xa302, 0xa438, 0xa480, + 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x8644, 0xa438, 0xf003, 0xa438, 0x0c44, 0xa438, 0x0604, + 0xa438, 0xa902, 0xa438, 0x8920, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa480, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9c2a, + 0xa438, 0xd707, 0xa438, 0x40ae, 0xa438, 0x0c18, 0xa438, 0x0a08, + 0xa438, 0x1000, 0xa438, 0x9c30, 0xa438, 0xcd60, 0xa438, 0xd101, + 0xa438, 0xd040, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa230, + 0xa438, 0x9503, 0xa438, 0xd703, 0xa438, 0x68d1, 0xa438, 0xcd62, + 0xa438, 0x1000, 0xa438, 0x9a89, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd707, 0xa438, 0x4058, 0xa438, 0xa901, 0xa438, 0xd700, + 0xa438, 0x62dc, 0xa438, 0xd71f, 0xa438, 0x628e, 0xa438, 0xd704, + 0xa438, 0x4067, 0xa438, 0x1800, 0xa438, 0x96ac, 0xa438, 0xd704, + 0xa438, 0x40ab, 0xa438, 0xd705, 0xa438, 0x607f, 0xa438, 0x1800, + 0xa438, 0x974b, 0xa438, 0xd704, 0xa438, 0x609f, 0xa438, 0xd705, + 0xa438, 0x405d, 0xa438, 0xf005, 0xa438, 0xd704, 0xa438, 0x5c96, + 0xa438, 0xd75f, 0xa438, 0x5c40, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xc5aa, 0xa438, 0x9503, 0xa438, 0xd705, 0xa438, 0x60de, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c38, 0xa438, 0x0d18, + 0xa438, 0x9503, 0xa438, 0xd702, 0xa438, 0x4357, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8f80, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf009, 0xa438, 0xce08, 0xa438, 0xf007, + 0xa438, 0xce08, 0xa438, 0xf005, 0xa438, 0xce08, 0xa438, 0xf003, + 0xa438, 0xce08, 0xa438, 0xf001, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xcd61, 0xa438, 0x1000, 0xa438, 0x9a89, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd707, 0xa438, 0x4058, 0xa438, 0xa901, + 0xa438, 0xd71f, 0xa438, 0x2e71, 0xa438, 0x925e, 0xa438, 0xd704, + 0xa438, 0x2739, 0xa438, 0x96ac, 0xa438, 0xd704, 0xa438, 0x40ab, + 0xa438, 0xd705, 0xa438, 0x607f, 0xa438, 0x1800, 0xa438, 0x974b, + 0xa438, 0xd704, 0xa438, 0x60bf, 0xa438, 0xd705, 0xa438, 0x407d, + 0xa438, 0x1800, 0xa438, 0x97da, 0xa438, 0xd704, 0xa438, 0x5cb6, + 0xa438, 0xd75f, 0xa438, 0x5c60, 0xa438, 0x1800, 0xa438, 0x997c, + 0xa438, 0xaa10, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8480, + 0xa438, 0x9503, 0xa438, 0x9904, 0xa438, 0xcd80, 0xa438, 0x1000, + 0xa438, 0x9c99, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8302, + 0xa438, 0x8480, 0xa438, 0x8646, 0xa438, 0x1000, 0xa438, 0x9d44, + 0xa438, 0xd707, 0xa438, 0x605d, 0xa438, 0x8320, 0xa438, 0x8c40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8750, 0xa438, 0x8340, + 0xa438, 0x8120, 0xa438, 0x8302, 0xa438, 0xa61c, 0xa438, 0x9503, + 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0xab20, 0xa438, 0x8b80, + 0xa438, 0x0c0c, 0xa438, 0x0808, 0xa438, 0xd702, 0xa438, 0x4191, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fa1, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8f02, 0xa438, 0x9503, + 0xa438, 0xd400, 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0xb910, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fae, + 0xa438, 0x9910, 0xa438, 0xd71f, 0xa438, 0x409e, 0xa438, 0xd1b8, + 0xa438, 0xd049, 0xa438, 0xf012, 0xa438, 0xd700, 0xa438, 0x2fa9, + 0xa438, 0x92a1, 0xa438, 0x6131, 0xa438, 0x33c9, 0xa438, 0x92a7, + 0xa438, 0xd15d, 0xa438, 0xd040, 0xa438, 0xf009, 0xa438, 0xd193, + 0xa438, 0xd040, 0xa438, 0xf006, 0xa438, 0xd1db, 0xa438, 0xd040, + 0xa438, 0xf003, 0xa438, 0xd16f, 0xa438, 0xd040, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa380, 0xa438, 0x9503, 0xa438, 0xd417, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd700, 0xa438, 0x4127, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8bc0, 0xa438, 0x8cc7, + 0xa438, 0x0c3f, 0xa438, 0x0d08, 0xa438, 0x9503, 0xa438, 0xf007, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8bc0, 0xa438, 0x8cc7, + 0xa438, 0xcd48, 0xa438, 0x9503, 0xa438, 0x0c06, 0xa438, 0x0102, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xc555, 0xa438, 0x9503, + 0xa438, 0xcd81, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd707, 0xa438, 0x40fd, 0xa438, 0xd193, + 0xa438, 0xd047, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xa810, 0xa438, 0x8902, 0xa438, 0xd707, + 0xa438, 0x407b, 0xa438, 0x8801, 0xa438, 0xa340, 0xa438, 0xd71f, + 0xa438, 0x3ffd, 0xa438, 0x9521, 0xa438, 0xcd82, 0xa438, 0xd704, + 0xa438, 0x407f, 0xa438, 0x1800, 0xa438, 0x9390, 0xa438, 0x8810, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa204, 0xa438, 0xa108, + 0xa438, 0x9503, 0xa438, 0xa604, 0xa438, 0xd131, 0xa438, 0xd046, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, + 0xa438, 0xd707, 0xa438, 0x5f75, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8108, 0xa438, 0x9503, 0xa438, 0x0c07, 0xa438, 0x0b02, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c03, 0xa438, 0x0200, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce0b, 0xa438, 0xf00a, 0xa438, 0xce0b, 0xa438, 0xf008, + 0xa438, 0xce0a, 0xa438, 0xf006, 0xa438, 0xce0a, 0xa438, 0xf004, + 0xa438, 0xce09, 0xa438, 0xf002, 0xa438, 0xce09, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xab08, 0xa438, 0xcdc0, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd704, 0xa438, 0x5fb7, 0xa438, 0x8b08, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8204, 0xa438, 0xa380, + 0xa438, 0x9503, 0xa438, 0xd414, 0xa438, 0xb308, 0xa438, 0xd202, + 0xa438, 0xb302, 0xa438, 0xb301, 0xa438, 0xcdc1, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fa1, 0xa438, 0xd400, + 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0xd71f, 0xa438, 0x6060, + 0xa438, 0xd704, 0xa438, 0x6074, 0xa438, 0x1800, 0xa438, 0x9390, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c91, 0xa438, 0x0080, + 0xa438, 0xa060, 0xa438, 0xa101, 0xa438, 0xa408, 0xa438, 0xa240, + 0xa438, 0x9503, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c26, + 0xa438, 0x0022, 0xa438, 0xa110, 0xa438, 0xa011, 0xa438, 0x9503, + 0xa438, 0xd17a, 0xa438, 0xd046, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c38, 0xa438, 0x0c18, 0xa438, 0x9503, 0xa438, 0xa318, + 0xa438, 0xd131, 0xa438, 0xd046, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c83, 0xa438, 0x0282, 0xa438, 0x9503, 0xa438, 0x0c07, + 0xa438, 0x0b02, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce11, 0xa438, 0xf00a, 0xa438, 0xce11, 0xa438, 0xf008, + 0xa438, 0xce10, 0xa438, 0xf006, 0xa438, 0xce10, 0xa438, 0xf004, + 0xa438, 0xce0f, 0xa438, 0xf002, 0xa438, 0xce0f, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa204, 0xa438, 0x9503, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xab08, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd704, 0xa438, 0x3ebc, 0xa438, 0x9370, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8284, 0xa438, 0x9503, 0xa438, 0x8b08, + 0xa438, 0xd704, 0xa438, 0x61b9, 0xa438, 0xd705, 0xa438, 0x407c, + 0xa438, 0x1800, 0xa438, 0x938e, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa402, 0xa438, 0x9503, 0xa438, 0x8310, 0xa438, 0xcdc3, + 0xa438, 0x1800, 0xa438, 0x9337, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8402, 0xa438, 0x9503, 0xa438, 0xcdc2, 0xa438, 0xf003, + 0xa438, 0x1000, 0xa438, 0x9c99, 0xa438, 0x8320, 0xa438, 0xd705, + 0xa438, 0x429e, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce02, 0xa438, 0xf01d, 0xa438, 0xce02, 0xa438, 0xf01b, + 0xa438, 0xce01, 0xa438, 0xf019, 0xa438, 0xce01, 0xa438, 0xf017, + 0xa438, 0xce01, 0xa438, 0xf015, 0xa438, 0xce01, 0xa438, 0xf013, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce06, + 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, 0xa438, 0xce06, + 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, 0xa438, 0xce06, + 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xd705, 0xa438, 0x61be, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b05, 0xa438, 0x0c38, 0xa438, 0x0c28, + 0xa438, 0x9503, 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0xa302, + 0xa438, 0xa4a0, 0xa438, 0xf012, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b06, 0xa438, 0x0c3f, 0xa438, 0x0c1b, + 0xa438, 0xd700, 0xa438, 0x6067, 0xa438, 0x0cc0, 0xa438, 0x0d80, + 0xa438, 0x9503, 0xa438, 0x8810, 0xa438, 0x800a, 0xa438, 0x8302, + 0xa438, 0xa8c0, 0xa438, 0x8420, 0xa438, 0xa480, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd704, 0xa438, 0x409d, 0xa438, 0x0ce3, + 0xa438, 0x0203, 0xa438, 0xf003, 0xa438, 0x0ce0, 0xa438, 0x02a0, + 0xa438, 0xa120, 0xa438, 0xd700, 0xa438, 0x4187, 0xa438, 0xd704, + 0xa438, 0x40dd, 0xa438, 0xd704, 0xa438, 0x617b, 0xa438, 0xd704, + 0xa438, 0x613c, 0xa438, 0xf00a, 0xa438, 0xd707, 0xa438, 0x411f, + 0xa438, 0x40e0, 0xa438, 0xf004, 0xa438, 0xd707, 0xa438, 0x409f, + 0xa438, 0x4062, 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa318, + 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x8604, 0xa438, 0xf002, 0xa438, 0xa604, 0xa438, 0xd700, + 0xa438, 0x6047, 0xa438, 0xa682, 0xa438, 0x1000, 0xa438, 0x9cc6, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, + 0xa438, 0xb920, 0xa438, 0xcd83, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x8810, 0xa438, 0x9920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x60e5, + 0xa438, 0x5f94, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, + 0xa438, 0x5fa7, 0xa438, 0xffef, 0xa438, 0xb820, 0xa438, 0xa810, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fa5, + 0xa438, 0x9820, 0xa438, 0xbb20, 0xa438, 0xd705, 0xa438, 0x605e, + 0xa438, 0xf018, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, + 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, + 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa00a, 0xa438, 0xa420, 0xa438, 0x88c0, + 0xa438, 0xd701, 0xa438, 0x40d0, 0xa438, 0xac40, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x9450, 0xa438, 0x33a9, 0xa438, 0x944d, + 0xa438, 0xd1b7, 0xa438, 0xd04b, 0xa438, 0xf006, 0xa438, 0xd1b7, + 0xa438, 0xd04b, 0xa438, 0xf003, 0xa438, 0xd1b7, 0xa438, 0xd04b, + 0xa438, 0xd707, 0xa438, 0x3ad1, 0xa438, 0x94d8, 0xa438, 0xd700, + 0xa438, 0x40c7, 0xa438, 0xd704, 0xa438, 0x409d, 0xa438, 0xd704, + 0xa438, 0x465c, 0xa438, 0xf004, 0xa438, 0xd707, 0xa438, 0x605f, + 0xa438, 0xf02e, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x946a, + 0xa438, 0x33a9, 0xa438, 0x9467, 0xa438, 0xd199, 0xa438, 0xd05a, + 0xa438, 0xf006, 0xa438, 0xd199, 0xa438, 0xd05a, 0xa438, 0xf003, + 0xa438, 0xd1d6, 0xa438, 0xd05a, 0xa438, 0xcd84, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7b, 0xa438, 0xd700, 0xa438, 0x42c7, 0xa438, 0xd704, + 0xa438, 0x429d, 0xa438, 0xd704, 0xa438, 0x425c, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd706, + 0xa438, 0x5f6f, 0xa438, 0x8a04, 0xa438, 0x8b20, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa380, 0xa438, 0x9503, 0xa438, 0xd401, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x1800, 0xa438, 0x94d8, + 0xa438, 0xd706, 0xa438, 0x69ab, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, + 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd700, 0xa438, 0x4167, + 0xa438, 0xd704, 0xa438, 0x40bd, 0xa438, 0xd704, 0xa438, 0x615b, + 0xa438, 0x1800, 0xa438, 0x94a5, 0xa438, 0xd707, 0xa438, 0x411f, + 0xa438, 0x40e0, 0xa438, 0xf004, 0xa438, 0xd707, 0xa438, 0x409f, + 0xa438, 0x4062, 0xa438, 0x8320, 0xa438, 0xf002, 0xa438, 0xa320, + 0xa438, 0x8310, 0xa438, 0xcd85, 0xa438, 0xd700, 0xa438, 0x37c9, + 0xa438, 0x94b3, 0xa438, 0x33a9, 0xa438, 0x94b0, 0xa438, 0xd1b7, + 0xa438, 0xd04a, 0xa438, 0xf006, 0xa438, 0xd1b7, 0xa438, 0xd04a, + 0xa438, 0xf003, 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd700, + 0xa438, 0x41a7, 0xa438, 0xd704, 0xa438, 0x40dd, 0xa438, 0xd704, + 0xa438, 0x605b, 0xa438, 0xf00d, 0xa438, 0x1800, 0xa438, 0x8f25, + 0xa438, 0xd707, 0xa438, 0x413f, 0xa438, 0x4100, 0xa438, 0x1800, + 0xa438, 0x8f25, 0xa438, 0xd707, 0xa438, 0x409f, 0xa438, 0x4062, + 0xa438, 0x1800, 0xa438, 0x8f25, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd701, 0xa438, 0x5f77, + 0xa438, 0x1800, 0xa438, 0x8f25, 0xa438, 0xd705, 0xa438, 0x413e, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, 0xa438, 0x0b03, + 0xa438, 0x0c38, 0xa438, 0x0c00, 0xa438, 0xa104, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa102, 0xa438, 0x9503, + 0xa438, 0xaa02, 0xa438, 0xa310, 0xa438, 0xcd86, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x94f4, 0xa438, 0x33a9, 0xa438, 0x94f1, + 0xa438, 0xd1d6, 0xa438, 0xd04a, 0xa438, 0xf006, 0xa438, 0xd1d6, + 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd199, 0xa438, 0xd04a, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, + 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0x0c30, 0xa438, 0x0320, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x9509, 0xa438, 0x33a9, + 0xa438, 0x9506, 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0xf006, + 0xa438, 0xd1b7, 0xa438, 0xd04a, 0xa438, 0xf003, 0xa438, 0xd1b7, + 0xa438, 0xd04a, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd701, + 0xa438, 0x5f37, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8102, + 0xa438, 0x8104, 0xa438, 0x9503, 0xa438, 0xd701, 0xa438, 0x40b0, + 0xa438, 0xaa02, 0xa438, 0xac80, 0xa438, 0x1800, 0xa438, 0x8f25, + 0xa438, 0x8a02, 0xa438, 0x1800, 0xa438, 0x8f25, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa708, 0xa438, 0x9503, 0xa438, 0xcd87, + 0xa438, 0xd704, 0xa438, 0x615f, 0xa438, 0x8810, 0xa438, 0xd702, + 0xa438, 0x40e6, 0xa438, 0xd17a, 0xa438, 0xd048, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce06, 0xa438, 0xf00a, + 0xa438, 0xce06, 0xa438, 0xf008, 0xa438, 0xce06, 0xa438, 0xf006, + 0xa438, 0xce06, 0xa438, 0xf004, 0xa438, 0xce06, 0xa438, 0xf002, + 0xa438, 0xce06, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xd707, + 0xa438, 0x413b, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b03, 0xa438, 0x0c38, 0xa438, 0x0c18, 0xa438, 0x9503, + 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b07, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x40c7, + 0xa438, 0xd704, 0xa438, 0x409d, 0xa438, 0x0ce3, 0xa438, 0x0203, + 0xa438, 0xf003, 0xa438, 0x0ce0, 0xa438, 0x02a0, 0xa438, 0xa120, + 0xa438, 0xd707, 0xa438, 0x425b, 0xa438, 0xd700, 0xa438, 0x4167, + 0xa438, 0xd704, 0xa438, 0x40dd, 0xa438, 0xd704, 0xa438, 0x613b, + 0xa438, 0xd704, 0xa438, 0x60fc, 0xa438, 0xf008, 0xa438, 0xd707, + 0xa438, 0x40df, 0xa438, 0xf003, 0xa438, 0xd707, 0xa438, 0x407f, + 0xa438, 0x8310, 0xa438, 0xf002, 0xa438, 0xa318, 0xa438, 0xa00a, + 0xa438, 0xa302, 0xa438, 0xa4a0, 0xa438, 0xd700, 0xa438, 0x40a7, + 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0x8604, 0xa438, 0xf002, + 0xa438, 0xa604, 0xa438, 0xd700, 0xa438, 0x6047, 0xa438, 0xa682, + 0xa438, 0x1000, 0xa438, 0x9cc6, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, 0xa438, 0xcd88, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, + 0xa438, 0x8810, 0xa438, 0x9920, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x60e5, 0xa438, 0x5f94, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xffef, + 0xa438, 0xb820, 0xa438, 0xa810, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0xbb20, + 0xa438, 0xd701, 0xa438, 0x40d0, 0xa438, 0xac40, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x95be, 0xa438, 0x33a9, 0xa438, 0x95b6, + 0xa438, 0xd702, 0xa438, 0x4086, 0xa438, 0xd161, 0xa438, 0xd049, + 0xa438, 0xf00e, 0xa438, 0xd19e, 0xa438, 0xd049, 0xa438, 0xf00b, + 0xa438, 0xd702, 0xa438, 0x4086, 0xa438, 0xd186, 0xa438, 0xd049, + 0xa438, 0xf006, 0xa438, 0xd1c3, 0xa438, 0xd049, 0xa438, 0xf003, + 0xa438, 0xd1cf, 0xa438, 0xd049, 0xa438, 0xcd89, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x5f2f, 0xa438, 0xd707, + 0xa438, 0x40fb, 0xa438, 0x8310, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c0f, 0xa438, 0x0b05, 0xa438, 0x9503, 0xa438, 0xcd8a, + 0xa438, 0xd707, 0xa438, 0x419b, 0xa438, 0xd700, 0xa438, 0x33ad, + 0xa438, 0x95df, 0xa438, 0xd189, 0xa438, 0xd045, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xcd8b, 0xa438, 0xd40a, 0xa438, 0x1000, + 0xa438, 0x9bbf, 0xa438, 0xbb10, 0xa438, 0xcd8c, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd71f, + 0xa438, 0x5f75, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8302, + 0xa438, 0x8480, 0xa438, 0xd700, 0xa438, 0x6047, 0xa438, 0xa680, + 0xa438, 0x8606, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8340, + 0xa438, 0x8750, 0xa438, 0x9503, 0xa438, 0x8c40, 0xa438, 0x9b30, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb5, + 0xa438, 0xcd8d, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x5f94, 0xa438, 0x8340, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8708, 0xa438, 0x9503, 0xa438, 0xa801, 0xa438, 0xd700, + 0xa438, 0x2fa9, 0xa438, 0x9610, 0xa438, 0x33c9, 0xa438, 0x9613, + 0xa438, 0x6117, 0xa438, 0xf00a, 0xa438, 0xd141, 0xa438, 0xd043, + 0xa438, 0xf009, 0xa438, 0xd121, 0xa438, 0xd043, 0xa438, 0xf006, + 0xa438, 0xd122, 0xa438, 0xd042, 0xa438, 0xf003, 0xa438, 0xd181, + 0xa438, 0xd043, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce06, 0xa438, 0xf00a, 0xa438, 0xce06, 0xa438, 0xf008, + 0xa438, 0xce06, 0xa438, 0xf006, 0xa438, 0xce06, 0xa438, 0xf004, + 0xa438, 0xce06, 0xa438, 0xf002, 0xa438, 0xce06, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c0f, + 0xa438, 0x0b02, 0xa438, 0x8cc7, 0xa438, 0x9503, 0xa438, 0xd700, + 0xa438, 0x40e7, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, + 0xa438, 0x0d09, 0xa438, 0x9503, 0xa438, 0xf006, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d08, 0xa438, 0x9503, + 0xa438, 0xa810, 0xa438, 0xa00a, 0xa438, 0x0ca0, 0xa438, 0x0120, + 0xa438, 0xa302, 0xa438, 0xa480, 0xa438, 0xd700, 0xa438, 0x40a7, + 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0x8604, 0xa438, 0xf002, + 0xa438, 0xa604, 0xa438, 0xd700, 0xa438, 0x6047, 0xa438, 0xa682, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, + 0xa438, 0xb920, 0xa438, 0xcd8e, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x8810, 0xa438, 0x9920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x60e5, + 0xa438, 0x5f94, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, + 0xa438, 0x5fa7, 0xa438, 0xffef, 0xa438, 0xb820, 0xa438, 0xa810, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fa5, + 0xa438, 0x9820, 0xa438, 0xbb20, 0xa438, 0xd701, 0xa438, 0x40d0, + 0xa438, 0xac40, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa750, + 0xa438, 0x9503, 0xa438, 0xd1c4, 0xa438, 0xd046, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x9c36, 0xa438, 0xd700, + 0xa438, 0x5f7a, 0xa438, 0xd706, 0xa438, 0x422f, 0xa438, 0x8c40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8750, 0xa438, 0x8380, + 0xa438, 0x9503, 0xa438, 0xd417, 0xa438, 0x1000, 0xa438, 0x9bbf, + 0xa438, 0xd701, 0xa438, 0x40d0, 0xa438, 0xac40, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, 0xa438, 0xd192, + 0xa438, 0xd047, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x9c36, 0xa438, 0xd700, 0xa438, 0x5f7a, 0xa438, 0xd706, + 0xa438, 0x5f2f, 0xa438, 0xd707, 0xa438, 0x409b, 0xa438, 0xd416, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xbb10, 0xa438, 0xcd8f, + 0xa438, 0x1800, 0xa438, 0x9120, 0xa438, 0x1000, 0xa438, 0x9c99, + 0xa438, 0x8302, 0xa438, 0x8646, 0xa438, 0xd700, 0xa438, 0x60c7, + 0xa438, 0xa680, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8702, + 0xa438, 0x9503, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa680, + 0xa438, 0x850f, 0xa438, 0x9503, 0xa438, 0xd701, 0xa438, 0x4050, + 0xa438, 0x8b20, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0ccf, + 0xa438, 0x0bc8, 0xa438, 0x0cc7, 0xa438, 0x0c44, 0xa438, 0x9503, + 0xa438, 0x0c06, 0xa438, 0x0102, 0xa438, 0xd700, 0xa438, 0x40e7, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d24, + 0xa438, 0x9503, 0xa438, 0xf005, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xcde4, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9cdd, + 0xa438, 0xcd90, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa601, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x2fa9, 0xa438, 0x96e1, + 0xa438, 0x60d1, 0xa438, 0x6113, 0xa438, 0x6157, 0xa438, 0xd150, + 0xa438, 0xd040, 0xa438, 0xf009, 0xa438, 0xd1a0, 0xa438, 0xd040, + 0xa438, 0xf006, 0xa438, 0xd128, 0xa438, 0xd040, 0xa438, 0xf003, + 0xa438, 0xd114, 0xa438, 0xd040, 0xa438, 0x1000, 0xa438, 0x99a8, + 0xa438, 0xd700, 0xa438, 0x5fba, 0xa438, 0xd704, 0xa438, 0x7f6b, + 0xa438, 0xa81a, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8603, + 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9d31, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8480, + 0xa438, 0x1000, 0xa438, 0x99a8, 0xa438, 0xd707, 0xa438, 0x4058, + 0xa438, 0xa901, 0xa438, 0xd704, 0xa438, 0x2b59, 0xa438, 0x974b, + 0xa438, 0xd704, 0xa438, 0x604c, 0xa438, 0xfff6, 0xa438, 0xd701, + 0xa438, 0x6056, 0xa438, 0xa704, 0xa438, 0xa302, 0xa438, 0xd700, + 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, 0xa438, 0x8604, + 0xa438, 0xf002, 0xa438, 0xa604, 0xa438, 0xd700, 0xa438, 0x60c7, + 0xa438, 0xa602, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa702, + 0xa438, 0x9503, 0xa438, 0xd701, 0xa438, 0x4055, 0xa438, 0xa940, + 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0xab20, 0xa438, 0xcd91, + 0xa438, 0x1000, 0xa438, 0x9cf4, 0xa438, 0x1000, 0xa438, 0x99a8, + 0xa438, 0xd704, 0xa438, 0x7f6c, 0xa438, 0xa120, 0xa438, 0xd707, + 0xa438, 0x4051, 0xa438, 0xa180, 0xa438, 0xa480, 0xa438, 0x8302, + 0xa438, 0x8606, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8702, + 0xa438, 0x9503, 0xa438, 0xd701, 0xa438, 0x6056, 0xa438, 0x8704, + 0xa438, 0x1000, 0xa438, 0x9cf4, 0xa438, 0xce93, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0x8b20, + 0xa438, 0xcd92, 0xa438, 0x1000, 0xa438, 0x99a8, 0xa438, 0xd707, + 0xa438, 0x7fb7, 0xa438, 0x81a0, 0xa438, 0x8480, 0xa438, 0x1000, + 0xa438, 0x9d31, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x1800, + 0xa438, 0x96fe, 0xa438, 0xd705, 0xa438, 0x40d9, 0xa438, 0xd701, + 0xa438, 0x6056, 0xa438, 0xa704, 0xa438, 0xa010, 0xa438, 0xf002, + 0xa438, 0x8010, 0xa438, 0xd701, 0xa438, 0x4095, 0xa438, 0xd705, + 0xa438, 0x4059, 0xa438, 0xa940, 0xa438, 0xd705, 0xa438, 0x6119, + 0xa438, 0x0c06, 0xa438, 0x0102, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c07, 0xa438, 0x0d04, 0xa438, 0x9503, 0xa438, 0xd701, + 0xa438, 0x4050, 0xa438, 0xab20, 0xa438, 0xcd93, 0xa438, 0x1000, + 0xa438, 0x99a8, 0xa438, 0xd704, 0xa438, 0x464b, 0xa438, 0xd707, + 0xa438, 0x5f78, 0xa438, 0x1000, 0xa438, 0x9d13, 0xa438, 0x800a, + 0xa438, 0x81a0, 0xa438, 0xd700, 0xa438, 0x6117, 0xa438, 0x6093, + 0xa438, 0xd15e, 0xa438, 0xd040, 0xa438, 0xf006, 0xa438, 0xd132, + 0xa438, 0xd040, 0xa438, 0xf003, 0xa438, 0xd119, 0xa438, 0xd040, + 0xa438, 0x1000, 0xa438, 0x9d08, 0xa438, 0x1000, 0xa438, 0x99a8, + 0xa438, 0xd73e, 0xa438, 0x6065, 0xa438, 0xd700, 0xa438, 0x5f3a, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x85f0, 0xa438, 0x9503, + 0xa438, 0xa008, 0xa438, 0xd707, 0xa438, 0x4052, 0xa438, 0xa002, + 0xa438, 0x8010, 0xa438, 0xd705, 0xa438, 0x4099, 0xa438, 0xd701, + 0xa438, 0x6056, 0xa438, 0x8704, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0x8b20, 0xa438, 0xcd94, + 0xa438, 0x1000, 0xa438, 0x99a8, 0xa438, 0xd707, 0xa438, 0x7fb8, + 0xa438, 0x8010, 0xa438, 0xd705, 0xa438, 0x4099, 0xa438, 0xd701, + 0xa438, 0x6056, 0xa438, 0x8704, 0xa438, 0xd705, 0xa438, 0x4099, + 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0x8b20, 0xa438, 0xd705, + 0xa438, 0x61f9, 0xa438, 0x8106, 0xa438, 0xd700, 0xa438, 0x60e7, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, + 0xa438, 0x9503, 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c07, 0xa438, 0x0d01, 0xa438, 0x9503, 0xa438, 0x800a, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0cf0, 0xa438, 0x05a0, + 0xa438, 0x9503, 0xa438, 0xd705, 0xa438, 0x4099, 0xa438, 0x1000, + 0xa438, 0x9d31, 0xa438, 0xf014, 0xa438, 0xa1a0, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00a, + 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, 0xa438, 0xf006, + 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, 0xa438, 0xf002, + 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xd705, + 0xa438, 0x39cc, 0xa438, 0x91f3, 0xa438, 0x1800, 0xa438, 0x96fe, + 0xa438, 0xd75e, 0xa438, 0x60f4, 0xa438, 0x9b08, 0xa438, 0x9920, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8403, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8208, 0xa438, 0x8404, + 0xa438, 0x9503, 0xa438, 0xcda1, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, + 0xa438, 0xd704, 0xa438, 0x5f37, 0xa438, 0x8b08, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8204, 0xa438, 0x9503, 0xa438, 0xd707, + 0xa438, 0x6536, 0xa438, 0x0c07, 0xa438, 0x0b00, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c03, 0xa438, 0x0200, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce0b, + 0xa438, 0xf00a, 0xa438, 0xce0b, 0xa438, 0xf008, 0xa438, 0xce0a, + 0xa438, 0xf006, 0xa438, 0xce0a, 0xa438, 0xf004, 0xa438, 0xce09, + 0xa438, 0xf002, 0xa438, 0xce09, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa204, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xab08, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd704, + 0xa438, 0x5f37, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8204, + 0xa438, 0x9503, 0xa438, 0x8b08, 0xa438, 0xd414, 0xa438, 0xd202, + 0xa438, 0xb308, 0xa438, 0xb302, 0xa438, 0xb301, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xaf02, 0xa438, 0x9503, 0xa438, 0xcdaf, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x99a0, + 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd701, 0xa438, 0x5f21, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8f02, 0xa438, 0x9503, + 0xa438, 0xd400, 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0xd704, + 0xa438, 0x67b4, 0xa438, 0x1000, 0xa438, 0x9cc6, 0xa438, 0xbb08, + 0xa438, 0x9a10, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, + 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, + 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa302, + 0xa438, 0xa480, 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, + 0xa438, 0x407d, 0xa438, 0x8644, 0xa438, 0xf003, 0xa438, 0x0c44, + 0xa438, 0x0604, 0xa438, 0xd700, 0xa438, 0x6047, 0xa438, 0xa682, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8230, 0xa438, 0x8410, + 0xa438, 0x9503, 0xa438, 0xd707, 0xa438, 0x404e, 0xa438, 0x8a10, + 0xa438, 0xcda9, 0xa438, 0xd110, 0xa438, 0xd040, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, + 0xa438, 0x99a4, 0xa438, 0xd700, 0xa438, 0x5f3a, 0xa438, 0x1800, + 0xa438, 0x91f3, 0xa438, 0xba10, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c97, 0xa438, 0x0086, 0xa438, 0xa060, 0xa438, 0xa101, + 0xa438, 0xa408, 0xa438, 0xa240, 0xa438, 0x9503, 0xa438, 0xcda2, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa020, 0xa438, 0xa110, + 0xa438, 0xa011, 0xa438, 0x9503, 0xa438, 0xd17a, 0xa438, 0xd046, + 0xa438, 0xcda3, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd700, + 0xa438, 0x5f3a, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c83, + 0xa438, 0x0282, 0xa438, 0x9503, 0xa438, 0x0c07, 0xa438, 0x0b02, + 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, + 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce11, + 0xa438, 0xf00a, 0xa438, 0xce11, 0xa438, 0xf008, 0xa438, 0xce10, + 0xa438, 0xf006, 0xa438, 0xce10, 0xa438, 0xf004, 0xa438, 0xce0f, + 0xa438, 0xf002, 0xa438, 0xce0f, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa204, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0xab08, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd704, + 0xa438, 0x3ebc, 0xa438, 0x98b7, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8284, 0xa438, 0x9503, 0xa438, 0x8b08, 0xa438, 0xcda4, + 0xa438, 0xd704, 0xa438, 0x6419, 0xa438, 0x3ad1, 0xa438, 0x9969, + 0xa438, 0xd705, 0xa438, 0x60dc, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa402, 0xa438, 0x9503, 0xa438, 0xf012, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8402, 0xa438, 0x9503, 0xa438, 0xd705, + 0xa438, 0x60db, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa401, + 0xa438, 0x9503, 0xa438, 0xf007, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c21, 0xa438, 0x0420, 0xa438, 0x9503, 0xa438, 0x9a10, + 0xa438, 0xd705, 0xa438, 0x34a4, 0xa438, 0x9886, 0xa438, 0x1800, + 0xa438, 0x9969, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c10, + 0xa438, 0x0010, 0xa438, 0x8410, 0xa438, 0xa250, 0xa438, 0x9503, + 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, 0xa438, 0x407d, + 0xa438, 0x8644, 0xa438, 0xf003, 0xa438, 0x0c44, 0xa438, 0x0604, + 0xa438, 0xd700, 0xa438, 0x40e7, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c38, 0xa438, 0x0d28, 0xa438, 0x9503, 0xa438, 0xf007, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0cf8, 0xa438, 0x0d90, + 0xa438, 0x9503, 0xa438, 0xa682, 0xa438, 0xd1f1, 0xa438, 0xd046, + 0xa438, 0xcda5, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd704, + 0xa438, 0x60ef, 0xa438, 0xd706, 0xa438, 0x40b1, 0xa438, 0xd700, + 0xa438, 0x5eba, 0xa438, 0x1800, 0xa438, 0x9925, 0xa438, 0xd162, + 0xa438, 0xd055, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0x1000, + 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, 0xa438, 0xd700, + 0xa438, 0x5f3b, 0xa438, 0xd704, 0xa438, 0x692f, 0xa438, 0xd706, + 0xa438, 0x48f1, 0xa438, 0x1800, 0xa438, 0x9907, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00a, + 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, 0xa438, 0xf006, + 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, 0xa438, 0xf002, + 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa00a, + 0xa438, 0xd179, 0xa438, 0xd047, 0xa438, 0xcda6, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, + 0xa438, 0x99a4, 0xa438, 0xd704, 0xa438, 0x60ef, 0xa438, 0xd706, + 0xa438, 0x40b1, 0xa438, 0xd700, 0xa438, 0x5eba, 0xa438, 0x1800, + 0xa438, 0x995b, 0xa438, 0xd162, 0xa438, 0xd055, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, + 0xa438, 0x99a4, 0xa438, 0xd700, 0xa438, 0x5f3b, 0xa438, 0xd704, + 0xa438, 0x626f, 0xa438, 0xd706, 0xa438, 0x4231, 0xa438, 0x1800, + 0xa438, 0x993d, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0ccf, + 0xa438, 0x0b46, 0xa438, 0x0c07, 0xa438, 0x0c03, 0xa438, 0x9503, + 0xa438, 0xa1a0, 0xa438, 0xa480, 0xa438, 0xb920, 0xa438, 0x9a10, + 0xa438, 0xcda7, 0xa438, 0x1800, 0xa438, 0x91f3, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8091, 0xa438, 0x8101, 0xa438, 0x8250, + 0xa438, 0x0c28, 0xa438, 0x0420, 0xa438, 0xd700, 0xa438, 0x4087, + 0xa438, 0x0c38, 0xa438, 0x0d18, 0xa438, 0xf003, 0xa438, 0x0cf8, + 0xa438, 0x0d58, 0xa438, 0x9503, 0xa438, 0x8106, 0xa438, 0xcda8, + 0xa438, 0x1800, 0xa438, 0x91f3, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8090, 0xa438, 0x8101, 0xa438, 0x8248, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9cb7, 0xa438, 0x1000, 0xa438, 0x9c90, + 0xa438, 0x8106, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0ccf, + 0xa438, 0x0b03, 0xa438, 0x8c07, 0xa438, 0xd700, 0xa438, 0x4087, + 0xa438, 0x0c38, 0xa438, 0x0d18, 0xa438, 0xf003, 0xa438, 0x0cf8, + 0xa438, 0x0d98, 0xa438, 0x8408, 0xa438, 0x9503, 0xa438, 0x9c01, + 0xa438, 0xa1a0, 0xa438, 0xa302, 0xa438, 0xd707, 0xa438, 0x404e, + 0xa438, 0x8a10, 0xa438, 0x1000, 0xa438, 0x9cc6, 0xa438, 0xcdab, + 0xa438, 0x1800, 0xa438, 0x923e, 0xa438, 0xd71f, 0xa438, 0x2e71, + 0xa438, 0x925e, 0xa438, 0x0800, 0xa438, 0xd704, 0xa438, 0x2739, + 0xa438, 0x96ac, 0xa438, 0x0800, 0xa438, 0xd71f, 0xa438, 0x62ae, + 0xa438, 0xd706, 0xa438, 0x6400, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd705, 0xa438, 0x61d9, 0xa438, 0xd704, 0xa438, 0x4187, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa601, 0xa438, 0x9503, + 0xa438, 0x800a, 0xa438, 0x1000, 0xa438, 0x9d31, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0x1800, 0xa438, 0x96ac, 0xa438, 0x0800, + 0xa438, 0xcd99, 0xa438, 0xa70c, 0xa438, 0x881a, 0xa438, 0x8010, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8683, 0xa438, 0x9503, + 0xa438, 0xd701, 0xa438, 0x4050, 0xa438, 0xab20, 0xa438, 0x1800, + 0xa438, 0x925e, 0xa438, 0x8010, 0xa438, 0xa704, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa60c, 0xa438, 0xa603, 0xa438, 0x9503, + 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8302, 0xa438, 0x8480, + 0xa438, 0x8604, 0xa438, 0x8602, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8702, 0xa438, 0x9503, 0xa438, 0xd701, 0xa438, 0x4050, + 0xa438, 0xab20, 0xa438, 0xcd96, 0xa438, 0xa901, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x7ce0, 0xa438, 0xd706, + 0xa438, 0x2109, 0xa438, 0x9a75, 0xa438, 0xd71f, 0xa438, 0x7aae, + 0xa438, 0xd704, 0xa438, 0x7ea7, 0xa438, 0xd706, 0xa438, 0x5e62, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8f40, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa610, 0xa438, 0x9503, + 0xa438, 0xa708, 0xa438, 0x881a, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0ccf, 0xa438, 0x0b03, 0xa438, 0x8cc7, 0xa438, 0x9503, + 0xa438, 0x8106, 0xa438, 0xd700, 0xa438, 0x60c7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xcd5a, 0xa438, 0x9503, 0xa438, 0xf006, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c3f, 0xa438, 0x0d19, + 0xa438, 0x9503, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa302, + 0xa438, 0xa480, 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, + 0xa438, 0x407d, 0xa438, 0x8604, 0xa438, 0xf002, 0xa438, 0xa604, + 0xa438, 0xd700, 0xa438, 0x60c7, 0xa438, 0xa602, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa702, 0xa438, 0x9503, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8680, 0xa438, 0x9503, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xc500, 0xa438, 0x9503, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x72ce, 0xa438, 0xd704, + 0xa438, 0x5f65, 0xa438, 0xd705, 0xa438, 0x4378, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce1d, 0xa438, 0xf00b, + 0xa438, 0xce1d, 0xa438, 0xf009, 0xa438, 0xce1d, 0xa438, 0xf007, + 0xa438, 0xce1d, 0xa438, 0xf005, 0xa438, 0xce1c, 0xa438, 0xf003, + 0xa438, 0xce1c, 0xa438, 0xf001, 0xa438, 0x1000, 0xa438, 0x9bde, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8640, 0xa438, 0x9503, + 0xa438, 0xf01a, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce05, 0xa438, 0xf00b, 0xa438, 0xce05, 0xa438, 0xf009, + 0xa438, 0xce04, 0xa438, 0xf007, 0xa438, 0xce04, 0xa438, 0xf005, + 0xa438, 0xce04, 0xa438, 0xf003, 0xa438, 0xce04, 0xa438, 0xf001, + 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa640, 0xa438, 0x9503, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8602, 0xa438, 0x8601, 0xa438, 0x9503, 0xa438, 0x8640, + 0xa438, 0x1000, 0xa438, 0x9cc6, 0xa438, 0xd703, 0xa438, 0x6131, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xaf80, 0xa438, 0x0c38, + 0xa438, 0x0d30, 0xa438, 0x9503, 0xa438, 0xd193, 0xa438, 0xd067, + 0xa438, 0xcd98, 0xa438, 0x1800, 0xa438, 0x91f3, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa608, 0xa438, 0x9503, 0xa438, 0xa708, + 0xa438, 0xd701, 0xa438, 0x6056, 0xa438, 0x8704, 0xa438, 0x1000, + 0xa438, 0x9d31, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa81a, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8603, 0xa438, 0x9503, + 0xa438, 0xcd97, 0xa438, 0x1800, 0xa438, 0x96fe, 0xa438, 0xd706, + 0xa438, 0x4103, 0xa438, 0xd705, 0xa438, 0x407f, 0xa438, 0x1800, + 0xa438, 0x9b77, 0xa438, 0xd105, 0xa438, 0xd056, 0xa438, 0xf03f, + 0xa438, 0xd705, 0xa438, 0x3fa7, 0xa438, 0x9a9a, 0xa438, 0xd705, + 0xa438, 0x605a, 0xa438, 0xf092, 0xa438, 0x1800, 0xa438, 0x9b1e, + 0xa438, 0xd704, 0xa438, 0x4146, 0xa438, 0x800a, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d04, 0xa438, 0x9503, + 0xa438, 0x0c06, 0xa438, 0x0102, 0xa438, 0xf010, 0xa438, 0xa00a, + 0xa438, 0x8106, 0xa438, 0xd700, 0xa438, 0x60e7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c07, 0xa438, 0x0d02, 0xa438, 0x9503, + 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c07, + 0xa438, 0x0d01, 0xa438, 0x9503, 0xa438, 0xd705, 0xa438, 0x61d4, + 0xa438, 0xd704, 0xa438, 0x609f, 0xa438, 0x6170, 0xa438, 0x2d71, + 0xa438, 0x9ac3, 0xa438, 0xd702, 0xa438, 0x60d7, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0cf0, 0xa438, 0x05a0, 0xa438, 0x9503, + 0xa438, 0xf0fc, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa108, + 0xa438, 0xa204, 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, + 0xa438, 0xd707, 0xa438, 0x5f35, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa410, 0xa438, 0x9503, 0xa438, 0x800a, 0xa438, 0x81a0, + 0xa438, 0x8302, 0xa438, 0x8480, 0xa438, 0x8642, 0xa438, 0x1000, + 0xa438, 0x9d44, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8108, + 0xa438, 0x9503, 0xa438, 0x8c40, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8750, 0xa438, 0x9503, 0xa438, 0xaa10, 0xa438, 0xd706, + 0xa438, 0x4063, 0xa438, 0x8604, 0xa438, 0xf022, 0xa438, 0xa604, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8220, 0xa438, 0x8301, + 0xa438, 0x8420, 0xa438, 0x8108, 0xa438, 0x0c03, 0xa438, 0x0201, + 0xa438, 0x9503, 0xa438, 0x0c07, 0xa438, 0x0b01, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce0e, 0xa438, 0xf00a, + 0xa438, 0xce0e, 0xa438, 0xf008, 0xa438, 0xce0d, 0xa438, 0xf006, + 0xa438, 0xce0d, 0xa438, 0xf004, 0xa438, 0xce0c, 0xa438, 0xf002, + 0xa438, 0xce0c, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xab08, + 0xa438, 0xcda0, 0xa438, 0xd706, 0xa438, 0x4043, 0xa438, 0xf0b1, + 0xa438, 0xd705, 0xa438, 0x411e, 0xa438, 0xd162, 0xa438, 0xd045, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa008, 0xa438, 0x9503, + 0xa438, 0xf0a8, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8008, + 0xa438, 0x9503, 0xa438, 0xd1c4, 0xa438, 0xd055, 0xa438, 0xf0a1, + 0xa438, 0xd700, 0xa438, 0x605a, 0xa438, 0xf09e, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8008, 0xa438, 0x8210, 0xa438, 0xa210, + 0xa438, 0x9503, 0xa438, 0xd162, 0xa438, 0xd055, 0xa438, 0xd704, + 0xa438, 0x608f, 0xa438, 0xd706, 0xa438, 0x4051, 0xa438, 0xf04a, + 0xa438, 0xd705, 0xa438, 0x415e, 0xa438, 0xd705, 0xa438, 0x6112, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c06, 0xa438, 0x0002, + 0xa438, 0xa110, 0xa438, 0xa305, 0xa438, 0x9503, 0xa438, 0xd705, + 0xa438, 0x40de, 0xa438, 0xd706, 0xa438, 0x6771, 0xa438, 0xd700, + 0xa438, 0x60bb, 0xa438, 0xf07f, 0xa438, 0xd700, 0xa438, 0x661b, + 0xa438, 0xf07c, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8090, + 0xa438, 0x8101, 0xa438, 0x0ccf, 0xa438, 0x0b03, 0xa438, 0x8c07, + 0xa438, 0xd700, 0xa438, 0x4087, 0xa438, 0x0c38, 0xa438, 0x0d18, + 0xa438, 0xf003, 0xa438, 0x0cf8, 0xa438, 0x0d58, 0xa438, 0x8270, + 0xa438, 0x8408, 0xa438, 0x8301, 0xa438, 0x9503, 0xa438, 0x8106, + 0xa438, 0x9c01, 0xa438, 0xd705, 0xa438, 0x40b5, 0xa438, 0x1000, + 0xa438, 0x9cb7, 0xa438, 0xd601, 0xa438, 0xd628, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8208, 0xa438, 0x9503, 0xa438, 0xcdac, + 0xa438, 0xd1c4, 0xa438, 0xd054, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0x1000, 0xa438, 0x99a0, 0xa438, 0x1000, 0xa438, 0x99a4, + 0xa438, 0xd700, 0xa438, 0x5f3b, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0xa230, 0xa438, 0x9503, 0xa438, 0xf04e, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa404, 0xa438, 0x8230, 0xa438, 0x9503, + 0xa438, 0xf048, 0xa438, 0xd700, 0xa438, 0x48db, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8204, 0xa438, 0x8410, 0xa438, 0x9503, + 0xa438, 0x8b08, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, + 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, + 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa480, + 0xa438, 0xd705, 0xa438, 0x611e, 0xa438, 0xa302, 0xa438, 0x1000, + 0xa438, 0x9cc6, 0xa438, 0xd707, 0xa438, 0x404e, 0xa438, 0x8a10, + 0xa438, 0xf006, 0xa438, 0x8302, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8301, 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x6047, + 0xa438, 0xa682, 0xa438, 0xd700, 0xa438, 0x40a7, 0xa438, 0xd704, + 0xa438, 0x407d, 0xa438, 0x8644, 0xa438, 0xf003, 0xa438, 0x0c44, + 0xa438, 0x0604, 0xa438, 0xd701, 0xa438, 0x40d0, 0xa438, 0xac40, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa750, 0xa438, 0x9503, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa220, 0xa438, 0x8210, + 0xa438, 0xa210, 0xa438, 0xa620, 0xa438, 0x9503, 0xa438, 0xcdaa, + 0xa438, 0x0800, 0xa438, 0xd202, 0xa438, 0xb309, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fa1, 0xa438, 0xd400, + 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0x0800, 0xa438, 0xd71f, + 0xa438, 0x6261, 0xa438, 0xcdff, 0xa438, 0xd705, 0xa438, 0x41d5, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c1f, 0xa438, 0x0e12, + 0xa438, 0x9503, 0xa438, 0xd70c, 0xa438, 0x5ff3, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x8e1f, 0xa438, 0x9503, 0xa438, 0xd70c, + 0xa438, 0x7f73, 0xa438, 0x1800, 0xa438, 0x001c, 0xa438, 0x0800, + 0xa438, 0xab10, 0xa438, 0xd71f, 0xa438, 0x210c, 0xa438, 0x001c, + 0xa438, 0xd701, 0xa438, 0x5f98, 0xa438, 0x8b10, 0xa438, 0x0800, + 0xa438, 0xd71f, 0xa438, 0x210c, 0xa438, 0x001c, 0xa438, 0xd701, + 0xa438, 0x5f99, 0xa438, 0x0800, 0xa438, 0xa110, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8110, 0xa438, 0xa140, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8140, 0xa438, 0xa004, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8004, 0xa438, 0xa001, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8001, 0xa438, 0xa020, 0xa438, 0x1000, + 0xa438, 0x9be6, 0xa438, 0x8020, 0xa438, 0xa101, 0xa438, 0x0800, + 0xa438, 0xd202, 0xa438, 0x0c09, 0xa438, 0x1301, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fa1, 0xa438, 0xb302, + 0xa438, 0xd200, 0xa438, 0x0800, 0xa438, 0xd204, 0xa438, 0x0c09, + 0xa438, 0x1301, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, + 0xa438, 0x5fa2, 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0x0800, + 0xa438, 0xd208, 0xa438, 0x0c09, 0xa438, 0x1301, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fa3, 0xa438, 0xb302, + 0xa438, 0xd200, 0xa438, 0x0800, 0xa438, 0xd210, 0xa438, 0x0c09, + 0xa438, 0x1301, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, + 0xa438, 0x5fa4, 0xa438, 0xb302, 0xa438, 0xd200, 0xa438, 0x0800, + 0xa438, 0xac01, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd707, + 0xa438, 0x5fad, 0xa438, 0x0800, 0xa438, 0xac04, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd707, 0xa438, 0x5fab, 0xa438, 0x0800, + 0xa438, 0xd71f, 0xa438, 0x4354, 0xa438, 0x8810, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5fa7, 0xa438, 0xb920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, + 0xa438, 0x9920, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x6085, 0xa438, 0xd71f, 0xa438, 0x7e34, 0xa438, 0xfffa, + 0xa438, 0xb820, 0xa438, 0xa810, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x7fa5, 0xa438, 0x9820, 0xa438, 0x0800, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa308, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, 0xa438, 0x5fb3, + 0xa438, 0x0800, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8308, + 0xa438, 0x9503, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd701, + 0xa438, 0x7fb3, 0xa438, 0x0800, 0xa438, 0xc000, 0xa438, 0xc100, + 0xa438, 0xc200, 0xa438, 0xc300, 0xa438, 0xc400, 0xa438, 0xc500, + 0xa438, 0xc600, 0xa438, 0xc700, 0xa438, 0xc828, 0xa438, 0xc904, + 0xa438, 0xca00, 0xa438, 0xcb00, 0xa438, 0xcc00, 0xa438, 0xce00, + 0xa438, 0xcf00, 0xa438, 0xd000, 0xa438, 0xd100, 0xa438, 0xd200, + 0xa438, 0xd300, 0xa438, 0xd400, 0xa438, 0xd500, 0xa438, 0xd700, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xc000, 0xa438, 0xc100, + 0xa438, 0xc200, 0xa438, 0xc300, 0xa438, 0xc400, 0xa438, 0xc500, + 0xa438, 0xc600, 0xa438, 0xa63c, 0xa438, 0xc700, 0xa438, 0xc800, + 0xa438, 0xc900, 0xa438, 0xca00, 0xa438, 0xcb00, 0xa438, 0xcc00, + 0xa438, 0xcd00, 0xa438, 0xce00, 0xa438, 0xcf00, 0xa438, 0x9503, + 0xa438, 0xcd00, 0xa438, 0x0800, 0xa438, 0xd601, 0xa438, 0xd608, + 0xa438, 0xd610, 0xa438, 0xd618, 0xa438, 0xd620, 0xa438, 0xd628, + 0xa438, 0xd630, 0xa438, 0xd638, 0xa438, 0x0800, 0xa438, 0x8b0f, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8097, 0xa438, 0x8109, + 0xa438, 0x82ff, 0xa438, 0x843f, 0xa438, 0x9503, 0xa438, 0x9920, + 0xa438, 0x9b08, 0xa438, 0x9a10, 0xa438, 0xd705, 0xa438, 0x4235, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c1f, 0xa438, 0x0e12, + 0xa438, 0x8440, 0xa438, 0x9503, 0xa438, 0xd70c, 0xa438, 0x5ff3, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8e1f, 0xa438, 0x9503, + 0xa438, 0xd70c, 0xa438, 0x7f73, 0xa438, 0xd601, 0xa438, 0xd628, + 0xa438, 0x0800, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c1f, + 0xa438, 0x0e12, 0xa438, 0x8440, 0xa438, 0x9503, 0xa438, 0xd70c, + 0xa438, 0x5ff3, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8e1f, + 0xa438, 0x9503, 0xa438, 0xd70c, 0xa438, 0x7f73, 0xa438, 0x0800, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x9ccf, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c70, 0xa438, 0x0350, 0xa438, 0x9503, + 0xa438, 0xf00e, 0xa438, 0xd702, 0xa438, 0x40f9, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0x0c70, 0xa438, 0x0340, 0xa438, 0x9503, + 0xa438, 0xf006, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c70, + 0xa438, 0x0340, 0xa438, 0x9503, 0xa438, 0x0800, 0xa438, 0xd700, + 0xa438, 0x37c9, 0xa438, 0x9ce6, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c70, 0xa438, 0x0370, 0xa438, 0x9503, 0xa438, 0xf00e, + 0xa438, 0xd702, 0xa438, 0x40f9, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x0c70, 0xa438, 0x0350, 0xa438, 0x9503, 0xa438, 0xf006, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x0c70, 0xa438, 0x0350, + 0xa438, 0x9503, 0xa438, 0x0800, 0xa438, 0x8910, 0xa438, 0xd704, + 0xa438, 0x61a8, 0xa438, 0xd702, 0xa438, 0x60d6, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xaf40, 0xa438, 0x9503, 0xa438, 0xf00a, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8f40, 0xa438, 0x9503, + 0xa438, 0xf005, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8f40, + 0xa438, 0x9503, 0xa438, 0x0800, 0xa438, 0x8910, 0xa438, 0xd705, + 0xa438, 0x4059, 0xa438, 0xa910, 0xa438, 0xd704, 0xa438, 0x6068, + 0xa438, 0x0000, 0xa438, 0xf002, 0xa438, 0x0000, 0xa438, 0x8910, + 0xa438, 0x0800, 0xa438, 0xd703, 0xa438, 0x6080, 0xa438, 0x6121, + 0xa438, 0x61c2, 0xa438, 0x6263, 0xa438, 0xd707, 0xa438, 0x4070, + 0xa438, 0xce98, 0xa438, 0xf015, 0xa438, 0xce94, 0xa438, 0xf013, + 0xa438, 0xd707, 0xa438, 0x4070, 0xa438, 0xce99, 0xa438, 0xf00f, + 0xa438, 0xce95, 0xa438, 0xf00d, 0xa438, 0xd707, 0xa438, 0x4070, + 0xa438, 0xce9a, 0xa438, 0xf009, 0xa438, 0xce96, 0xa438, 0xf007, + 0xa438, 0xd707, 0xa438, 0x4070, 0xa438, 0xce9b, 0xa438, 0xf003, + 0xa438, 0xce97, 0xa438, 0xf001, 0xa438, 0x0800, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce00, 0xa438, 0xf00a, + 0xa438, 0xce00, 0xa438, 0xf008, 0xa438, 0xce00, 0xa438, 0xf006, + 0xa438, 0xce00, 0xa438, 0xf004, 0xa438, 0xce00, 0xa438, 0xf002, + 0xa438, 0xce00, 0xa438, 0x0800, 0xa438, 0x0c03, 0xa438, 0x1502, + 0xa438, 0x8702, 0xa438, 0x9503, 0xa438, 0xd101, 0xa438, 0xd040, + 0xa438, 0xd700, 0xa438, 0x5ffa, 0xa438, 0x8680, 0xa438, 0xd700, + 0xa438, 0x60a7, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa702, + 0xa438, 0x9503, 0xa438, 0x0800, 0xa438, 0xcdfe, 0xa438, 0xa708, + 0xa438, 0xa2fc, 0xa438, 0xba20, 0xa438, 0xa980, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, + 0xa438, 0x9920, 0xa438, 0x0ca0, 0xa438, 0x0480, 0xa438, 0xd706, + 0xa438, 0x5fe7, 0xa438, 0x84a0, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, 0xa438, 0x9920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fa5, + 0xa438, 0xb820, 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, + 0xa438, 0x7f85, 0xa438, 0x9820, 0xa438, 0xa810, 0xa438, 0xd700, + 0xa438, 0x60cf, 0xa438, 0x60f1, 0xa438, 0x6113, 0xa438, 0x6135, + 0xa438, 0x6157, 0xa438, 0xf00b, 0xa438, 0xce08, 0xa438, 0xf00a, + 0xa438, 0xce08, 0xa438, 0xf008, 0xa438, 0xce08, 0xa438, 0xf006, + 0xa438, 0xce08, 0xa438, 0xf004, 0xa438, 0xce08, 0xa438, 0xf002, + 0xa438, 0xce08, 0xa438, 0x1000, 0xa438, 0x9bde, 0xa438, 0xa120, + 0xa438, 0xa4a0, 0xa438, 0xa980, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd706, 0xa438, 0x5faf, 0xa438, 0x1800, 0xa438, 0x9de8, + 0xa438, 0xcdfd, 0xa438, 0xd700, 0xa438, 0x60cf, 0xa438, 0x60f1, + 0xa438, 0x6113, 0xa438, 0x6135, 0xa438, 0x6157, 0xa438, 0xf00b, + 0xa438, 0xce08, 0xa438, 0xf00a, 0xa438, 0xce08, 0xa438, 0xf008, + 0xa438, 0xce08, 0xa438, 0xf006, 0xa438, 0xce08, 0xa438, 0xf004, + 0xa438, 0xce08, 0xa438, 0xf002, 0xa438, 0xce08, 0xa438, 0x1000, + 0xa438, 0x9bde, 0xa438, 0xa980, 0xa438, 0xa810, 0xa438, 0xa00a, + 0xa438, 0xa1a0, 0xa438, 0xa312, 0xa438, 0xa4a0, 0xa438, 0xa604, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa340, 0xa438, 0x9503, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd706, 0xa438, 0x5faf, + 0xa438, 0x0c30, 0xa438, 0x0320, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd704, 0xa438, 0x5fa4, 0xa438, 0x800a, 0xa438, 0x81a0, + 0xa438, 0x8312, 0xa438, 0x84a0, 0xa438, 0x8604, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x5fb4, 0xa438, 0xb920, + 0xa438, 0x1000, 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb4, + 0xa438, 0x9920, 0xa438, 0x8340, 0xa438, 0xa801, 0xa438, 0xa980, + 0xa438, 0x8240, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa302, + 0xa438, 0xa4a0, 0xa438, 0xa604, 0xa438, 0xd1c8, 0xa438, 0xd045, + 0xa438, 0xd700, 0xa438, 0x5ffa, 0xa438, 0xd706, 0xa438, 0x5faf, + 0xa438, 0xd40a, 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0xd416, + 0xa438, 0x1000, 0xa438, 0x9bbf, 0xa438, 0x800a, 0xa438, 0x81a0, + 0xa438, 0x8302, 0xa438, 0x8480, 0xa438, 0xa604, 0xa438, 0xa801, + 0xa438, 0x8818, 0xa438, 0xbb10, 0xa438, 0x1000, 0xa438, 0x9bc9, + 0xa438, 0xd71f, 0xa438, 0x5fb5, 0xa438, 0x9b10, 0xa438, 0x1000, + 0xa438, 0x9bc9, 0xa438, 0xd71f, 0xa438, 0x7fb5, 0xa438, 0x1800, + 0xa438, 0x915e, 0xa438, 0x1000, 0xa438, 0x1c45, 0xa438, 0x1000, + 0xa438, 0x9c64, 0xa438, 0xac10, 0xa438, 0x1800, 0xa438, 0x0021, + 0xa436, 0xA10E, 0xa438, 0xffff, 0xa436, 0xA10C, 0xa438, 0xffff, + 0xa436, 0xA10A, 0xa438, 0xffff, 0xa436, 0xA108, 0xa438, 0xffff, + 0xa436, 0xA106, 0xa438, 0xffff, 0xa436, 0xA104, 0xa438, 0xffff, + 0xa436, 0xA102, 0xa438, 0x001c, 0xa436, 0xA100, 0xa438, 0x0073, + 0xa436, 0xA110, 0xa438, 0x0003, 0xa436, 0xA016, 0xa438, 0x0020, + 0xa436, 0xA012, 0xa438, 0x1ff8, 0xa436, 0xA014, 0xa438, 0xb904, + 0xa438, 0xd18a, 0xa438, 0xd17a, 0xa438, 0x9c10, 0xa438, 0x0000, + 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, 0xa436, 0xA164, + 0xa438, 0x1CAC, 0xa436, 0xA166, 0xa438, 0x0666, 0xa436, 0xA168, + 0xa438, 0x0669, 0xa436, 0xA16A, 0xa438, 0x001f, 0xa436, 0xA16C, + 0xa438, 0x3fff, 0xa436, 0xA16E, 0xa438, 0x3fff, 0xa436, 0xA170, + 0xa438, 0x3fff, 0xa436, 0xA172, 0xa438, 0x3fff, 0xa436, 0xA162, + 0xa438, 0x000f, 0xa436, 0xb87c, 0xa438, 0x8a23, 0xa436, 0xb87e, + 0xa438, 0xaf8a, 0xa438, 0x3baf, 0xa438, 0x8a4c, 0xa438, 0xaf8a, + 0xa438, 0x5caf, 0xa438, 0x8a62, 0xa438, 0xaf8a, 0xa438, 0x68af, + 0xa438, 0x8a9e, 0xa438, 0xaf8b, 0xa438, 0x4daf, 0xa438, 0x8ba8, + 0xa438, 0xe080, 0xa438, 0x15ad, 0xa438, 0x2505, 0xa438, 0xd101, + 0xa438, 0xaf58, 0xa438, 0xcc02, 0xa438, 0x1e44, 0xa438, 0xaf59, + 0xa438, 0x05ee, 0xa438, 0x88af, 0xa438, 0x0102, 0xa438, 0x7eb6, + 0xa438, 0xad2a, 0xa438, 0x03af, 0xa438, 0x5b41, 0xa438, 0xaf5b, + 0xa438, 0x4a02, 0xa438, 0x8c4c, 0xa438, 0xaf69, 0xa438, 0x4f02, + 0xa438, 0x8d51, 0xa438, 0xaf65, 0xa438, 0xaf02, 0xa438, 0x7b73, + 0xa438, 0xbf7a, 0xa438, 0xa302, 0xa438, 0x7b73, 0xa438, 0xbf8e, + 0xa438, 0x5b02, 0xa438, 0x7b73, 0xa438, 0xbf8e, 0xa438, 0x5e02, + 0xa438, 0x7b73, 0xa438, 0xbf8e, 0xa438, 0x6102, 0xa438, 0x7b73, + 0xa438, 0xbf7a, 0xa438, 0xa602, 0xa438, 0x7b73, 0xa438, 0xbf8e, + 0xa438, 0x6402, 0xa438, 0x7b73, 0xa438, 0xbf8e, 0xa438, 0x6702, + 0xa438, 0x7b73, 0xa438, 0xbf8e, 0xa438, 0x6a02, 0xa438, 0x7b73, + 0xa438, 0xaf64, 0xa438, 0xcfe0, 0xa438, 0x898a, 0xa438, 0xe189, + 0xa438, 0x8bc4, 0xa438, 0xef67, 0xa438, 0xe089, 0xa438, 0x8ce1, + 0xa438, 0x898e, 0xa438, 0x1b01, 0xa438, 0xef20, 0xa438, 0xad27, + 0xa438, 0x04c7, 0xa438, 0x78ff, 0xa438, 0x109e, 0xa438, 0x050d, + 0xa438, 0x8180, 0xa438, 0xaef9, 0xa438, 0xac37, 0xa438, 0x09cf, + 0xa438, 0x027a, 0xa438, 0xcdac, 0xa438, 0x5030, 0xa438, 0xae13, + 0xa438, 0xe089, 0xa438, 0x8ae1, 0xa438, 0x898b, 0xa438, 0xef74, + 0xa438, 0xef46, 0xa438, 0xce02, 0xa438, 0x7acd, 0xa438, 0xef64, + 0xa438, 0xac50, 0xa438, 0x1be0, 0xa438, 0x8991, 0xa438, 0x3802, + 0xa438, 0xac27, 0xa438, 0x13ef, 0xa438, 0x46e4, 0xa438, 0x898a, + 0xa438, 0xe589, 0xa438, 0x8b3b, 0xa438, 0x02e7, 0xa438, 0x898d, + 0xa438, 0xe289, 0xa438, 0x8ee6, 0xa438, 0x898c, 0xa438, 0x1f44, + 0xa438, 0xe189, 0xa438, 0x68e2, 0xa438, 0x898e, 0xa438, 0x3a09, + 0xa438, 0xef32, 0xa438, 0xad37, 0xa438, 0x06c6, 0xa438, 0xef64, + 0xa438, 0x7aff, 0xa438, 0x129e, 0xa438, 0x050d, 0xa438, 0x6182, + 0xa438, 0xaef9, 0xa438, 0xac3f, 0xa438, 0x04ef, 0xa438, 0x74ae, + 0xa438, 0x03ef, 0xa438, 0x76ce, 0xa438, 0x027a, 0xa438, 0xcdad, + 0xa438, 0x501e, 0xa438, 0xe089, 0xa438, 0x9138, 0xa438, 0x02ac, + 0xa438, 0x2709, 0xa438, 0xe089, 0xa438, 0x8f10, 0xa438, 0xe489, + 0xa438, 0x8fae, 0xa438, 0x0de0, 0xa438, 0x8991, 0xa438, 0x2801, + 0xa438, 0xe189, 0xa438, 0x921e, 0xa438, 0x10e5, 0xa438, 0x8992, + 0xa438, 0xe089, 0xa438, 0x9110, 0xa438, 0xe489, 0xa438, 0x91af, + 0xa438, 0x68c5, 0xa438, 0xcffb, 0xa438, 0xee89, 0xa438, 0x6700, + 0xa438, 0xe389, 0xa438, 0x671f, 0xa438, 0x77d8, 0xa438, 0xa000, + 0xa438, 0x1213, 0xa438, 0xe789, 0xa438, 0x67d2, 0xa438, 0x031a, + 0xa438, 0x92e1, 0xa438, 0x8966, 0xa438, 0x1b13, 0xa438, 0x9fed, + 0xa438, 0xaf8b, 0xa438, 0x87a0, 0xa438, 0x040b, 0xa438, 0xbcbf, + 0xa438, 0x7ab2, 0xa438, 0x027d, 0xa438, 0x5abd, 0xa438, 0xac28, + 0xa438, 0xe0d8, 0xa438, 0x1f11, 0xa438, 0x0267, 0xa438, 0xc3d7, + 0xa438, 0x0001, 0xa438, 0xaed5, 0xa438, 0xad50, 0xa438, 0x02ae, + 0xa438, 0x12e0, 0xa438, 0x8987, 0xa438, 0xad20, 0xa438, 0x06ee, + 0xa438, 0x8987, 0xa438, 0x00ae, 0xa438, 0x06ee, 0xa438, 0x8989, + 0xa438, 0x00ae, 0xa438, 0x06e0, 0xa438, 0x895c, 0xa438, 0xe489, + 0xa438, 0x89ff, 0xa438, 0xaf6b, 0xa438, 0x3cd4, 0xa438, 0x000f, + 0xa438, 0xbf79, 0xa438, 0x3202, 0xa438, 0x7b9f, 0xa438, 0xd400, + 0xa438, 0x04bf, 0xa438, 0x5075, 0xa438, 0xd700, 0xa438, 0x0802, + 0xa438, 0x7da0, 0xa438, 0xbf8e, 0xa438, 0x6d02, 0xa438, 0x7b73, + 0xa438, 0xd402, 0xa438, 0x00bf, 0xa438, 0x8e70, 0xa438, 0x027b, + 0xa438, 0x9fd4, 0xa438, 0x000f, 0xa438, 0xbf8e, 0xa438, 0x7302, + 0xa438, 0x7b9f, 0xa438, 0xbf8e, 0xa438, 0x7302, 0xa438, 0x7b73, + 0xa438, 0xd41b, 0xa438, 0x3ebf, 0xa438, 0x8e6d, 0xa438, 0x027b, + 0xa438, 0x9fd4, 0xa438, 0x00dc, 0xa438, 0xbf8e, 0xa438, 0x7002, + 0xa438, 0x7b9f, 0xa438, 0xd400, 0xa438, 0x0fbf, 0xa438, 0x8e73, + 0xa438, 0x027b, 0xa438, 0x9fbf, 0xa438, 0x8e73, 0xa438, 0x027b, + 0xa438, 0x73d4, 0xa438, 0x1bbe, 0xa438, 0xbf8e, 0xa438, 0x6d02, + 0xa438, 0x7b9f, 0xa438, 0xd400, 0xa438, 0xdcbf, 0xa438, 0x8e70, + 0xa438, 0x027b, 0xa438, 0x9fd4, 0xa438, 0x000f, 0xa438, 0xbf8e, + 0xa438, 0x7302, 0xa438, 0x7b9f, 0xa438, 0xbf8e, 0xa438, 0x7302, + 0xa438, 0x7b73, 0xa438, 0xd700, 0xa438, 0x86b7, 0xa438, 0xfed4, + 0xa438, 0x003e, 0xa438, 0xbf8e, 0xa438, 0x6d02, 0xa438, 0x7b9f, + 0xa438, 0xd400, 0xa438, 0xdcbf, 0xa438, 0x8e70, 0xa438, 0x027b, + 0xa438, 0x9fd4, 0xa438, 0x000f, 0xa438, 0xbf8e, 0xa438, 0x7302, + 0xa438, 0x7b9f, 0xa438, 0xbf8e, 0xa438, 0x7302, 0xa438, 0x7b73, + 0xa438, 0xbf79, 0xa438, 0x3202, 0xa438, 0x7b73, 0xa438, 0xaf51, + 0xa438, 0x4ff8, 0xa438, 0xf9fa, 0xa438, 0xcefb, 0xa438, 0xef79, + 0xa438, 0xfbe0, 0xa438, 0x8967, 0xa438, 0xef10, 0xa438, 0x4803, + 0xa438, 0xbf89, 0xa438, 0x721a, 0xa438, 0x90ef, 0xa438, 0x79bf, + 0xa438, 0x8982, 0xa438, 0x1a91, 0xa438, 0xec01, 0xa438, 0x0702, + 0xa438, 0x7eb6, 0xa438, 0xad29, 0xa438, 0x10e0, 0xa438, 0x898f, + 0xa438, 0x3804, 0xa438, 0x9f06, 0xa438, 0xef97, 0xa438, 0x1919, + 0xa438, 0xec02, 0xa438, 0xaf8d, 0xa438, 0x48ef, 0xa438, 0x97ec, + 0xa438, 0x04e0, 0xa438, 0x898f, 0xa438, 0x380c, 0xa438, 0x9e3d, + 0xa438, 0xe089, 0xa438, 0x8d38, 0xa438, 0x03ac, 0xa438, 0x273d, + 0xa438, 0xe089, 0xa438, 0x8d38, 0xa438, 0x039e, 0xa438, 0x3f07, + 0xa438, 0xec03, 0xa438, 0x07e0, 0xa438, 0x898d, 0xa438, 0x3804, + 0xa438, 0x9e53, 0xa438, 0x19ec, 0xa438, 0x02e0, 0xa438, 0x898d, + 0xa438, 0x3807, 0xa438, 0xac27, 0xa438, 0x6089, 0xa438, 0xec03, + 0xa438, 0xe089, 0xa438, 0x8d38, 0xa438, 0x079e, 0xa438, 0x5807, + 0xa438, 0xec04, 0xa438, 0x07e0, 0xa438, 0x898d, 0xa438, 0x3808, + 0xa438, 0x9e61, 0xa438, 0xaf8d, 0xa438, 0x4819, 0xa438, 0xec03, + 0xa438, 0x19ec, 0xa438, 0x02ae, 0xa438, 0x7819, 0xa438, 0xec03, + 0xa438, 0x07ec, 0xa438, 0x0207, 0xa438, 0xae6f, 0xa438, 0xe089, + 0xa438, 0x5da0, 0xa438, 0x010e, 0xa438, 0xe089, 0xa438, 0x92a0, + 0xa438, 0x0308, 0xa438, 0x1f55, 0xa438, 0x07ec, 0xa438, 0x0307, + 0xa438, 0xae55, 0xa438, 0x07ec, 0xa438, 0x0207, 0xa438, 0xd103, + 0xa438, 0x026b, 0xa438, 0x74ae, 0xa438, 0x50e0, 0xa438, 0x895d, + 0xa438, 0xa001, 0xa438, 0x0be0, 0xa438, 0x8992, 0xa438, 0xad21, + 0xa438, 0x05d5, 0xa438, 0x010f, 0xa438, 0xae39, 0xa438, 0xd102, + 0xa438, 0x026b, 0xa438, 0x74ae, 0xa438, 0x38ae, 0xa438, 0x36e0, + 0xa438, 0x895d, 0xa438, 0xa001, 0xa438, 0x1fe0, 0xa438, 0x8992, + 0xa438, 0xa003, 0xa438, 0x191f, 0xa438, 0x5507, 0xa438, 0xec04, + 0xa438, 0x07ae, 0xa438, 0x17e0, 0xa438, 0x895d, 0xa438, 0xa001, + 0xa438, 0x0be0, 0xa438, 0x8992, 0xa438, 0xad21, 0xa438, 0x05d5, + 0xa438, 0x010f, 0xa438, 0xae06, 0xa438, 0xef97, 0xa438, 0xec00, + 0xa438, 0xae0b, 0xa438, 0x0266, 0xa438, 0xa7ae, 0xa438, 0x0602, + 0xa438, 0x66a7, 0xa438, 0x19ec, 0xa438, 0x02ff, 0xa438, 0xef97, + 0xa438, 0xffc6, 0xa438, 0xfefd, 0xa438, 0xfc04, 0xa438, 0xf8f9, + 0xa438, 0xfaef, 0xa438, 0x69fb, 0xa438, 0xcfd3, 0xa438, 0x00e7, + 0xa438, 0x8967, 0xa438, 0xbf89, 0xa438, 0x6e1a, 0xa438, 0x93bc, + 0xa438, 0xdaad, 0xa438, 0x302d, 0xa438, 0xbf76, 0xa438, 0x111f, + 0xa438, 0x4402, 0xa438, 0x7d66, 0xa438, 0x0266, 0xa438, 0xfcbf, + 0xa438, 0x8982, 0xa438, 0xe189, 0xa438, 0x67a1, 0xa438, 0x0001, + 0xa438, 0x191f, 0xa438, 0x44d9, 0xa438, 0x81bf, 0xa438, 0x7aa0, + 0xa438, 0x027d, 0xa438, 0x66bf, 0xa438, 0x7611, 0xa438, 0xd400, + 0xa438, 0x0102, 0xa438, 0x7d66, 0xa438, 0xbdae, 0xa438, 0x02ae, + 0xa438, 0xc9da, 0xa438, 0xad31, 0xa438, 0x31bf, 0xa438, 0x761d, + 0xa438, 0x1f44, 0xa438, 0x027d, 0xa438, 0x6602, 0xa438, 0x66fc, + 0xa438, 0xbf89, 0xa438, 0x8219, 0xa438, 0xe189, 0xa438, 0x67a1, + 0xa438, 0x0202, 0xa438, 0xae06, 0xa438, 0xa103, 0xa438, 0x02ae, + 0xa438, 0x0119, 0xa438, 0x1f44, 0xa438, 0xd981, 0xa438, 0xbf7a, + 0xa438, 0xa302, 0xa438, 0x7d66, 0xa438, 0xbf76, 0xa438, 0x1dd4, + 0xa438, 0x0001, 0xa438, 0x027d, 0xa438, 0x66bd, 0xa438, 0xdaad, + 0xa438, 0x322c, 0xa438, 0xbf76, 0xa438, 0x291f, 0xa438, 0x4402, + 0xa438, 0x7d66, 0xa438, 0x0266, 0xa438, 0xfcbf, 0xa438, 0x8982, + 0xa438, 0x1919, 0xa438, 0xe189, 0xa438, 0x67a1, 0xa438, 0x0302, + 0xa438, 0xae01, 0xa438, 0x191f, 0xa438, 0x44d9, 0xa438, 0x81bf, + 0xa438, 0x7aa6, 0xa438, 0x027d, 0xa438, 0x66bf, 0xa438, 0x7629, + 0xa438, 0xd400, 0xa438, 0x0102, 0xa438, 0x7d66, 0xa438, 0x028e, + 0xa438, 0x0f13, 0xa438, 0xe789, 0xa438, 0x67e0, 0xa438, 0x8966, + 0xa438, 0x1b03, 0xa438, 0x9f8b, 0xa438, 0xc7ff, 0xa438, 0xef96, + 0xa438, 0xfefd, 0xa438, 0xfc04, 0xa438, 0xf8f9, 0xa438, 0xfaef, + 0xa438, 0x69fb, 0xa438, 0xcfe3, 0xa438, 0x8967, 0xa438, 0xbf89, + 0xa438, 0x7e1a, 0xa438, 0x93bc, 0xa438, 0xda6a, 0xa438, 0x009e, + 0xa438, 0x2fbf, 0xa438, 0x7653, 0xa438, 0xe389, 0xa438, 0x671f, + 0xa438, 0x4402, 0xa438, 0x7d66, 0xa438, 0x0267, 0xa438, 0x1abd, + 0xa438, 0x1f00, 0xa438, 0xd959, 0xa438, 0x010c, 0xa438, 0x1269, + 0xa438, 0x08a3, 0xa438, 0x0104, 0xa438, 0x6902, 0xa438, 0xae02, + 0xa438, 0x6901, 0xa438, 0xbf7a, 0xa438, 0xb202, 0xa438, 0x7d66, + 0xa438, 0xd101, 0xa438, 0xbf76, 0xa438, 0x5302, 0xa438, 0x7d66, + 0xa438, 0xc7ff, 0xa438, 0xef96, 0xa438, 0xfefd, 0xa438, 0xfc04, + 0xa438, 0x32b1, 0xa438, 0x4a32, 0xa438, 0xb24a, 0xa438, 0x32b3, + 0xa438, 0x4a10, 0xa438, 0xb14a, 0xa438, 0x10b2, 0xa438, 0x4a10, + 0xa438, 0xb34a, 0xa438, 0xf0bd, 0xa438, 0x94f0, 0xa438, 0xbd92, + 0xa438, 0x74bd, 0xa438, 0x9600, 0xa436, 0xb85e, 0xa438, 0x58ca, + 0xa436, 0xb860, 0xa438, 0x5b3d, 0xa436, 0xb862, 0xa438, 0x694c, + 0xa436, 0xb864, 0xa438, 0x65ac, 0xa436, 0xb886, 0xa438, 0x64cc, + 0xa436, 0xb888, 0xa438, 0x683e, 0xa436, 0xb88a, 0xa438, 0x6ae6, + 0xa436, 0xb88c, 0xa438, 0x5119, 0xa436, 0xb838, 0xa438, 0x00ff, + 0xb820, 0x0010, 0xa464, 0x0001, 0xa436, 0x8474, 0xa438, 0x0000, + 0xa436, 0x8608, 0xa438, 0xaf86, 0xa438, 0x20af, 0xa438, 0x865e, + 0xa438, 0xaf86, 0xa438, 0xacaf, 0xa438, 0x86b7, 0xa438, 0xaf88, + 0xa438, 0xbdaf, 0xa438, 0x88d4, 0xa438, 0xaf88, 0xa438, 0xe3af, + 0xa438, 0x893f, 0xa438, 0xbf8a, 0xa438, 0x1102, 0xa438, 0x7589, + 0xa438, 0xbf8a, 0xa438, 0x1702, 0xa438, 0x756a, 0xa438, 0xbf8a, + 0xa438, 0x1d02, 0xa438, 0x756a, 0xa438, 0xbf8a, 0xa438, 0x1402, + 0xa438, 0x7589, 0xa438, 0xbf8a, 0xa438, 0x1a02, 0xa438, 0x756a, + 0xa438, 0xbf8a, 0xa438, 0x2002, 0xa438, 0x756a, 0xa438, 0xa200, + 0xa438, 0x08bf, 0xa438, 0x6e23, 0xa438, 0x0275, 0xa438, 0x3eae, + 0xa438, 0x0ca2, 0xa438, 0x0609, 0xa438, 0xe08f, 0xa438, 0x7dad, + 0xa438, 0x2003, 0xa438, 0xaf0f, 0xa438, 0xc9af, 0xa438, 0x0fd5, + 0xa438, 0xe084, 0xa438, 0x68a0, 0xa438, 0x0005, 0xa438, 0x0264, + 0xa438, 0xe6ae, 0xa438, 0x0ba0, 0xa438, 0x0105, 0xa438, 0x0265, + 0xa438, 0xb4ae, 0xa438, 0x0302, 0xa438, 0x8677, 0xa438, 0xaf64, + 0xa438, 0xe1f8, 0xa438, 0xf9e0, 0xa438, 0x8469, 0xa438, 0xe184, + 0xa438, 0x6a14, 0xa438, 0xe484, 0xa438, 0x69e5, 0xa438, 0x846a, + 0xa438, 0xe283, 0xa438, 0xade3, 0xa438, 0x83ae, 0xa438, 0x1b45, + 0xa438, 0x9f11, 0xa438, 0xee84, 0xa438, 0x6900, 0xa438, 0xee84, + 0xa438, 0x6a00, 0xa438, 0xee84, 0xa438, 0x6800, 0xa438, 0x0264, + 0xa438, 0xe6ae, 0xa438, 0x08e0, 0xa438, 0x8043, 0xa438, 0xf626, + 0xa438, 0xe480, 0xa438, 0x43fd, 0xa438, 0xfc04, 0xa438, 0xe080, + 0xa438, 0x43f6, 0xa438, 0x26e4, 0xa438, 0x8043, 0xa438, 0xaf65, + 0xa438, 0xa8ee, 0xa438, 0x8468, 0xa438, 0x0202, 0xa438, 0x86c9, + 0xa438, 0xe080, 0xa438, 0x43f6, 0xa438, 0x26e4, 0xa438, 0x8043, + 0xa438, 0xaf66, 0xa438, 0x0bf8, 0xa438, 0xf9ef, 0xa438, 0x59f9, + 0xa438, 0xfafb, 0xa438, 0xe18f, 0xa438, 0x7ea1, 0xa438, 0x0003, + 0xa438, 0xaf88, 0xa438, 0x64bf, 0xa438, 0x6c43, 0xa438, 0x0275, + 0xa438, 0x89d0, 0xa438, 0x003c, 0xa438, 0x008a, 0xa438, 0xad27, + 0xa438, 0x03af, 0xa438, 0x86fe, 0xa438, 0xee8f, 0xa438, 0x7e00, + 0xa438, 0xee84, 0xa438, 0x6d0f, 0xa438, 0xd401, 0xa438, 0x28d6, + 0xa438, 0x0010, 0xa438, 0x0288, 0xa438, 0x6caf, 0xa438, 0x8864, + 0xa438, 0xee84, 0xa438, 0x6d00, 0xa438, 0xd600, 0xa438, 0x8f02, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d01, 0xa438, 0xd600, + 0xa438, 0xc002, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d01, + 0xa438, 0xd600, 0xa438, 0x8f02, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0xc002, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0x8f02, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d04, 0xa438, 0xd600, + 0xa438, 0xc002, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d03, + 0xa438, 0xd600, 0xa438, 0x8f02, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d08, 0xa438, 0xd600, 0xa438, 0xc002, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d00, 0xa438, 0xd600, 0xa438, 0x9002, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d01, 0xa438, 0xd600, + 0xa438, 0xc102, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d01, + 0xa438, 0xd600, 0xa438, 0x9002, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0xc102, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0x9002, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d04, 0xa438, 0xd600, + 0xa438, 0xc102, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d03, + 0xa438, 0xd600, 0xa438, 0x9002, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d08, 0xa438, 0xd600, 0xa438, 0xc102, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d00, 0xa438, 0xd600, 0xa438, 0x9102, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d01, 0xa438, 0xd600, + 0xa438, 0xc202, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d01, + 0xa438, 0xd600, 0xa438, 0x9102, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0xc202, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d02, 0xa438, 0xd600, 0xa438, 0x9102, + 0xa438, 0x889b, 0xa438, 0xee84, 0xa438, 0x6d04, 0xa438, 0xd600, + 0xa438, 0xc202, 0xa438, 0x886c, 0xa438, 0xee84, 0xa438, 0x6d03, + 0xa438, 0xd600, 0xa438, 0x9102, 0xa438, 0x889b, 0xa438, 0xee84, + 0xa438, 0x6d08, 0xa438, 0xd600, 0xa438, 0xc202, 0xa438, 0x886c, + 0xa438, 0xee84, 0xa438, 0x6d0f, 0xa438, 0xd414, 0xa438, 0x00d6, + 0xa438, 0x000d, 0xa438, 0x0288, 0xa438, 0x6cbf, 0xa438, 0x8a2c, + 0xa438, 0x0275, 0xa438, 0x3ebf, 0xa438, 0x8a2f, 0xa438, 0x0275, + 0xa438, 0x3ebf, 0xa438, 0x8a32, 0xa438, 0x0275, 0xa438, 0x3ebf, + 0xa438, 0x8a35, 0xa438, 0x0275, 0xa438, 0x3ebf, 0xa438, 0x8a38, + 0xa438, 0x0275, 0xa438, 0x3ed4, 0xa438, 0x000f, 0xa438, 0xbf71, + 0xa438, 0x5c02, 0xa438, 0x756a, 0xa438, 0xbf71, 0xa438, 0x5f02, + 0xa438, 0x7589, 0xa438, 0xef31, 0xa438, 0xe783, 0xa438, 0x9fa3, + 0xa438, 0x0ff2, 0xa438, 0xd400, 0xa438, 0x00bf, 0xa438, 0x715c, + 0xa438, 0x0275, 0xa438, 0x6abf, 0xa438, 0x8a2c, 0xa438, 0x0275, + 0xa438, 0x47bf, 0xa438, 0x8a2f, 0xa438, 0x0275, 0xa438, 0x47bf, + 0xa438, 0x8a32, 0xa438, 0x0275, 0xa438, 0x47bf, 0xa438, 0x8a35, + 0xa438, 0x0275, 0xa438, 0x47bf, 0xa438, 0x8a38, 0xa438, 0x0275, + 0xa438, 0x47ee, 0xa438, 0x846d, 0xa438, 0x0fd4, 0xa438, 0x1000, + 0xa438, 0xd600, 0xa438, 0x0d02, 0xa438, 0x886c, 0xa438, 0xfffe, + 0xa438, 0xfdef, 0xa438, 0x95fd, 0xa438, 0xfc04, 0xa438, 0xf8f9, + 0xa438, 0xef59, 0xa438, 0xf9fa, 0xa438, 0xef44, 0xa438, 0xbf73, + 0xa438, 0xbf02, 0xa438, 0x756a, 0xa438, 0xef46, 0xa438, 0xbf73, + 0xa438, 0xc202, 0xa438, 0x756a, 0xa438, 0xe184, 0xa438, 0x6dbf, + 0xa438, 0x8a29, 0xa438, 0x0275, 0xa438, 0x6ad4, 0xa438, 0x0000, + 0xa438, 0xbf8a, 0xa438, 0x2902, 0xa438, 0x756a, 0xa438, 0xfefd, + 0xa438, 0xef95, 0xa438, 0xfdfc, 0xa438, 0x04f9, 0xa438, 0xef59, + 0xa438, 0xf9fa, 0xa438, 0xef46, 0xa438, 0xbf73, 0xa438, 0xc202, + 0xa438, 0x756a, 0xa438, 0xe184, 0xa438, 0x6dbf, 0xa438, 0x8a23, + 0xa438, 0x0275, 0xa438, 0x6abf, 0xa438, 0x8a26, 0xa438, 0x0275, + 0xa438, 0x89fe, 0xa438, 0xfdef, 0xa438, 0x95fd, 0xa438, 0x04e4, + 0xa438, 0x8044, 0xa438, 0xee84, 0xa438, 0x6d0f, 0xa438, 0xd401, + 0xa438, 0x20d6, 0xa438, 0x0010, 0xa438, 0x0288, 0xa438, 0x6cee, + 0xa438, 0x8f7e, 0xa438, 0x01af, 0xa438, 0x6879, 0xa438, 0xee84, + 0xa438, 0x6900, 0xa438, 0xee84, 0xa438, 0x6a00, 0xa438, 0xee8f, + 0xa438, 0x7e00, 0xa438, 0xaf64, 0xa438, 0x8702, 0xa438, 0x6a2a, + 0xa438, 0xe384, 0xa438, 0xf302, 0xa438, 0x6a4c, 0xa438, 0xac28, + 0xa438, 0x08e0, 0xa438, 0x84f6, 0xa438, 0xf722, 0xa438, 0xe484, + 0xa438, 0xf602, 0xa438, 0x6af9, 0xa438, 0xbf73, 0xa438, 0x4d02, + 0xa438, 0x753e, 0xa438, 0xbf73, 0xa438, 0x5002, 0xa438, 0x753e, + 0xa438, 0xbf73, 0xa438, 0x5302, 0xa438, 0x753e, 0xa438, 0xbf73, + 0xa438, 0x5602, 0xa438, 0x753e, 0xa438, 0xd500, 0xa438, 0x0002, + 0xa438, 0x6b1d, 0xa438, 0xbf73, 0xa438, 0x4402, 0xa438, 0x7547, + 0xa438, 0xbf73, 0xa438, 0x4702, 0xa438, 0x7547, 0xa438, 0xbf73, + 0xa438, 0x7702, 0xa438, 0x753e, 0xa438, 0xbf73, 0xa438, 0x4a02, + 0xa438, 0x7547, 0xa438, 0xbf73, 0xa438, 0x5302, 0xa438, 0x753e, + 0xa438, 0xbf73, 0xa438, 0x5602, 0xa438, 0x7547, 0xa438, 0xaf69, + 0xa438, 0x4d02, 0xa438, 0x8948, 0xa438, 0x021e, 0xa438, 0x40af, + 0xa438, 0x1e3f, 0xa438, 0xf8fa, 0xa438, 0xef69, 0xa438, 0xe080, + 0xa438, 0x4fac, 0xa438, 0x2417, 0xa438, 0xe080, 0xa438, 0x44ad, + 0xa438, 0x2417, 0xa438, 0x0289, 0xa438, 0x74e0, 0xa438, 0x8044, + 0xa438, 0xac24, 0xa438, 0x0ebf, 0xa438, 0x8a3b, 0xa438, 0x0275, + 0xa438, 0x47ae, 0xa438, 0x0602, 0xa438, 0x8a06, 0xa438, 0x0289, + 0xa438, 0xfbef, 0xa438, 0x96fe, 0xa438, 0xfc04, 0xa438, 0xf8f9, + 0xa438, 0xfaef, 0xa438, 0x69fa, 0xa438, 0xfbd2, 0xa438, 0x00a2, + 0xa438, 0x0403, 0xa438, 0xaf89, 0xa438, 0xeabf, 0xa438, 0x6af1, + 0xa438, 0x0277, 0xa438, 0x24ef, 0xa438, 0x010d, 0xa438, 0x11d0, + 0xa438, 0x00ef, 0xa438, 0x640c, 0xa438, 0x66ef, 0xa438, 0x12bf, + 0xa438, 0x8a23, 0xa438, 0x0275, 0xa438, 0x6ad3, 0xa438, 0x01a3, + 0xa438, 0x4302, 0xa438, 0xae44, 0xa438, 0x1f00, 0xa438, 0xef13, + 0xa438, 0xbf73, 0xa438, 0xc202, 0xa438, 0x756a, 0xa438, 0xbf8a, + 0xa438, 0x3e02, 0xa438, 0x7589, 0xa438, 0xd100, 0xa438, 0x0d01, + 0xa438, 0x0c01, 0xa438, 0x1a46, 0xa438, 0xbf73, 0xa438, 0xbf02, + 0xa438, 0x756a, 0xa438, 0x1f00, 0xa438, 0xef13, 0xa438, 0xbf73, + 0xa438, 0xc202, 0xa438, 0x756a, 0xa438, 0xd101, 0xa438, 0xef02, + 0xa438, 0x10b0, 0xa438, 0x02ae, 0xa438, 0x0449, 0xa438, 0x02ae, + 0xa438, 0xf8bf, 0xa438, 0x73c5, 0xa438, 0x0275, 0xa438, 0x6abf, + 0xa438, 0x73c5, 0xa438, 0x0275, 0xa438, 0x3e13, 0xa438, 0xaeb7, + 0xa438, 0x12af, 0xa438, 0x897d, 0xa438, 0xd500, 0xa438, 0x0102, + 0xa438, 0x6b1d, 0xa438, 0x0289, 0xa438, 0xfbff, 0xa438, 0xfeef, + 0xa438, 0x96fe, 0xa438, 0xfdfc, 0xa438, 0x04f8, 0xa438, 0xe080, + 0xa438, 0x44f6, 0xa438, 0x24e4, 0xa438, 0x8044, 0xa438, 0xfc04, + 0xa438, 0xf8e0, 0xa438, 0x804f, 0xa438, 0xf624, 0xa438, 0xe480, + 0xa438, 0x4ffc, 0xa438, 0x0455, 0xa438, 0xa6fe, 0xa438, 0x44a6, + 0xa438, 0xfe66, 0xa438, 0xa4b6, 0xa438, 0x55a4, 0xa438, 0xb666, + 0xa438, 0xac0e, 0xa438, 0x55ac, 0xa438, 0x0efe, 0xa438, 0xbda4, + 0xa438, 0xf0bd, 0xa438, 0x9830, 0xa438, 0xbd96, 0xa438, 0xffbd, + 0xa438, 0xdeee, 0xa438, 0xbdde, 0xa438, 0xddbd, 0xa438, 0xdebb, + 0xa438, 0xbdde, 0xa438, 0xaabd, 0xa438, 0xde44, 0xa438, 0xac00, + 0xa438, 0xf0bd, 0xa438, 0x9a00, 0xa436, 0xb818, 0xa438, 0x0fb9, + 0xa436, 0xb81a, 0xa438, 0x64c3, 0xa436, 0xb81c, 0xa438, 0x64f1, + 0xa436, 0xb81e, 0xa438, 0x6607, 0xa436, 0xb850, 0xa438, 0x6876, + 0xa436, 0xb852, 0xa438, 0x647f, 0xa436, 0xb878, 0xa438, 0x68e8, + 0xa436, 0xb884, 0xa438, 0x1e3c, 0xa436, 0xb832, 0xa438, 0x00df, + 0xB82E, 0x0000, 0xa436, 0x8023, 0xa438, 0x0000, 0xB820, 0x0000, + 0xFFFF, 0xFFFF +}; + +static const u16 phy_mcu_ram_code_8127a_1[] = { + 0xa436, 0x8023, 0xa438, 0x6100, 0xa436, 0xB82E, 0xa438, 0x0001, + 0xb820, 0x0090, 0xa436, 0xA016, 0xa438, 0x0000, 0xa436, 0xA012, + 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, 0xa438, 0x8010, + 0xa438, 0x1800, 0xa438, 0x801a, 0xa438, 0x1800, 0xa438, 0x801a, + 0xa438, 0x1800, 0xa438, 0x801a, 0xa438, 0x1800, 0xa438, 0x801a, + 0xa438, 0x1800, 0xa438, 0x801a, 0xa438, 0x1800, 0xa438, 0x801a, + 0xa438, 0x1800, 0xa438, 0x801a, 0xa438, 0xce00, 0xa438, 0x2941, + 0xa438, 0x8017, 0xa438, 0x2c59, 0xa438, 0x8017, 0xa438, 0x1800, + 0xa438, 0x0e11, 0xa438, 0x8aff, 0xa438, 0x1800, 0xa438, 0x0e11, + 0xa436, 0xA026, 0xa438, 0xffff, 0xa436, 0xA024, 0xa438, 0xffff, + 0xa436, 0xA022, 0xa438, 0xffff, 0xa436, 0xA020, 0xa438, 0xffff, + 0xa436, 0xA006, 0xa438, 0xffff, 0xa436, 0xA004, 0xa438, 0xffff, + 0xa436, 0xA002, 0xa438, 0xffff, 0xa436, 0xA000, 0xa438, 0x0e10, + 0xa436, 0xA008, 0xa438, 0x0100, 0xa436, 0xA016, 0xa438, 0x0000, + 0xa436, 0xA012, 0xa438, 0x0ff8, 0xa436, 0xA014, 0xa438, 0x219a, + 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, + 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, 0xa436, 0xA152, + 0xa438, 0x21a4, 0xa436, 0xA154, 0xa438, 0x3fff, 0xa436, 0xA156, + 0xa438, 0x3fff, 0xa436, 0xA158, 0xa438, 0x3fff, 0xa436, 0xA15A, + 0xa438, 0x3fff, 0xa436, 0xA15C, 0xa438, 0x3fff, 0xa436, 0xA15E, + 0xa438, 0x3fff, 0xa436, 0xA160, 0xa438, 0x3fff, 0xa436, 0xA150, + 0xa438, 0x0001, 0xa436, 0xA016, 0xa438, 0x0010, 0xa436, 0xA012, + 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, 0xa438, 0x8010, + 0xa438, 0x1800, 0xa438, 0x8014, 0xa438, 0x1800, 0xa438, 0x801a, + 0xa438, 0x1800, 0xa438, 0x801e, 0xa438, 0x1800, 0xa438, 0x8026, + 0xa438, 0x1800, 0xa438, 0x802e, 0xa438, 0x1800, 0xa438, 0x8036, + 0xa438, 0x1800, 0xa438, 0x803a, 0xa438, 0xce01, 0xa438, 0x8208, + 0xa438, 0x1800, 0xa438, 0x0028, 0xa438, 0x1000, 0xa438, 0x02c5, + 0xa438, 0x1000, 0xa438, 0x0304, 0xa438, 0x1800, 0xa438, 0x0119, + 0xa438, 0xce01, 0xa438, 0x8208, 0xa438, 0x1800, 0xa438, 0x009e, + 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0xa50f, 0xa438, 0x8208, + 0xa438, 0xd500, 0xa438, 0xaa0f, 0xa438, 0x1800, 0xa438, 0x015b, + 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0xa50f, 0xa438, 0x8208, + 0xa438, 0xd500, 0xa438, 0xaa0f, 0xa438, 0x1800, 0xa438, 0x01a9, + 0xa438, 0xd501, 0xa438, 0xce01, 0xa438, 0xa50f, 0xa438, 0x8208, + 0xa438, 0xd500, 0xa438, 0xaa0f, 0xa438, 0x1800, 0xa438, 0x01f4, + 0xa438, 0x8208, 0xa438, 0xd500, 0xa438, 0x1800, 0xa438, 0x02a5, + 0xa438, 0xa208, 0xa438, 0xd500, 0xa438, 0x1800, 0xa438, 0x02b8, + 0xa436, 0xA08E, 0xa438, 0x02b7, 0xa436, 0xA08C, 0xa438, 0x02a4, + 0xa436, 0xA08A, 0xa438, 0x01e7, 0xa436, 0xA088, 0xa438, 0x019c, + 0xa436, 0xA086, 0xa438, 0x014e, 0xa436, 0xA084, 0xa438, 0x009d, + 0xa436, 0xA082, 0xa438, 0x0117, 0xa436, 0xA080, 0xa438, 0x0027, + 0xa436, 0xA090, 0xa438, 0x00ff, 0xa436, 0xA016, 0xa438, 0x0020, + 0xa436, 0xA012, 0xa438, 0x0000, 0xa436, 0xA014, 0xa438, 0x1800, + 0xa438, 0x8010, 0xa438, 0x1800, 0xa438, 0x801d, 0xa438, 0x1800, + 0xa438, 0x803b, 0xa438, 0x1800, 0xa438, 0x8087, 0xa438, 0x1800, + 0xa438, 0x808e, 0xa438, 0x1800, 0xa438, 0x809d, 0xa438, 0x1800, + 0xa438, 0x80b7, 0xa438, 0x1800, 0xa438, 0x80c4, 0xa438, 0xd1bc, + 0xa438, 0xd040, 0xa438, 0x1000, 0xa438, 0x1cd2, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd700, 0xa438, 0x273d, 0xa438, 0x801b, + 0xa438, 0x1800, 0xa438, 0x07d1, 0xa438, 0x1800, 0xa438, 0x080e, + 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8032, 0xa438, 0x33a9, + 0xa438, 0x802a, 0xa438, 0xd705, 0xa438, 0x4084, 0xa438, 0xd1f4, + 0xa438, 0xd048, 0xa438, 0xf013, 0xa438, 0xd1b7, 0xa438, 0xd04b, + 0xa438, 0xf010, 0xa438, 0xd705, 0xa438, 0x4084, 0xa438, 0xd1f4, + 0xa438, 0xd048, 0xa438, 0xf00b, 0xa438, 0xd1b7, 0xa438, 0xd04b, + 0xa438, 0xf008, 0xa438, 0xd705, 0xa438, 0x4084, 0xa438, 0xd1f4, + 0xa438, 0xd048, 0xa438, 0xf003, 0xa438, 0xd1b7, 0xa438, 0xd04b, + 0xa438, 0x1800, 0xa438, 0x14cc, 0xa438, 0xd700, 0xa438, 0x2b59, + 0xa438, 0x803f, 0xa438, 0xf003, 0xa438, 0x1800, 0xa438, 0x118f, + 0xa438, 0x6060, 0xa438, 0x1800, 0xa438, 0x1167, 0xa438, 0xd700, + 0xa438, 0x60c7, 0xa438, 0xd704, 0xa438, 0x609f, 0xa438, 0xd705, + 0xa438, 0x4043, 0xa438, 0xf003, 0xa438, 0x1800, 0xa438, 0x1150, + 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0x8702, 0xa438, 0x8011, + 0xa438, 0x9503, 0xa438, 0x800a, 0xa438, 0x81a0, 0xa438, 0x8302, + 0xa438, 0x8480, 0xa438, 0x8686, 0xa438, 0xcde0, 0xa438, 0xd1ff, + 0xa438, 0xd049, 0xa438, 0x1000, 0xa438, 0x1cd2, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0xd705, 0xa438, 0x417e, 0xa438, 0x0c03, + 0xa438, 0x1502, 0xa438, 0xa011, 0xa438, 0x9503, 0xa438, 0xd1c8, + 0xa438, 0xd045, 0xa438, 0x1000, 0xa438, 0x1cd2, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x0c03, 0xa438, 0x1502, 0xa438, 0xa702, + 0xa438, 0x9503, 0xa438, 0xa00a, 0xa438, 0xa1a0, 0xa438, 0xa480, + 0xa438, 0xa686, 0xa438, 0xd705, 0xa438, 0x605e, 0xa438, 0xa302, + 0xa438, 0x9503, 0xa438, 0xd700, 0xa438, 0x37c9, 0xa438, 0x8083, + 0xa438, 0x33a9, 0xa438, 0x807f, 0xa438, 0xd178, 0xa438, 0xd04b, + 0xa438, 0x1800, 0xa438, 0x115d, 0xa438, 0xd1c8, 0xa438, 0xd04b, + 0xa438, 0x1800, 0xa438, 0x115d, 0xa438, 0xd1e6, 0xa438, 0xd04b, + 0xa438, 0x1800, 0xa438, 0x115d, 0xa438, 0xd71f, 0xa438, 0x6080, + 0xa438, 0xd704, 0xa438, 0x1800, 0xa438, 0x1bc0, 0xa438, 0x1800, + 0xa438, 0x1bc4, 0xa438, 0x4134, 0xa438, 0xd115, 0xa438, 0xd04f, + 0xa438, 0x1000, 0xa438, 0x1d0b, 0xa438, 0x1000, 0xa438, 0x80ad, + 0xa438, 0x1800, 0xa438, 0x01f2, 0xa438, 0x1000, 0xa438, 0x1d0b, + 0xa438, 0x1000, 0xa438, 0x80ad, 0xa438, 0x1800, 0xa438, 0x01f9, + 0xa438, 0x2969, 0xa438, 0x80a3, 0xa438, 0xd700, 0xa438, 0x606b, + 0xa438, 0xd701, 0xa438, 0x60b4, 0xa438, 0x1000, 0xa438, 0x80ad, + 0xa438, 0x1800, 0xa438, 0x0551, 0xa438, 0xd196, 0xa438, 0xd04d, + 0xa438, 0x1000, 0xa438, 0x80ad, 0xa438, 0x1800, 0xa438, 0x054d, + 0xa438, 0xd208, 0xa438, 0x0c09, 0xa438, 0x1301, 0xa438, 0x1000, + 0xa438, 0x1cd2, 0xa438, 0xd701, 0xa438, 0x5fa3, 0xa438, 0xb302, + 0xa438, 0xd200, 0xa438, 0x0800, 0xa438, 0xd705, 0xa438, 0x6064, + 0xa438, 0x1800, 0xa438, 0x140a, 0xa438, 0x8810, 0xa438, 0xd199, + 0xa438, 0xd04b, 0xa438, 0x1000, 0xa438, 0x1cd2, 0xa438, 0xd700, + 0xa438, 0x5fba, 0xa438, 0x1800, 0xa438, 0x140a, 0xa436, 0xA10E, + 0xa438, 0xffff, 0xa436, 0xA10C, 0xa438, 0x1352, 0xa436, 0xA10A, + 0xa438, 0x0545, 0xa436, 0xA108, 0xa438, 0x01ed, 0xa436, 0xA106, + 0xa438, 0x1bbf, 0xa436, 0xA104, 0xa438, 0x114b, 0xa436, 0xA102, + 0xa438, 0x14bf, 0xa436, 0xA100, 0xa438, 0x07ce, 0xa436, 0xA110, + 0xa438, 0x007f, 0xa436, 0xA016, 0xa438, 0x0020, 0xa436, 0xA012, + 0xa438, 0x1ff8, 0xa436, 0xA014, 0xa438, 0xd1ce, 0xa438, 0x0000, + 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, 0xa438, 0x0000, + 0xa438, 0x0000, 0xa438, 0x0000, 0xa436, 0xA164, 0xa438, 0x07fc, + 0xa436, 0xA166, 0xa438, 0x143d, 0xa436, 0xA168, 0xa438, 0x3fff, + 0xa436, 0xA16A, 0xa438, 0x3fff, 0xa436, 0xA16C, 0xa438, 0x3fff, + 0xa436, 0xA16E, 0xa438, 0x3fff, 0xa436, 0xA170, 0xa438, 0x3fff, + 0xa436, 0xA172, 0xa438, 0x3fff, 0xa436, 0xA162, 0xa438, 0x0003, + 0xa436, 0xb87c, 0xa438, 0x8994, 0xa436, 0xb87e, 0xa438, 0xaf89, + 0xa438, 0xacaf, 0xa438, 0x89e4, 0xa438, 0xaf89, 0xa438, 0xecaf, + 0xa438, 0x8a04, 0xa438, 0xaf8a, 0xa438, 0x2eaf, 0xa438, 0x8a4a, + 0xa438, 0xaf8d, 0xa438, 0x31af, 0xa438, 0x8dc6, 0xa438, 0x1f55, + 0xa438, 0xe18f, 0xa438, 0xe3a1, 0xa438, 0x0007, 0xa438, 0xee86, + 0xa438, 0xe900, 0xa438, 0xaf4f, 0xa438, 0x9ead, 0xa438, 0x281b, + 0xa438, 0xe18f, 0xa438, 0xfcef, 0xa438, 0x71bf, 0xa438, 0x74f6, + 0xa438, 0x027e, 0xa438, 0xd2ef, 0xa438, 0x641c, 0xa438, 0x670d, + 0xa438, 0x67ef, 0xa438, 0x461f, 0xa438, 0x00bf, 0xa438, 0x74f6, + 0xa438, 0x027e, 0xa438, 0xdee1, 0xa438, 0x8fe3, 0xa438, 0x0d11, + 0xa438, 0xe58f, 0xa438, 0xe313, 0xa438, 0xaeca, 0xa438, 0x028d, + 0xa438, 0xd1d3, 0xa438, 0x01af, 0xa438, 0x40d1, 0xa438, 0xbf7a, + 0xa438, 0x6102, 0xa438, 0x7d44, 0xa438, 0xa100, 0xa438, 0x09e0, + 0xa438, 0x8ffa, 0xa438, 0xe18f, 0xa438, 0xfbaf, 0xa438, 0x683d, + 0xa438, 0x027f, 0xa438, 0xa9af, 0xa438, 0x682c, 0xa438, 0xbf8e, + 0xa438, 0x4102, 0xa438, 0x7d44, 0xa438, 0xe58f, 0xa438, 0xecbf, + 0xa438, 0x74cc, 0xa438, 0x027d, 0xa438, 0x44e3, 0xa438, 0x8fed, + 0xa438, 0x0d31, 0xa438, 0xf63f, 0xa438, 0x0d11, 0xa438, 0xf62f, + 0xa438, 0x1b13, 0xa438, 0xad2f, 0xa438, 0x06bf, 0xa438, 0x8e41, + 0xa438, 0x027c, 0xa438, 0xf9d1, 0xa438, 0x01af, 0xa438, 0x5974, + 0xa438, 0xee88, 0xa438, 0x8600, 0xa438, 0xe08f, 0xa438, 0xebad, + 0xa438, 0x200b, 0xa438, 0xe18f, 0xa438, 0xecbf, 0xa438, 0x8e41, + 0xa438, 0x027d, 0xa438, 0x25ae, 0xa438, 0x04ee, 0xa438, 0x8feb, + 0xa438, 0x01af, 0xa438, 0x5945, 0xa438, 0xad28, 0xa438, 0x2ce0, + 0xa438, 0x8fea, 0xa438, 0xa000, 0xa438, 0x0502, 0xa438, 0x8af0, + 0xa438, 0xae1e, 0xa438, 0xa001, 0xa438, 0x0502, 0xa438, 0x8b9f, + 0xa438, 0xae16, 0xa438, 0xa002, 0xa438, 0x0502, 0xa438, 0x8c0f, + 0xa438, 0xae0e, 0xa438, 0xa003, 0xa438, 0x0502, 0xa438, 0x8c95, + 0xa438, 0xae06, 0xa438, 0xa004, 0xa438, 0x0302, 0xa438, 0x8d08, + 0xa438, 0xaf63, 0xa438, 0x8902, 0xa438, 0x8a7f, 0xa438, 0xaf63, + 0xa438, 0x81f8, 0xa438, 0xef49, 0xa438, 0xf8e0, 0xa438, 0x8015, + 0xa438, 0xad21, 0xa438, 0x19bf, 0xa438, 0x7bd8, 0xa438, 0x027c, + 0xa438, 0xf9bf, 0xa438, 0x7bf3, 0xa438, 0x027d, 0xa438, 0x44bf, + 0xa438, 0x7bf6, 0xa438, 0x027c, 0xa438, 0xf902, 0xa438, 0x638e, + 0xa438, 0xee8f, 0xa438, 0xea00, 0xa438, 0xe080, 0xa438, 0x16ad, + 0xa438, 0x233d, 0xa438, 0xbf7b, 0xa438, 0xf302, 0xa438, 0x7d44, + 0xa438, 0xbf7a, 0xa438, 0x9402, 0xa438, 0x7cf9, 0xa438, 0xbf8e, + 0xa438, 0x4402, 0xa438, 0x7cf9, 0xa438, 0xbf7a, 0xa438, 0xa602, + 0xa438, 0x7cf9, 0xa438, 0xbf7a, 0xa438, 0xa302, 0xa438, 0x7cf9, + 0xa438, 0xbf7a, 0xa438, 0xa902, 0xa438, 0x7cf9, 0xa438, 0xbf7a, + 0xa438, 0xac02, 0xa438, 0x7cf9, 0xa438, 0xbf8e, 0xa438, 0x4702, + 0xa438, 0x7cf9, 0xa438, 0xbf8e, 0xa438, 0x4a02, 0xa438, 0x7cf9, + 0xa438, 0x0263, 0xa438, 0x8eee, 0xa438, 0x8fea, 0xa438, 0x00bf, + 0xa438, 0x7c02, 0xa438, 0x027c, 0xa438, 0xf9fc, 0xa438, 0xef94, + 0xa438, 0xfc04, 0xa438, 0xf8f9, 0xa438, 0xfbef, 0xa438, 0x79fb, + 0xa438, 0xe080, 0xa438, 0x15ac, 0xa438, 0x2103, 0xa438, 0xaf8b, + 0xa438, 0x70ee, 0xa438, 0x8888, 0xa438, 0x00ee, 0xa438, 0x888a, + 0xa438, 0x00ee, 0xa438, 0x888b, 0xa438, 0x00bf, 0xa438, 0x7bd8, + 0xa438, 0x027d, 0xa438, 0x02bf, 0xa438, 0x6000, 0xa438, 0xd788, + 0xa438, 0x881f, 0xa438, 0x44d4, 0xa438, 0x000c, 0xa438, 0x0273, + 0xa438, 0x3b02, 0xa438, 0x7fa9, 0xa438, 0xac28, 0xa438, 0x05ac, + 0xa438, 0x290d, 0xa438, 0xae18, 0xa438, 0xe188, 0xa438, 0x98bf, + 0xa438, 0x7be1, 0xa438, 0x027d, 0xa438, 0x25ae, 0xa438, 0x18e1, + 0xa438, 0x8898, 0xa438, 0x0d11, 0xa438, 0xbf7b, 0xa438, 0xe102, + 0xa438, 0x7d25, 0xa438, 0xae0b, 0xa438, 0xe188, 0xa438, 0x980d, + 0xa438, 0x12bf, 0xa438, 0x7be1, 0xa438, 0x027d, 0xa438, 0x25bf, + 0xa438, 0x88a0, 0xa438, 0xda19, 0xa438, 0xdb19, 0xa438, 0xd819, + 0xa438, 0xd91f, 0xa438, 0x77bf, 0xa438, 0x88b1, 0xa438, 0xde19, + 0xa438, 0xdf19, 0xa438, 0xdc19, 0xa438, 0xdd19, 0xa438, 0x17a7, + 0xa438, 0x0004, 0xa438, 0xf302, 0xa438, 0x63cd, 0xa438, 0xee8f, + 0xa438, 0xea01, 0xa438, 0xe080, 0xa438, 0x16ad, 0xa438, 0x2319, + 0xa438, 0xee88, 0xa438, 0x8800, 0xa438, 0xee88, 0xa438, 0x8a00, + 0xa438, 0xee88, 0xa438, 0x8b00, 0xa438, 0xbf8e, 0xa438, 0x4402, + 0xa438, 0x7d02, 0xa438, 0x0263, 0xa438, 0xcdee, 0xa438, 0x8fea, + 0xa438, 0x0102, 0xa438, 0x70de, 0xa438, 0xbf7c, 0xa438, 0x0202, + 0xa438, 0x7d02, 0xa438, 0xffef, 0xa438, 0x97ff, 0xa438, 0xfdfc, + 0xa438, 0x04f8, 0xa438, 0xf9fa, 0xa438, 0xef69, 0xa438, 0xfae0, + 0xa438, 0x888a, 0xa438, 0xe188, 0xa438, 0x8b14, 0xa438, 0xe488, + 0xa438, 0x8ae5, 0xa438, 0x888b, 0xa438, 0xbf88, 0xa438, 0x94d8, + 0xa438, 0x19d9, 0xa438, 0xef64, 0xa438, 0xe088, 0xa438, 0x8ae1, + 0xa438, 0x888b, 0xa438, 0x1b46, 0xa438, 0x9f30, 0xa438, 0x1f44, + 0xa438, 0xe488, 0xa438, 0x8ae5, 0xa438, 0x888b, 0xa438, 0xe080, + 0xa438, 0x15ad, 0xa438, 0x211a, 0xa438, 0x0260, 0xa438, 0xece0, + 0xa438, 0x8016, 0xa438, 0xad23, 0xa438, 0x1602, 0xa438, 0x7c86, + 0xa438, 0xef47, 0xa438, 0xe48f, 0xa438, 0xe9e5, 0xa438, 0x8fe8, + 0xa438, 0xee8f, 0xa438, 0xea02, 0xa438, 0xae0b, 0xa438, 0x028c, + 0xa438, 0x2eae, 0xa438, 0x0602, 0xa438, 0x8bfe, 0xa438, 0x0270, + 0xa438, 0xdefe, 0xa438, 0xef96, 0xa438, 0xfefd, 0xa438, 0xfc04, + 0xa438, 0xf8e1, 0xa438, 0x8888, 0xa438, 0x11e5, 0xa438, 0x8888, + 0xa438, 0xad2a, 0xa438, 0x04ee, 0xa438, 0x8888, 0xa438, 0x00fc, + 0xa438, 0x04f8, 0xa438, 0xfafb, 0xa438, 0xe08f, 0xa438, 0xe9e1, + 0xa438, 0x8fe8, 0xa438, 0xef64, 0xa438, 0x1f00, 0xa438, 0xe18f, + 0xa438, 0xe6ef, 0xa438, 0x7402, 0xa438, 0x7ca1, 0xa438, 0xad50, + 0xa438, 0x0302, 0xa438, 0x8c2e, 0xa438, 0xfffe, 0xa438, 0xfc04, + 0xa438, 0xf8fa, 0xa438, 0xef69, 0xa438, 0xfbbf, 0xa438, 0x7bf3, + 0xa438, 0x027d, 0xa438, 0x44ac, 0xa438, 0x284c, 0xa438, 0x0264, + 0xa438, 0x1cbf, 0xa438, 0x8e47, 0xa438, 0x027d, 0xa438, 0x02bf, + 0xa438, 0x8e4a, 0xa438, 0x027d, 0xa438, 0x02d1, 0xa438, 0x43b1, + 0xa438, 0xfebf, 0xa438, 0x7aa6, 0xa438, 0x027c, 0xa438, 0xf9bf, + 0xa438, 0x7aa3, 0xa438, 0x027c, 0xa438, 0xf9bf, 0xa438, 0x7aa9, + 0xa438, 0x027c, 0xa438, 0xf9bf, 0xa438, 0x7aac, 0xa438, 0x027d, + 0xa438, 0x02d1, 0xa438, 0x80e0, 0xa438, 0x8888, 0xa438, 0x100e, + 0xa438, 0x11b0, 0xa438, 0xfcbf, 0xa438, 0x7a94, 0xa438, 0x027d, + 0xa438, 0x2502, 0xa438, 0x7c86, 0xa438, 0xef47, 0xa438, 0xe48f, + 0xa438, 0xe9e5, 0xa438, 0x8fe8, 0xa438, 0xee8f, 0xa438, 0xea03, + 0xa438, 0xae07, 0xa438, 0xee8f, 0xa438, 0xea01, 0xa438, 0x0270, + 0xa438, 0xdeff, 0xa438, 0xef96, 0xa438, 0xfefc, 0xa438, 0x04f8, + 0xa438, 0xf9fa, 0xa438, 0xfbef, 0xa438, 0x79fb, 0xa438, 0xbf7a, + 0xa438, 0x9402, 0xa438, 0x7d44, 0xa438, 0xef21, 0xa438, 0xbf7a, + 0xa438, 0xb802, 0xa438, 0x7d44, 0xa438, 0x1f21, 0xa438, 0x9e19, + 0xa438, 0xe08f, 0xa438, 0xe9e1, 0xa438, 0x8fe8, 0xa438, 0xef64, + 0xa438, 0x1f00, 0xa438, 0xe18f, 0xa438, 0xe4ef, 0xa438, 0x7402, + 0xa438, 0x7ca1, 0xa438, 0xad50, 0xa438, 0x3dee, 0xa438, 0x8fe7, + 0xa438, 0x01bf, 0xa438, 0x7a94, 0xa438, 0x027c, 0xa438, 0xf9bf, + 0xa438, 0x7aa6, 0xa438, 0x027c, 0xa438, 0xf9bf, 0xa438, 0x7aa3, + 0xa438, 0x027c, 0xa438, 0xf9bf, 0xa438, 0x7aa9, 0xa438, 0x027c, + 0xa438, 0xf9bf, 0xa438, 0x7aac, 0xa438, 0x027d, 0xa438, 0x02bf, + 0xa438, 0x8e47, 0xa438, 0x027c, 0xa438, 0xf9bf, 0xa438, 0x8e4a, + 0xa438, 0x027c, 0xa438, 0xf902, 0xa438, 0x7c86, 0xa438, 0xef47, + 0xa438, 0xe48f, 0xa438, 0xe9e5, 0xa438, 0x8fe8, 0xa438, 0xee8f, + 0xa438, 0xea04, 0xa438, 0xffef, 0xa438, 0x97ff, 0xa438, 0xfefd, + 0xa438, 0xfc04, 0xa438, 0xf8fa, 0xa438, 0xfbe0, 0xa438, 0x8fe9, + 0xa438, 0xe18f, 0xa438, 0xe8ef, 0xa438, 0x641f, 0xa438, 0x00e1, + 0xa438, 0x8fe5, 0xa438, 0xef74, 0xa438, 0x027c, 0xa438, 0xa1ad, + 0xa438, 0x500d, 0xa438, 0x0263, 0xa438, 0x8e02, 0xa438, 0x8bfe, + 0xa438, 0xee8f, 0xa438, 0xea01, 0xa438, 0x0270, 0xa438, 0xdeff, + 0xa438, 0xfefc, 0xa438, 0x04e3, 0xa438, 0x8fd8, 0xa438, 0xe787, + 0xa438, 0x75e4, 0xa438, 0x8fe1, 0xa438, 0xe58f, 0xa438, 0xe2bf, + 0xa438, 0x8fd9, 0xa438, 0xef32, 0xa438, 0x0c31, 0xa438, 0x1a93, + 0xa438, 0xdc19, 0xa438, 0xdd02, 0xa438, 0x7fa9, 0xa438, 0xac2a, + 0xa438, 0x18e0, 0xa438, 0x8fe1, 0xa438, 0xe18f, 0xa438, 0xe2ef, + 0xa438, 0x74e1, 0xa438, 0x8775, 0xa438, 0x1f00, 0xa438, 0xef64, + 0xa438, 0xe18f, 0xa438, 0xd8e5, 0xa438, 0x8775, 0xa438, 0xaf4d, + 0xa438, 0x72bf, 0xa438, 0x7b3c, 0xa438, 0xef32, 0xa438, 0x4b03, + 0xa438, 0x1a93, 0xa438, 0x027d, 0xa438, 0x44ef, 0xa438, 0x64e1, + 0xa438, 0x8fff, 0xa438, 0x1f00, 0xa438, 0xef74, 0xa438, 0x1b67, + 0xa438, 0xac4f, 0xa438, 0xcee0, 0xa438, 0x8ffd, 0xa438, 0xe18f, + 0xa438, 0xfeef, 0xa438, 0x64e0, 0xa438, 0x8fe1, 0xa438, 0xe18f, + 0xa438, 0xe2ef, 0xa438, 0x7402, 0xa438, 0x7c53, 0xa438, 0xac50, + 0xa438, 0x02ae, 0xa438, 0xb6e1, 0xa438, 0x8775, 0xa438, 0x1f00, + 0xa438, 0xef64, 0xa438, 0xe18f, 0xa438, 0xfcef, 0xa438, 0x711c, + 0xa438, 0x670d, 0xa438, 0x67ef, 0xa438, 0x46e5, 0xa438, 0x8775, + 0xa438, 0xef32, 0xa438, 0xd101, 0xa438, 0xa300, 0xa438, 0x02ae, + 0xa438, 0x050c, 0xa438, 0x1183, 0xa438, 0xaef6, 0xa438, 0xe08f, + 0xa438, 0xe31e, 0xa438, 0x10e5, 0xa438, 0x8fe3, 0xa438, 0xae89, + 0xa438, 0xe287, 0xa438, 0x75e6, 0xa438, 0x8fd8, 0xa438, 0x1f22, + 0xa438, 0xaf4d, 0xa438, 0x42f8, 0xa438, 0xf9ef, 0xa438, 0x59fa, + 0xa438, 0xfbbf, 0xa438, 0x8fee, 0xa438, 0x027f, 0xa438, 0xa90d, + 0xa438, 0x1149, 0xa438, 0x041a, 0xa438, 0x91d7, 0xa438, 0x8df3, + 0xa438, 0xd68e, 0xa438, 0x2302, 0xa438, 0x72aa, 0xa438, 0xfffe, + 0xa438, 0xef95, 0xa438, 0xfdfc, 0xa438, 0x0400, 0xa438, 0x7591, + 0xa438, 0x0275, 0xa438, 0x4404, 0xa438, 0x758e, 0xa438, 0x2675, + 0xa438, 0x4100, 0xa438, 0x8e26, 0xa438, 0x028e, 0xa438, 0x2304, + 0xa438, 0x759d, 0xa438, 0x2675, 0xa438, 0x4700, 0xa438, 0x8e32, + 0xa438, 0x028e, 0xa438, 0x2f04, 0xa438, 0x8e2c, 0xa438, 0x268e, + 0xa438, 0x2900, 0xa438, 0x8e3e, 0xa438, 0x028e, 0xa438, 0x3b04, + 0xa438, 0x8e38, 0xa438, 0x268e, 0xa438, 0x35fe, 0xa438, 0xad96, + 0xa438, 0xdcad, 0xa438, 0x96ba, 0xa438, 0xad96, 0xa438, 0x98ad, + 0xa438, 0x9676, 0xa438, 0xad98, 0xa438, 0x54ad, 0xa438, 0x9876, + 0xa438, 0xae38, 0xa438, 0x54ae, 0xa438, 0x38fe, 0xa438, 0xae3a, + 0xa438, 0xdcae, 0xa438, 0x3abb, 0xa438, 0xbf14, 0xa438, 0x99bd, + 0xa438, 0xe0cc, 0xa438, 0xbdc8, 0xa438, 0xddbd, 0xa438, 0xc800, + 0xa436, 0xb85e, 0xa438, 0x4f9a, 0xa436, 0xb860, 0xa438, 0x40cf, + 0xa436, 0xb862, 0xa438, 0x6829, 0xa436, 0xb864, 0xa438, 0x5972, + 0xa436, 0xb886, 0xa438, 0x5941, 0xa436, 0xb888, 0xa438, 0x636b, + 0xa436, 0xb88a, 0xa438, 0x4d6b, 0xa436, 0xb88c, 0xa438, 0x4d40, + 0xa436, 0xb838, 0xa438, 0x00ff, 0xb820, 0x0010, 0xa436, 0x8608, + 0xa438, 0xaf86, 0xa438, 0xdaaf, 0xa438, 0x894c, 0xa438, 0xaf8a, + 0xa438, 0xf8af, 0xa438, 0x8bf3, 0xa438, 0xaf8b, 0xa438, 0xf3af, + 0xa438, 0x8bf3, 0xa438, 0xaf8b, 0xa438, 0xf3af, 0xa438, 0x8bf3, + 0xa438, 0x006f, 0xa438, 0x4a03, 0xa438, 0x6f47, 0xa438, 0x266f, + 0xa438, 0x5900, 0xa438, 0x6f4d, 0xa438, 0x016f, 0xa438, 0x5004, + 0xa438, 0x6f56, 0xa438, 0x056f, 0xa438, 0x5f06, 0xa438, 0x6f5c, + 0xa438, 0x2774, 0xa438, 0x7800, 0xa438, 0x6f68, 0xa438, 0x246f, + 0xa438, 0x6b20, 0xa438, 0x6f6e, 0xa438, 0x206f, 0xa438, 0x7410, + 0xa438, 0x7469, 0xa438, 0x1074, 0xa438, 0x6c10, 0xa438, 0x746f, + 0xa438, 0x1074, 0xa438, 0x7225, 0xa438, 0x8bfc, 0xa438, 0x008c, + 0xa438, 0x0802, 0xa438, 0x8c02, 0xa438, 0x038b, 0xa438, 0xff04, + 0xa438, 0x6eed, 0xa438, 0x278c, 0xa438, 0x0520, 0xa438, 0x74da, + 0xa438, 0x2074, 0xa438, 0xdd20, 0xa438, 0x74e0, 0xa438, 0x0074, + 0xa438, 0xe300, 0xa438, 0x6ef3, 0xa438, 0x006e, 0xa438, 0xf600, + 0xa438, 0x6ef9, 0xa438, 0x006e, 0xa438, 0xfc00, 0xa438, 0x6eff, + 0xa438, 0x006f, 0xa438, 0x0200, 0xa438, 0x6f05, 0xa438, 0x026f, + 0xa438, 0x0802, 0xa438, 0x6f0b, 0xa438, 0x026f, 0xa438, 0x0e02, + 0xa438, 0x6f11, 0xa438, 0x026f, 0xa438, 0x1402, 0xa438, 0x6f17, + 0xa438, 0x226f, 0xa438, 0x1a00, 0xa438, 0x723e, 0xa438, 0x016e, + 0xa438, 0xed24, 0xa438, 0x6f50, 0xa438, 0x0072, 0xa438, 0x4701, + 0xa438, 0x724a, 0xa438, 0x0272, 0xa438, 0x4d23, 0xa438, 0x7250, + 0xa438, 0x1074, 0xa438, 0x6910, 0xa438, 0x746c, 0xa438, 0x1074, + 0xa438, 0x6f00, 0xa438, 0x7472, 0xa438, 0x158c, 0xa438, 0x0b15, + 0xa438, 0x8c0e, 0xa438, 0x158c, 0xa438, 0x1105, 0xa438, 0x8c14, + 0xa438, 0x006f, 0xa438, 0x4a03, 0xa438, 0x6f47, 0xa438, 0x266f, + 0xa438, 0x5900, 0xa438, 0x731f, 0xa438, 0x0273, 0xa438, 0x2203, + 0xa438, 0x8c08, 0xa438, 0xee84, 0xa438, 0x7100, 0xa438, 0x0286, + 0xa438, 0xece0, 0xa438, 0x8043, 0xa438, 0xf626, 0xa438, 0xe480, + 0xa438, 0x43af, 0xa438, 0x6611, 0xa438, 0xf8e0, 0xa438, 0x8012, + 0xa438, 0xac26, 0xa438, 0x03af, 0xa438, 0x86ff, 0xa438, 0x0287, + 0xa438, 0x0102, 0xa438, 0x8906, 0xa438, 0x0289, 0xa438, 0x29fc, + 0xa438, 0x04f8, 0xa438, 0xf9ef, 0xa438, 0x59f9, 0xa438, 0xfaee, + 0xa438, 0x8476, 0xa438, 0x00d6, 0xa438, 0x008f, 0xa438, 0x0266, + 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, + 0xa438, 0x08e4, 0xa438, 0x8fe7, 0xa438, 0xe58f, 0xa438, 0xe8ae, + 0xa438, 0x06e0, 0xa438, 0x8fe7, 0xa438, 0xe18f, 0xa438, 0xe8ee, + 0xa438, 0x8476, 0xa438, 0x01d6, 0xa438, 0x00c0, 0xa438, 0x0266, + 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x00d6, 0xa438, 0x0090, + 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, + 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8fe9, 0xa438, 0xe58f, + 0xa438, 0xeaae, 0xa438, 0x06e0, 0xa438, 0x8fe9, 0xa438, 0xe18f, + 0xa438, 0xeaee, 0xa438, 0x8476, 0xa438, 0x01d6, 0xa438, 0x00c1, + 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x00d6, + 0xa438, 0x0091, 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, + 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8feb, + 0xa438, 0xe58f, 0xa438, 0xecae, 0xa438, 0x06e0, 0xa438, 0x8feb, + 0xa438, 0xe18f, 0xa438, 0xecee, 0xa438, 0x8476, 0xa438, 0x01d6, + 0xa438, 0x00c2, 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, + 0xa438, 0x01d6, 0xa438, 0x008f, 0xa438, 0x0266, 0xa438, 0x53ef, + 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, + 0xa438, 0x8fed, 0xa438, 0xe58f, 0xa438, 0xeeae, 0xa438, 0x06e0, + 0xa438, 0x8fed, 0xa438, 0xe18f, 0xa438, 0xeeee, 0xa438, 0x8476, + 0xa438, 0x02d6, 0xa438, 0x00c0, 0xa438, 0x0266, 0xa438, 0x71ee, + 0xa438, 0x8476, 0xa438, 0x01d6, 0xa438, 0x0090, 0xa438, 0x0266, + 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, + 0xa438, 0x08e4, 0xa438, 0x8fef, 0xa438, 0xe58f, 0xa438, 0xf0ae, + 0xa438, 0x06e0, 0xa438, 0x8fef, 0xa438, 0xe18f, 0xa438, 0xf0ee, + 0xa438, 0x8476, 0xa438, 0x02d6, 0xa438, 0x00c1, 0xa438, 0x0266, + 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x01d6, 0xa438, 0x0091, + 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, + 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8ff1, 0xa438, 0xe58f, + 0xa438, 0xf2ae, 0xa438, 0x06e0, 0xa438, 0x8ff1, 0xa438, 0xe18f, + 0xa438, 0xf2ee, 0xa438, 0x8476, 0xa438, 0x02d6, 0xa438, 0x00c2, + 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x02d6, + 0xa438, 0x008f, 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, + 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8ff3, + 0xa438, 0xe58f, 0xa438, 0xf4ae, 0xa438, 0x06e0, 0xa438, 0x8ff3, + 0xa438, 0xe18f, 0xa438, 0xf4ee, 0xa438, 0x8476, 0xa438, 0x04d6, + 0xa438, 0x00c0, 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, + 0xa438, 0x02d6, 0xa438, 0x0090, 0xa438, 0x0266, 0xa438, 0x53ef, + 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, + 0xa438, 0x8ff5, 0xa438, 0xe58f, 0xa438, 0xf6ae, 0xa438, 0x06e0, + 0xa438, 0x8ff5, 0xa438, 0xe18f, 0xa438, 0xf6ee, 0xa438, 0x8476, + 0xa438, 0x04d6, 0xa438, 0x00c1, 0xa438, 0x0266, 0xa438, 0x71ee, + 0xa438, 0x8476, 0xa438, 0x02d6, 0xa438, 0x0091, 0xa438, 0x0266, + 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, + 0xa438, 0x08e4, 0xa438, 0x8ff7, 0xa438, 0xe58f, 0xa438, 0xf8ae, + 0xa438, 0x06e0, 0xa438, 0x8ff7, 0xa438, 0xe18f, 0xa438, 0xf8ee, + 0xa438, 0x8476, 0xa438, 0x04d6, 0xa438, 0x00c2, 0xa438, 0x0266, + 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x03d6, 0xa438, 0x008f, + 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, 0xa438, 0x1200, + 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8ff9, 0xa438, 0xe58f, + 0xa438, 0xfaae, 0xa438, 0x06e0, 0xa438, 0x8ff9, 0xa438, 0xe18f, + 0xa438, 0xfaee, 0xa438, 0x8476, 0xa438, 0x08d6, 0xa438, 0x00c0, + 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, 0xa438, 0x03d6, + 0xa438, 0x0090, 0xa438, 0x0266, 0xa438, 0x53ef, 0xa438, 0x643e, + 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, 0xa438, 0x8ffb, + 0xa438, 0xe58f, 0xa438, 0xfcae, 0xa438, 0x06e0, 0xa438, 0x8ffb, + 0xa438, 0xe18f, 0xa438, 0xfcee, 0xa438, 0x8476, 0xa438, 0x08d6, + 0xa438, 0x00c1, 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x8476, + 0xa438, 0x03d6, 0xa438, 0x0091, 0xa438, 0x0266, 0xa438, 0x53ef, + 0xa438, 0x643e, 0xa438, 0x1200, 0xa438, 0xac4f, 0xa438, 0x08e4, + 0xa438, 0x8ffd, 0xa438, 0xe58f, 0xa438, 0xfeae, 0xa438, 0x06e0, + 0xa438, 0x8ffd, 0xa438, 0xe18f, 0xa438, 0xfeee, 0xa438, 0x8476, + 0xa438, 0x08d6, 0xa438, 0x00c2, 0xa438, 0x0266, 0xa438, 0x71fe, + 0xa438, 0xfdef, 0xa438, 0x95fd, 0xa438, 0xfc04, 0xa438, 0xf8f9, + 0xa438, 0xfad4, 0xa438, 0x0400, 0xa438, 0xd600, 0xa438, 0x0dd3, + 0xa438, 0x0fe7, 0xa438, 0x8476, 0xa438, 0x0266, 0xa438, 0x71d4, + 0xa438, 0x1400, 0xa438, 0xd600, 0xa438, 0x0dd3, 0xa438, 0x0fe7, + 0xa438, 0x8476, 0xa438, 0x0266, 0xa438, 0x71fe, 0xa438, 0xfdfc, + 0xa438, 0x04f8, 0xa438, 0xf9fa, 0xa438, 0xd410, 0xa438, 0x00d6, + 0xa438, 0x000d, 0xa438, 0xd30f, 0xa438, 0xe784, 0xa438, 0x7602, + 0xa438, 0x6671, 0xa438, 0xd400, 0xa438, 0x00d6, 0xa438, 0x000d, + 0xa438, 0xd30f, 0xa438, 0xe784, 0xa438, 0x7602, 0xa438, 0x6671, + 0xa438, 0xfefd, 0xa438, 0xfc04, 0xa438, 0xe080, 0xa438, 0x4fac, + 0xa438, 0x2317, 0xa438, 0xe080, 0xa438, 0x44ad, 0xa438, 0x231a, + 0xa438, 0x0289, 0xa438, 0x75e0, 0xa438, 0x8044, 0xa438, 0xac23, + 0xa438, 0x11bf, 0xa438, 0x6ecf, 0xa438, 0x0276, 0xa438, 0x74ae, + 0xa438, 0x0902, 0xa438, 0x8adb, 0xa438, 0x021f, 0xa438, 0xe702, + 0xa438, 0x1fbb, 0xa438, 0xaf1f, 0xa438, 0x95f8, 0xa438, 0xf9ef, + 0xa438, 0x59f9, 0xa438, 0xfafb, 0xa438, 0xe080, 0xa438, 0x12ac, + 0xa438, 0x2303, 0xa438, 0xaf8a, 0xa438, 0xd0d4, 0xa438, 0x0120, + 0xa438, 0xd600, 0xa438, 0x10d2, 0xa438, 0x0fe6, 0xa438, 0x8476, + 0xa438, 0x0266, 0xa438, 0x71ee, 0xa438, 0x846f, 0xa438, 0x00d4, + 0xa438, 0x000f, 0xa438, 0xbf72, 0xa438, 0x9e02, 0xa438, 0x7697, + 0xa438, 0x0275, 0xa438, 0xbeef, 0xa438, 0x47e4, 0xa438, 0x8474, + 0xa438, 0xe584, 0xa438, 0x75bf, 0xa438, 0x729b, 0xa438, 0x0276, + 0xa438, 0xb6e5, 0xa438, 0x846f, 0xa438, 0xef31, 0xa438, 0xbf6e, + 0xa438, 0x0602, 0xa438, 0x76b6, 0xa438, 0xef64, 0xa438, 0xbf6e, + 0xa438, 0x0902, 0xa438, 0x76b6, 0xa438, 0x1e64, 0xa438, 0xbf6e, + 0xa438, 0x0f02, 0xa438, 0x76b6, 0xa438, 0x1e64, 0xa438, 0xac40, + 0xa438, 0x05a3, 0xa438, 0x0f0c, 0xa438, 0xae26, 0xa438, 0xa303, + 0xa438, 0x02ae, 0xa438, 0x21a3, 0xa438, 0x0c02, 0xa438, 0xae1c, + 0xa438, 0xe084, 0xa438, 0x74e1, 0xa438, 0x8475, 0xa438, 0xef64, + 0xa438, 0xd000, 0xa438, 0xd196, 0xa438, 0xef74, 0xa438, 0x0275, + 0xa438, 0xd9ad, 0xa438, 0x50b7, 0xa438, 0xe083, 0xa438, 0xecf7, + 0xa438, 0x23e4, 0xa438, 0x83ec, 0xa438, 0xbf72, 0xa438, 0x9e02, + 0xa438, 0x766b, 0xa438, 0x0287, 0xa438, 0x0102, 0xa438, 0x8906, + 0xa438, 0xee83, 0xa438, 0xe800, 0xa438, 0xbf72, 0xa438, 0x6b02, + 0xa438, 0x766b, 0xa438, 0xbf72, 0xa438, 0x6e02, 0xa438, 0x766b, + 0xa438, 0xbf72, 0xa438, 0x7102, 0xa438, 0x766b, 0xa438, 0xbf72, + 0xa438, 0x7402, 0xa438, 0x766b, 0xa438, 0xbf72, 0xa438, 0x7702, + 0xa438, 0x766b, 0xa438, 0xbf72, 0xa438, 0x7a02, 0xa438, 0x766b, + 0xa438, 0xd400, 0xa438, 0x0fbf, 0xa438, 0x7295, 0xa438, 0x0276, + 0xa438, 0x97d7, 0xa438, 0x0400, 0xa438, 0xbf6e, 0xa438, 0x0602, + 0xa438, 0x76b6, 0xa438, 0xef64, 0xa438, 0xbf6e, 0xa438, 0x0902, + 0xa438, 0x76b6, 0xa438, 0x1e64, 0xa438, 0xbf6e, 0xa438, 0x0f02, + 0xa438, 0x76b6, 0xa438, 0x1e64, 0xa438, 0xac40, 0xa438, 0x0fbf, + 0xa438, 0x7298, 0xa438, 0x0276, 0xa438, 0xb6e5, 0xa438, 0x83e8, + 0xa438, 0xa10f, 0xa438, 0x28af, 0xa438, 0x8a95, 0xa438, 0xbf8b, + 0xa438, 0xf302, 0xa438, 0x76b6, 0xa438, 0xac28, 0xa438, 0x02ae, + 0xa438, 0x0bbf, 0xa438, 0x8bf9, 0xa438, 0x0276, 0xa438, 0xb6e5, + 0xa438, 0x83e8, 0xa438, 0xae09, 0xa438, 0xbf8b, 0xa438, 0xf602, + 0xa438, 0x76b6, 0xa438, 0xe583, 0xa438, 0xe8a1, 0xa438, 0x0303, + 0xa438, 0xaf8a, 0xa438, 0x95b7, 0xa438, 0xafe2, 0xa438, 0x83ec, + 0xa438, 0xf735, 0xa438, 0xe683, 0xa438, 0xecbf, 0xa438, 0x7295, + 0xa438, 0x0276, 0xa438, 0x6bbf, 0xa438, 0x726b, 0xa438, 0x0276, + 0xa438, 0x74bf, 0xa438, 0x726e, 0xa438, 0x0276, 0xa438, 0x74bf, + 0xa438, 0x7271, 0xa438, 0x0276, 0xa438, 0x74bf, 0xa438, 0x7274, + 0xa438, 0x0276, 0xa438, 0x74bf, 0xa438, 0x7277, 0xa438, 0x0276, + 0xa438, 0x74bf, 0xa438, 0x727a, 0xa438, 0x0276, 0xa438, 0x7402, + 0xa438, 0x8929, 0xa438, 0xd401, 0xa438, 0x28d6, 0xa438, 0x0010, + 0xa438, 0xd20f, 0xa438, 0xe684, 0xa438, 0x7602, 0xa438, 0x6671, + 0xa438, 0x021f, 0xa438, 0xbbff, 0xa438, 0xfefd, 0xa438, 0xef95, + 0xa438, 0xfdfc, 0xa438, 0x04f8, 0xa438, 0xf9ef, 0xa438, 0x59f9, + 0xa438, 0xe080, 0xa438, 0x12ad, 0xa438, 0x230c, 0xa438, 0xbf72, + 0xa438, 0x9e02, 0xa438, 0x766b, 0xa438, 0xbf72, 0xa438, 0x9502, + 0xa438, 0x766b, 0xa438, 0xfdef, 0xa438, 0x95fd, 0xa438, 0xfc04, + 0xa438, 0xbf6e, 0xa438, 0x0602, 0xa438, 0x76b6, 0xa438, 0xef64, + 0xa438, 0xbf6e, 0xa438, 0x0902, 0xa438, 0x76b6, 0xa438, 0x1e64, + 0xa438, 0xbf6e, 0xa438, 0x0f02, 0xa438, 0x76b6, 0xa438, 0x1e64, + 0xa438, 0xac40, 0xa438, 0x0ebf, 0xa438, 0x7298, 0xa438, 0x0276, + 0xa438, 0xb6e5, 0xa438, 0x8478, 0xa438, 0xa10f, 0xa438, 0x26ae, + 0xa438, 0x47bf, 0xa438, 0x8bf3, 0xa438, 0x0276, 0xa438, 0xb6ac, + 0xa438, 0x2802, 0xa438, 0xae0b, 0xa438, 0xbf8b, 0xa438, 0xf902, + 0xa438, 0x76b6, 0xa438, 0xe584, 0xa438, 0x78ae, 0xa438, 0x09bf, + 0xa438, 0x8bf6, 0xa438, 0x0276, 0xa438, 0xb6e5, 0xa438, 0x8478, + 0xa438, 0xa103, 0xa438, 0x02ae, 0xa438, 0x23e0, 0xa438, 0x8474, + 0xa438, 0xe184, 0xa438, 0x75ef, 0xa438, 0x64e0, 0xa438, 0x83fc, + 0xa438, 0xe183, 0xa438, 0xfdef, 0xa438, 0x7402, 0xa438, 0x75d9, + 0xa438, 0xad50, 0xa438, 0x0ae0, 0xa438, 0x83ec, 0xa438, 0xf721, + 0xa438, 0xe483, 0xa438, 0xecae, 0xa438, 0x03af, 0xa438, 0x68e4, + 0xa438, 0xbf72, 0xa438, 0x9502, 0xa438, 0x766b, 0xa438, 0xe083, + 0xa438, 0xebad, 0xa438, 0x2170, 0xa438, 0xbf73, 0xa438, 0x7f02, + 0xa438, 0x766b, 0xa438, 0xd700, 0xa438, 0x64bf, 0xa438, 0x73c4, + 0xa438, 0x0276, 0xa438, 0xb6a4, 0xa438, 0x0000, 0xa438, 0x02ae, + 0xa438, 0x0d87, 0xa438, 0xa700, 0xa438, 0x00ef, 0xa438, 0xe183, + 0xa438, 0xecf7, 0xa438, 0x2ae5, 0xa438, 0x83ec, 0xa438, 0xbf73, + 0xa438, 0xbe02, 0xa438, 0x766b, 0xa438, 0xbf73, 0xa438, 0xb802, + 0xa438, 0x766b, 0xa438, 0xbf73, 0xa438, 0xc102, 0xa438, 0x766b, + 0xa438, 0xbf73, 0xa438, 0xbb02, 0xa438, 0x766b, 0xa438, 0xe084, + 0xa438, 0x9ee1, 0xa438, 0x849f, 0xa438, 0xbf72, 0xa438, 0x7d02, + 0xa438, 0x7697, 0xa438, 0xbf72, 0xa438, 0x8002, 0xa438, 0x7697, + 0xa438, 0xbf72, 0xa438, 0x8302, 0xa438, 0x7697, 0xa438, 0xbf72, + 0xa438, 0x8602, 0xa438, 0x7697, 0xa438, 0xbf72, 0xa438, 0x8902, + 0xa438, 0x7674, 0xa438, 0xbf72, 0xa438, 0x8c02, 0xa438, 0x7674, + 0xa438, 0xbf72, 0xa438, 0x8f02, 0xa438, 0x7674, 0xa438, 0xbf72, + 0xa438, 0x9202, 0xa438, 0x7674, 0xa438, 0xee84, 0xa438, 0x7700, + 0xa438, 0xe080, 0xa438, 0x44f6, 0xa438, 0x21e4, 0xa438, 0x8044, + 0xa438, 0xaf68, 0xa438, 0xe411, 0xa438, 0xd1a4, 0xa438, 0x10bc, + 0xa438, 0x7432, 0xa438, 0xbc74, 0xa438, 0xbbbf, 0xa438, 0x14cc, + 0xa438, 0xbfaa, 0xa438, 0x00bf, 0xa438, 0x9055, 0xa438, 0xbf06, + 0xa438, 0x10bf, 0xa438, 0xb876, 0xa438, 0xbe02, 0xa438, 0x54be, + 0xa438, 0x0232, 0xa438, 0xbe02, 0xa438, 0x10be, 0xa438, 0x0200, + 0xa436, 0x8fe7, 0xa438, 0x1200, 0xa436, 0x8fe9, 0xa438, 0x1200, + 0xa436, 0x8feb, 0xa438, 0x1200, 0xa436, 0x8fed, 0xa438, 0x1200, + 0xa436, 0x8fef, 0xa438, 0x1200, 0xa436, 0x8ff1, 0xa438, 0x1200, + 0xa436, 0x8ff3, 0xa438, 0x1200, 0xa436, 0x8ff5, 0xa438, 0x1200, + 0xa436, 0x8ff7, 0xa438, 0x1200, 0xa436, 0x8ff9, 0xa438, 0x1200, + 0xa436, 0x8ffb, 0xa438, 0x1200, 0xa436, 0x8ffd, 0xa438, 0x1200, + 0xa436, 0xb818, 0xa438, 0x6602, 0xa436, 0xb81a, 0xa438, 0x1f75, + 0xa436, 0xb81c, 0xa438, 0x67eb, 0xa436, 0xb81e, 0xa438, 0xffff, + 0xa436, 0xb850, 0xa438, 0xffff, 0xa436, 0xb852, 0xa438, 0xffff, + 0xa436, 0xb878, 0xa438, 0xffff, 0xa436, 0xb884, 0xa438, 0xffff, + 0xa436, 0xb832, 0xa438, 0x0007, 0xB82E, 0x0000, 0xa436, 0x8023, + 0xa438, 0x0000, 0xB820, 0x0000, 0xFFFF, 0xFFFF +}; + +static const u16 phy_mcu_ram_code_8127a_2[] = { + 0xb892, 0x0000, 0xB88E, 0xc07c, 0xB890, 0x0203, 0xB890, 0x0304, + 0xB890, 0x0405, 0xB890, 0x0607, 0xB890, 0x0809, 0xB890, 0x0B0D, + 0xB890, 0x0F11, 0xB890, 0x1418, 0xB890, 0x1B20, 0xB890, 0x252B, + 0xB890, 0x343E, 0xB890, 0x4854, 0xB890, 0x6203, 0xB890, 0x0304, + 0xB890, 0x0506, 0xB890, 0x080A, 0xB890, 0x0C0E, 0xB890, 0x1216, + 0xB890, 0x1B22, 0xB890, 0x2A34, 0xB890, 0x404F, 0xB890, 0x6171, + 0xB890, 0x7884, 0xB890, 0x9097, 0xB890, 0x0203, 0xB890, 0x0406, + 0xB890, 0x080B, 0xB890, 0x0E13, 0xB890, 0x1820, 0xB890, 0x2A39, + 0xB890, 0x4856, 0xB890, 0xE060, 0xB890, 0xE050, 0xB890, 0xD080, + 0xB890, 0x8070, 0xB890, 0x70A0, 0xB890, 0x1000, 0xB890, 0x60D0, + 0xB890, 0xB010, 0xB890, 0xE0B0, 0xB890, 0x80C0, 0xB890, 0xE000, + 0xB890, 0x2020, 0xB890, 0x1020, 0xB890, 0xE090, 0xB890, 0x80C0, + 0xB890, 0x3020, 0xB890, 0x00E0, 0xB890, 0x40A0, 0xB890, 0xE020, + 0xB890, 0x5060, 0xB890, 0xE0D0, 0xB890, 0xA000, 0xB890, 0x3030, + 0xB890, 0x4070, 0xB890, 0xE0E0, 0xB890, 0xD080, 0xB890, 0xA010, + 0xB890, 0xE040, 0xB890, 0x80B0, 0xB890, 0x50B0, 0xB890, 0x2090, + 0xB820, 0x0000, 0xFFFF, 0xFFFF +}; + +static void +rtl8127_real_set_phy_mcu_8127a_tc_1(struct net_device *dev) +{ + rtl8127_set_phy_mcu_ram_code(dev, + phy_mcu_ram_code_8127a_tc_1, + ARRAY_SIZE(phy_mcu_ram_code_8127a_tc_1)); +} + +static void +rtl8127_set_phy_mcu_8127a_tc(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_set_phy_mcu_patch_request(tp); + + rtl8127_real_set_phy_mcu_8127a_tc_1(dev); + + rtl8127_clear_phy_mcu_patch_request(tp); +} + +static void +rtl8127_real_set_phy_mcu_8127a_1(struct net_device *dev) +{ + rtl8127_set_phy_mcu_ram_code(dev, + phy_mcu_ram_code_8127a_1, + ARRAY_SIZE(phy_mcu_ram_code_8127a_1)); +} + +static void +rtl8127_real_set_phy_mcu_8127a_2(struct net_device *dev) +{ + rtl8127_set_phy_mcu_ram_code(dev, + phy_mcu_ram_code_8127a_2, + ARRAY_SIZE(phy_mcu_ram_code_8127a_2)); +} + +static void +rtl8127_set_phy_mcu_8127a_1(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_set_phy_mcu_patch_request(tp); + + rtl8127_real_set_phy_mcu_8127a_1(dev); + + rtl8127_clear_phy_mcu_patch_request(tp); + + rtl8127_set_phy_mcu_patch_request(tp); + + rtl8127_real_set_phy_mcu_8127a_2(dev); + + rtl8127_clear_phy_mcu_patch_request(tp); +} + +static void +rtl8127_init_hw_phy_mcu(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u8 require_disable_phy_disable_mode = FALSE; + + if (tp->NotWrRamCodeToMicroP == TRUE) + return; + + if (rtl8127_check_hw_phy_mcu_code_ver(dev)) + return; + + if (HW_SUPPORT_CHECK_PHY_DISABLE_MODE(tp) && rtl8127_is_in_phy_disable_mode(dev)) + require_disable_phy_disable_mode = TRUE; + + if (require_disable_phy_disable_mode) + rtl8127_disable_phy_disable_mode(dev); + + switch (tp->mcfg) { + case CFG_METHOD_1: + rtl8127_set_phy_mcu_8127a_tc(dev); + break; + case CFG_METHOD_2: + rtl8127_set_phy_mcu_8127a_1(dev); + break; + default: + break; + } + + if (require_disable_phy_disable_mode) + rtl8127_enable_phy_disable_mode(dev); + + rtl8127_write_hw_phy_mcu_code_ver(dev); + + rtl8127_mdio_write(tp,0x1F, 0x0000); + + tp->HwHasWrRamCodeToMicroP = TRUE; +} +#endif + +static void +rtl8127_enable_phy_aldps(struct rtl8127_private *tp) +{ + //enable aldps + //GPHY OCP 0xA430 bit[2] = 0x1 (en_aldps) + rtl8127_set_eth_phy_ocp_bit(tp, 0xA430, BIT_2); +} + +static void +rtl8127_tgphy_irq_mask_and_ack(struct rtl8127_private *tp) +{ + switch (tp->mcfg) { + case CFG_METHOD_2: + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA4D2, 0x0000); + (void)rtl8127_mdio_direct_read_phy_ocp(tp, 0xA4D4); + break; + default: + break; + } +} + +static void +rtl8127_hw_phy_config_8127a_tc_1(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_set_eth_phy_ocp_bit(tp, 0xA442, BIT_11); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x815E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x8600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x8169); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x8600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x8174); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0xA100); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x83BF); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x5A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x83C5); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x5A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x83CB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x8B00); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8238); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xC000, + 0x4000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x823A); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xA000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xa436, 0x8148); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xa438, + 0xFF00, + 0x0100); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84AD); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0C00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84B2); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0800); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84B7); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x1400); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84BC); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0040); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84C0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00D6); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84BE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00A0); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84AE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0C0C); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84B0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0C0C); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xBD7A, 0xAAAA); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xBCE0, 0x6666); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x85FC); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0AAA); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x85FF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0AAA); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAC32, BIT_3); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAC32, BIT_11); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xADDC, + 0x3FFF, + 0x2000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8111); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0F00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80E9); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0F00); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAEC4, + 0xFF00, + 0x4600); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAC56, + 0x0007, + 0x0005); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x825B); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0D00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8283); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0334); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8289); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xADB8, 0x0190); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAE3A, + 0x00FF, + 0x0026); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAE4A, + 0x0FF0, + 0x0150); + + rtl8127_set_eth_phy_ocp_bit(tp, 0xAEC2, BIT_12); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAE22, 0x0352); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAEC0, 0x00FA); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8188); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8203); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x827E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81CB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8246); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x818A); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8205); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8280); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xF500); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x83DD); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, BIT_9); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x840B); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, BIT_9); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83BC); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_10); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83BE); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_10); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83C0); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x0700, + 0x0400); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83C2); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_10); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83C4); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_10); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x83C6); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x0700, + 0x0400); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84C2); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0xFEFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84C4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0003); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84C6); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0116); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x84C8); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x6300); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x822A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x4FFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81AF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x4067); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8134); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x5069); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x822C); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x1A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81B1); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x3A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8136); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x810E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x9000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8114); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x9000); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB63C, BIT_9); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80B4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xB63B); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80E4); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0700); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80E5); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x7000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80EA); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80EB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x5000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8291); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0416); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8015); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, BIT_11); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x895E); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x01A0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8960); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x01A0); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x826B); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xF0AF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81F0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xF0AF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8175); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xF0AF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x826D); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0100); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81F2); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0100); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8177); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0100); + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAC1C, BIT_8 | BIT_7); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAC1E, BIT_13 | BIT_12); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAD96, 0xAAFF); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAD98, + 0x00FF, + 0x00AA); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAE38, + 0x3FFF, + 0x2554); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAE3A, + 0xF000, + 0xA000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8932); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0900); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x892F); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0900); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x892C); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0900); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x85B6); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, 0xFF00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x85B4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8905); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0xFF00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8853); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x2800); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x884B); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x3F00); + + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xBDE6, 0x3FFF); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xBDE8, 0x3FFF); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xBF0E, 0x0003); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8156); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x1600); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x80AB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x7500); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x80C3); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x090D); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x80C6); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xC600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x80BF); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x5500); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8096); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x4500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x809D); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0200); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x809B); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0xE50A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8099); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x9906); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x831F); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x5000); + + + rtl8127_set_eth_phy_ocp_bit(tp, 0xB648, BIT_14); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA4E0, BIT_15); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x849A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0004); + + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAC1C, 0x0C00); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA42C, BIT_6); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xACBA, 0xFC00); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8122); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xC000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8123); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xC000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80FA); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x2000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x825B); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xB000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80D2); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xD000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80D3); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xD000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80C8); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0200); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80CA); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0300); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80E2); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x2300); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80A9); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x0F00, + 0x0A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80AA); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x5000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x80AB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0xA000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x805A); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x2000); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8106); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x40CC); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x812C); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x40CC); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8096); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x7500); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x809C); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x6300); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x859E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x1F00); + + + if (aspm && HW_HAS_WRITE_PHY_MCU_RAM_CODE(tp)) + rtl8127_enable_phy_aldps(tp); +} + +static void +rtl8127_hw_phy_config_8127a_1(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + rtl8127_tgphy_irq_mask_and_ack(tp); + + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA442, BIT_11); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8415); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x9300); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81A3); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0F00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81AE); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0F00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81B9); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xB900); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x83B0); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0x0E00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x83C5); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0x0E00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x83DA); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0x0E00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x83EF); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0x0E00); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8173); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x8620); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8175); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x8671); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x817C); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8187); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8192); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x819D); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81A8); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81B3); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81BE); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_13); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x817D); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xA600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8188); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xA600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8193); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xA600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x819E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xA600); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81A9); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x1400); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81B4); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x1400); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81BF); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0xA600); + + + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAEAA, (BIT_5 | BIT_3)); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84F0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x201C); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84F2); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x3117); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAEC6, 0x0000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAE20, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAECE, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAED2, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xAEC8, 0x0000); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xAED0, BIT_0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xADB8, 0x0150); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8197); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8231); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x82CB); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x82CD); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5700); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8233); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5700); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8199); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x5700); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x815A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0150); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81F4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0150); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x828E); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0150); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81B1); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x824B); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x82E5); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84F7); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x2800); + rtl8127_set_eth_phy_ocp_bit(tp, 0xAEC2, BIT_12); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x81B3); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xAD00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x824D); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xAD00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x82E7); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xAD00); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xAE4E, + 0x000F, + 0x0001); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x82CE); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xF000, + 0x4000); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84AC); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84AE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84B0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xF818); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x84B2); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x6000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FFC); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x6008); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FFE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xF450); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8015); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, BIT_9); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8016); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB87E, BIT_11); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FE6); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x0800); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FE4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x2114); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8647); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xA7B1); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8649); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xBBCA); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x864B); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0xDC00); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8154); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xC000, + 0x4000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8158); + rtl8127_clear_eth_phy_ocp_bit(tp, 0xB87E, 0xC000); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x826C); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x826E); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xFFFF); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8872); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x0E00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8012); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_11); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8012); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA438, BIT_14); + rtl8127_set_eth_phy_ocp_bit(tp, 0xB576, BIT_0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x834A); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0700); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8217); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0x3F00, + 0x2A00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81B1); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0xFF00, + 0x0B00); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FED); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x4E00); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8370); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x8671); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8372); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x86C8); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8401); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x86C8); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8403); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x86DA); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8406); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8408); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x840A); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x840C); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x840E); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8410); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8412); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8414); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8416); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xA438, + 0x1800, + 0x1000); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x82BD); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x1F40); + + + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xBFB4, + 0x07FF, + 0x0328); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xBFB6, 0x3E14); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x81C4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x003B); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0086); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00B7); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00DB); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00FE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00FE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00FE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00FE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x00C3); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0078); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0047); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA438, 0x0023); + + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x88D7); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x01A0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x88D9); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x01A0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FFA); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x002A); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FEE); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xFFDF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FF0); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xFFFF); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FF2); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0A4A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FF4); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xAA5A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FF6); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0x0A4A); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x8FF8); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87E, 0xAA5A); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xB87C, 0x88D5); + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + 0xB87E, + 0xFF00, + 0x0200); + + + rtl8127_set_eth_phy_ocp_bit(tp, 0xA430, BIT_1 | BIT_0); + + + if (aspm && HW_HAS_WRITE_PHY_MCU_RAM_CODE(tp)) + rtl8127_enable_phy_aldps(tp); +} + +static void +rtl8127_hw_phy_config(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + + if (tp->resume_not_chg_speed) + return; + + tp->phy_reset_enable(dev); + + if (HW_DASH_SUPPORT_TYPE_3(tp) && tp->HwPkgDet == 0x06) + return; + + spin_lock_irqsave(&tp->phy_lock, flags); + +#ifndef ENABLE_USE_FIRMWARE_FILE + if (!tp->rtl_fw) + rtl8127_init_hw_phy_mcu(dev); +#endif + + switch (tp->mcfg) { + case CFG_METHOD_1: + rtl8127_hw_phy_config_8127a_tc_1(dev); + break; + case CFG_METHOD_2: + rtl8127_hw_phy_config_8127a_1(dev); + break; + default: + break; + } + + //legacy force mode(Chap 22) + rtl8127_clear_eth_phy_ocp_bit(tp, 0xA5B4, BIT_15); + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + + if (HW_HAS_WRITE_PHY_MCU_RAM_CODE(tp)) { + if (tp->eee.eee_enabled) + rtl8127_enable_eee(tp); + else + rtl8127_disable_eee(tp); + } + + spin_unlock_irqrestore(&tp->phy_lock, flags); +} + +static void +rtl8127_up(struct net_device *dev) +{ + rtl8127_hw_init(dev); + rtl8127_hw_reset(dev); + rtl8127_powerup_pll(dev); + rtl8127_hw_ephy_config(dev); + rtl8127_hw_phy_config(dev); + rtl8127_hw_config(dev); +} + +/* +static inline void rtl8127_delete_esd_timer(struct net_device *dev, struct timer_list *timer) +{ + del_timer_sync(timer); +} + +static inline void rtl8127_request_esd_timer(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct timer_list *timer = &tp->esd_timer; +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) + setup_timer(timer, rtl8127_esd_timer, (unsigned long)dev); +#else + timer_setup(timer, rtl8127_esd_timer, 0); +#endif + mod_timer(timer, jiffies + RTL8127_ESD_TIMEOUT); +} +*/ + +/* +static inline void rtl8127_delete_link_timer(struct net_device *dev, struct timer_list *timer) +{ + del_timer_sync(timer); +} + +static inline void rtl8127_request_link_timer(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct timer_list *timer = &tp->link_timer; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) + setup_timer(timer, rtl8127_link_timer, (unsigned long)dev); +#else + timer_setup(timer, rtl8127_link_timer, 0); +#endif + mod_timer(timer, jiffies + RTL8127_LINK_TIMEOUT); +} +*/ + +#ifdef CONFIG_NET_POLL_CONTROLLER +/* + * Polling 'interrupt' - used by things like netconsole to send skbs + * without having to re-enable interrupts. It's not called while + * the interrupt routine is executing. + */ +static void +rtl8127_netpoll(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + for (i = 0; i < tp->irq_nvecs; i++) { + struct r8127_irq *irq = &tp->irq_tbl[i]; + struct r8127_napi *r8127napi = &tp->r8127napi[i]; + + disable_irq(irq->vector); + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,12,0) + irq->handler(irq->vector, r8127napi); +#elif LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) + irq->handler(irq->vector, r8127napi, NULL); +#else + irq->handler(irq->vector, r8127napi); +#endif + + enable_irq(irq->vector); + } +} +#endif //CONFIG_NET_POLL_CONTROLLER + +static void +rtl8127_setup_interrupt_mask(struct rtl8127_private *tp) +{ + int i; + + if (tp->HwCurrIsrVer == 6) { + tp->intr_mask = ISRIMR_V6_LINKCHG | ISRIMR_V6_TOK_Q0; + if (tp->num_tx_rings > 1) + tp->intr_mask |= ISRIMR_V6_TOK_Q1; + for (i = 0; i < tp->num_rx_rings; i++) + tp->intr_mask |= ISRIMR_V6_ROK_Q0 << i; + } else if (tp->HwCurrIsrVer == 5) { + tp->intr_mask = ISRIMR_V5_LINKCHG | ISRIMR_V5_TOK_Q0; + if (tp->num_tx_rings > 1) + tp->intr_mask |= ISRIMR_V5_TOK_Q1; + for (i = 0; i < tp->num_rx_rings; i++) + tp->intr_mask |= ISRIMR_V5_ROK_Q0 << i; + } else if (tp->HwCurrIsrVer == 4) { + tp->intr_mask = ISRIMR_V4_LINKCHG; + for (i = 0; i < tp->num_rx_rings; i++) + tp->intr_mask |= ISRIMR_V4_ROK_Q0 << i; + } else if (tp->HwCurrIsrVer == 3) { + tp->intr_mask = ISRIMR_V2_LINKCHG; + for (i = 0; i < max(tp->num_tx_rings, tp->num_rx_rings); i++) + tp->intr_mask |= ISRIMR_V2_ROK_Q0 << i; + } else if (tp->HwCurrIsrVer == 2) { + tp->intr_mask = ISRIMR_V2_LINKCHG | ISRIMR_TOK_Q0; + if (tp->num_tx_rings > 1) + tp->intr_mask |= ISRIMR_TOK_Q1; + + for (i = 0; i < tp->num_rx_rings; i++) + tp->intr_mask |= ISRIMR_V2_ROK_Q0 << i; + } else { + tp->intr_mask = LinkChg | RxDescUnavail | TxOK | RxOK | SWInt; + tp->timer_intr_mask = LinkChg | PCSTimeout; + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) { + if (HW_DASH_SUPPORT_TYPE_3(tp)) { + tp->timer_intr_mask |= (ISRIMR_DASH_INTR_EN | ISRIMR_DASH_INTR_CMAC_RESET); + tp->intr_mask |= (ISRIMR_DASH_INTR_EN | ISRIMR_DASH_INTR_CMAC_RESET); + } + } +#endif + } +} + +static void +rtl8127_setup_mqs_reg(struct rtl8127_private *tp) +{ + u16 hw_clo_ptr0_reg, sw_tail_ptr0_reg; + u16 reg_len; + int i; + + //tx + tp->tx_ring[0].tdsar_reg = TxDescStartAddrLow; + for (i = 1; i < tp->HwSuppNumTxQueues; i++) + tp->tx_ring[i].tdsar_reg = (u16)(TNPDS_Q1_LOW_8125 + (i - 1) * 8); + + switch (tp->HwSuppTxNoCloseVer) { + case 4: + case 5: + hw_clo_ptr0_reg = HW_CLO_PTR0_8126; + sw_tail_ptr0_reg = SW_TAIL_PTR0_8126; + reg_len = 4; + break; + case 6: + hw_clo_ptr0_reg = HW_CLO_PTR0_8125BP; + sw_tail_ptr0_reg = SW_TAIL_PTR0_8125BP; + reg_len = 8; + break; + default: + hw_clo_ptr0_reg = HW_CLO_PTR0_8125; + sw_tail_ptr0_reg = SW_TAIL_PTR0_8125; + reg_len = 4; + break; + } + + for (i = 0; i < tp->HwSuppNumTxQueues; i++) { + tp->tx_ring[i].hw_clo_ptr_reg = (u16)(hw_clo_ptr0_reg + i * reg_len); + tp->tx_ring[i].sw_tail_ptr_reg = (u16)(sw_tail_ptr0_reg + i * reg_len); + } + + //rx + tp->rx_ring[0].rdsar_reg = RxDescAddrLow; + for (i = 1; i < tp->HwSuppNumRxQueues; i++) + tp->rx_ring[i].rdsar_reg = (u16)(RDSAR_Q1_LOW_8125 + (i - 1) * 8); + + tp->isr_reg[0] = ISR0_8125; + for (i = 1; i < tp->hw_supp_irq_nvecs; i++) + tp->isr_reg[i] = (u16)(ISR1_8125 + (i - 1) * 4); + + tp->imr_reg[0] = IMR0_8125; + for (i = 1; i < tp->hw_supp_irq_nvecs; i++) + tp->imr_reg[i] = (u16)(IMR1_8125 + (i - 1) * 4); +} + +static void +rtl8127_init_software_variable(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct pci_dev *pdev = tp->pci_dev; + +#ifdef ENABLE_LIB_SUPPORT + tp->ring_lib_enabled = 1; +#endif + + switch (tp->mcfg) { + default: + tp->HwSuppDashVer = 0; + break; + } + tp->AllowAccessDashOcp = rtl8127_is_allow_access_dash_ocp(tp); + + tp->HwPkgDet = rtl8127_mac_ocp_read(tp, 0xDC00); + tp->HwPkgDet = (tp->HwPkgDet >> 3) & 0x07; + + if (HW_DASH_SUPPORT_TYPE_3(tp) && tp->HwPkgDet == 0x06) + eee_enable = 0; + + tp->HwSuppNowIsOobVer = 1; + + tp->HwPcieSNOffset = 0x168; + +#ifdef ENABLE_REALWOW_SUPPORT + rtl8127_get_realwow_hw_version(dev); +#endif //ENABLE_REALWOW_SUPPORT + + if (HW_DASH_SUPPORT_DASH(tp) && rtl8127_check_dash(tp)) + tp->DASH = 1; + else + tp->DASH = 0; + + if (tp->DASH) { + if (HW_DASH_SUPPORT_TYPE_3(tp)) { + u64 CmacMemPhysAddress; + void __iomem *cmac_ioaddr = NULL; + + //map CMAC IO space + CmacMemPhysAddress = rtl8127_csi_other_fun_read(tp, 0, 0x18); + if (!(CmacMemPhysAddress & BIT_0)) { + if (CmacMemPhysAddress & BIT_2) + CmacMemPhysAddress |= (u64)rtl8127_csi_other_fun_read(tp, 0, 0x1C) << 32; + + CmacMemPhysAddress &= 0xFFFFFFF0; + /* ioremap MMIO region */ + cmac_ioaddr = ioremap(CmacMemPhysAddress, R8127_REGS_SIZE); + } + + if (cmac_ioaddr == NULL) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "cannot remap CMAC MMIO, aborting\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + } + + if (cmac_ioaddr == NULL) + tp->DASH = 0; + else + tp->mapped_cmac_ioaddr = cmac_ioaddr; + } + + eee_enable = 0; + } + + if (HW_DASH_SUPPORT_TYPE_3(tp)) + tp->cmac_ioaddr = tp->mapped_cmac_ioaddr; + + if (aspm) { + tp->org_pci_offset_99 = rtl8127_csi_fun0_read_byte(tp, 0x99); + tp->org_pci_offset_99 &= ~(BIT_5|BIT_6); + + tp->org_pci_offset_180 = rtl8127_csi_fun0_read_byte(tp, 0x22c); + } + + pci_read_config_byte(pdev, 0x80, &tp->org_pci_offset_80); + pci_read_config_byte(pdev, 0x81, &tp->org_pci_offset_81); + + tp->use_timer_interrupt = TRUE; + + tp->HwSuppMaxPhyLinkSpeed = 10000; + + if (timer_count == 0 || tp->mcfg == CFG_METHOD_DEFAULT) + tp->use_timer_interrupt = FALSE; + + tp->ShortPacketSwChecksum = TRUE; + tp->UseSwPaddingShortPkt = TRUE; + + tp->HwSuppMagicPktVer = WAKEUP_MAGIC_PACKET_V3; + + tp->HwSuppLinkChgWakeUpVer = 3; + + tp->HwSuppD0SpeedUpVer = 2; + + tp->HwSuppCheckPhyDisableModeVer = 3; + + tp->HwSuppTxNoCloseVer = 6; + + switch (tp->HwSuppTxNoCloseVer) { + case 5: + case 6: + tp->MaxTxDescPtrMask = MAX_TX_NO_CLOSE_DESC_PTR_MASK_V4; + break; + case 4: + tp->MaxTxDescPtrMask = MAX_TX_NO_CLOSE_DESC_PTR_MASK_V3; + break; + case 3: + tp->MaxTxDescPtrMask = MAX_TX_NO_CLOSE_DESC_PTR_MASK_V2; + break; + default: + tx_no_close_enable = 0; + break; + } + + if (tp->HwSuppTxNoCloseVer > 0 && tx_no_close_enable == 1) + tp->EnableTxNoClose = TRUE; + + switch (tp->mcfg) { + case CFG_METHOD_1: + tp->sw_ram_code_ver = NIC_RAMCODE_VERSION_CFG_METHOD_1; + break; + case CFG_METHOD_2: + tp->sw_ram_code_ver = NIC_RAMCODE_VERSION_CFG_METHOD_2; + break; + default: + break; + } + + if (tp->HwIcVerUnknown) { + tp->NotWrRamCodeToMicroP = TRUE; + tp->NotWrMcuPatchCode = TRUE; + } + + tp->HwSuppMacMcuVer = 2; + + tp->MacMcuPageSize = RTL8127_MAC_MCU_PAGE_SIZE; + + tp->HwSuppNumTxQueues = 2; + tp->HwSuppNumRxQueues = 4; + + //init interrupt + tp->HwSuppIsrVer = 6; + + tp->HwCurrIsrVer = tp->HwSuppIsrVer; + if (tp->HwCurrIsrVer > 1) { + if (!(tp->features & RTL_FEATURE_MSIX) || + tp->irq_nvecs < tp->min_irq_nvecs) + tp->HwCurrIsrVer = 1; + } + + tp->num_tx_rings = 1; +#ifdef ENABLE_MULTIPLE_TX_QUEUE +#ifndef ENABLE_LIB_SUPPORT + tp->num_tx_rings = tp->HwSuppNumTxQueues; +#endif +#endif + if (tp->HwCurrIsrVer < 2 || + (tp->HwCurrIsrVer == 2 && tp->irq_nvecs < 19)) + tp->num_tx_rings = 1; + + //RSS + tp->HwSuppRssVer = 5; + tp->HwSuppIndirTblEntries = 128; + + tp->num_rx_rings = 1; +#ifdef ENABLE_RSS_SUPPORT +#ifdef ENABLE_LIB_SUPPORT + if (tp->HwSuppRssVer > 0) + tp->EnableRss = 1; +#else + if (tp->HwSuppRssVer > 0 && tp->HwCurrIsrVer > 1) { + u8 rss_queue_num = netif_get_num_default_rss_queues(); + tp->num_rx_rings = (tp->HwSuppNumRxQueues > rss_queue_num)? + rss_queue_num : tp->HwSuppNumRxQueues; + + if (!(tp->num_rx_rings >= 2 && tp->irq_nvecs >= tp->num_rx_rings)) + tp->num_rx_rings = 1; + + if (tp->num_rx_rings >= 2) + tp->EnableRss = 1; + } +#endif +#endif + + //interrupt mask + rtl8127_setup_interrupt_mask(tp); + + rtl8127_setup_mqs_reg(tp); + + rtl8127_set_ring_size(tp, NUM_RX_DESC, NUM_TX_DESC); + + tp->HwSuppPtpVer = 2; +#ifdef ENABLE_PTP_SUPPORT + if (tp->HwSuppPtpVer > 0) + tp->EnablePtp = 1; +#endif + + tp->HwSuppIntMitiVer = 6; + + tp->HwSuppTcamVer = 2; + + tp->TcamNotValidReg = TCAM_NOTVALID_ADDR_V2; + tp->TcamValidReg = TCAM_VALID_ADDR_V2; + tp->TcamMaAddrcOffset = TCAM_MAC_ADDR_V2; + tp->TcamVlanTagOffset = TCAM_VLAN_TAG_V2; + + tp->HwSuppExtendTallyCounterVer = 1; + + timer_count_v2 = (timer_count / 0x200); + + tp->HwSuppRxDescType = RX_DESC_RING_TYPE_4; + + tp->InitRxDescType = RX_DESC_RING_TYPE_1; + tp->RxDescLength = RX_DESC_LEN_TYPE_1; + switch (tp->HwSuppRxDescType) { + case RX_DESC_RING_TYPE_3: + if (tp->EnableRss) { + tp->InitRxDescType = RX_DESC_RING_TYPE_3; + tp->RxDescLength = RX_DESC_LEN_TYPE_3; + } + break; + case RX_DESC_RING_TYPE_4: + if (tp->EnableRss) { + tp->InitRxDescType = RX_DESC_RING_TYPE_4; + tp->RxDescLength = RX_DESC_LEN_TYPE_4; + } + break; + } + + tp->rtl8127_rx_config = rtl_chip_info[tp->chipset].RCR_Cfg; + if (tp->InitRxDescType == RX_DESC_RING_TYPE_3) + tp->rtl8127_rx_config |= EnableRxDescV3; + else if (tp->InitRxDescType == RX_DESC_RING_TYPE_4) + tp->rtl8127_rx_config &= ~EnableRxDescV4_1; + + tp->NicCustLedValue = RTL_R16(tp, CustomLED); + + tp->wol_opts = rtl8127_get_hw_wol(tp); + tp->wol_enabled = (tp->wol_opts) ? WOL_ENABLED : WOL_DISABLED; + + rtl8127_set_link_option(tp, autoneg_mode, speed_mode, duplex_mode, + rtl8127_fc_full); + + tp->max_jumbo_frame_size = rtl_chip_info[tp->chipset].jumbo_frame_sz; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + /* MTU range: 60 - hw-specific max */ + dev->min_mtu = ETH_MIN_MTU; + dev->max_mtu = tp->max_jumbo_frame_size; +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + + if (tp->mcfg != CFG_METHOD_DEFAULT) { + struct ethtool_keee *eee = &tp->eee; + + eee->eee_enabled = eee_enable; +#if LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0) + eee->supported = SUPPORTED_100baseT_Full | + SUPPORTED_1000baseT_Full | + SUPPORTED_2500baseX_Full; + eee->advertised = mmd_eee_adv_to_ethtool_adv_t(MDIO_EEE_1000T | MDIO_EEE_100TX); + eee->advertised |= SUPPORTED_2500baseX_Full; +#else + linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Full_BIT, eee->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, eee->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, eee->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, eee->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, eee->supported); + linkmode_set_bit(ETHTOOL_LINK_MODE_100baseT_Full_BIT, eee->advertised); + linkmode_set_bit(ETHTOOL_LINK_MODE_1000baseT_Full_BIT, eee->advertised); + linkmode_set_bit(ETHTOOL_LINK_MODE_2500baseT_Full_BIT, eee->advertised); + linkmode_set_bit(ETHTOOL_LINK_MODE_5000baseT_Full_BIT, eee->advertised); + linkmode_set_bit(ETHTOOL_LINK_MODE_10000baseT_Full_BIT, eee->advertised); +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0) */ + eee->tx_lpi_enabled = eee_enable; + eee->tx_lpi_timer = dev->mtu + ETH_HLEN + 0x20; + } + +#ifdef ENABLE_RSS_SUPPORT + if (tp->EnableRss) + rtl8127_init_rss(tp); +#endif +} + +static void +rtl8127_release_board(struct pci_dev *pdev, + struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + void __iomem *ioaddr = tp->mmio_addr; + + rtl8127_rar_set(tp, tp->org_mac_addr); + tp->wol_enabled = WOL_DISABLED; + + if (!tp->DASH) + rtl8127_phy_power_down(dev); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + FreeAllocatedDashShareMemory(dev); +#endif + + if (tp->mapped_cmac_ioaddr != NULL) + iounmap(tp->mapped_cmac_ioaddr); + + iounmap(ioaddr); + pci_release_regions(pdev); + pci_clear_mwi(pdev); + pci_disable_device(pdev); + free_netdev(dev); +} + +static void +rtl8127_hw_address_set(struct net_device *dev, u8 mac_addr[MAC_ADDR_LEN]) +{ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) + eth_hw_addr_set(dev, mac_addr); +#else + memcpy(dev->dev_addr, mac_addr, MAC_ADDR_LEN); +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,17,0) +} + +static int +rtl8127_get_mac_address(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + u8 mac_addr[MAC_ADDR_LEN]; + + for (i = 0; i < MAC_ADDR_LEN; i++) + mac_addr[i] = RTL_R8(tp, MAC0 + i); + + *(u32*)&mac_addr[0] = RTL_R32(tp, BACKUP_ADDR0_8125); + *(u16*)&mac_addr[4] = RTL_R16(tp, BACKUP_ADDR1_8125); + + if (!is_valid_ether_addr(mac_addr)) { + netif_err(tp, probe, dev, "Invalid ether addr %pM\n", + mac_addr); + eth_random_addr(mac_addr); + dev->addr_assign_type = NET_ADDR_RANDOM; + netif_info(tp, probe, dev, "Random ether addr %pM\n", + mac_addr); + tp->random_mac = 1; + } + + rtl8127_hw_address_set(dev, mac_addr); + rtl8127_rar_set(tp, mac_addr); + + /* keep the original MAC address */ + memcpy(tp->org_mac_addr, dev->dev_addr, MAC_ADDR_LEN); +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + memcpy(dev->perm_addr, dev->dev_addr, MAC_ADDR_LEN); +#endif + return 0; +} + +/** + * rtl8127_set_mac_address - Change the Ethernet Address of the NIC + * @dev: network interface device structure + * @p: pointer to an address structure + * + * Return 0 on success, negative on failure + **/ +static int +rtl8127_set_mac_address(struct net_device *dev, + void *p) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct sockaddr *addr = p; + + if (!is_valid_ether_addr(addr->sa_data)) + return -EADDRNOTAVAIL; + + rtl8127_hw_address_set(dev, addr->sa_data); + + rtl8127_rar_set(tp, dev->dev_addr); + + return 0; +} + +/****************************************************************************** + * rtl8127_rar_set - Puts an ethernet address into a receive address register. + * + * tp - The private data structure for driver + * addr - Address to put into receive address register + *****************************************************************************/ +void +rtl8127_rar_set(struct rtl8127_private *tp, + const u8 *addr) +{ + uint32_t rar_low = 0; + uint32_t rar_high = 0; + + rar_low = ((uint32_t) addr[0] | + ((uint32_t) addr[1] << 8) | + ((uint32_t) addr[2] << 16) | + ((uint32_t) addr[3] << 24)); + + rar_high = ((uint32_t) addr[4] | + ((uint32_t) addr[5] << 8)); + + rtl8127_enable_cfg9346_write(tp); + RTL_W32(tp, MAC0, rar_low); + RTL_W32(tp, MAC4, rar_high); + + rtl8127_disable_cfg9346_write(tp); +} + +#ifdef ETHTOOL_OPS_COMPAT +static int ethtool_get_settings(struct net_device *dev, void *useraddr) +{ + struct ethtool_cmd cmd = { ETHTOOL_GSET }; + int err; + + if (!ethtool_ops->get_settings) + return -EOPNOTSUPP; + + err = ethtool_ops->get_settings(dev, &cmd); + if (err < 0) + return err; + + if (copy_to_user(useraddr, &cmd, sizeof(cmd))) + return -EFAULT; + return 0; +} + +static int ethtool_set_settings(struct net_device *dev, void *useraddr) +{ + struct ethtool_cmd cmd; + + if (!ethtool_ops->set_settings) + return -EOPNOTSUPP; + + if (copy_from_user(&cmd, useraddr, sizeof(cmd))) + return -EFAULT; + + return ethtool_ops->set_settings(dev, &cmd); +} + +static int ethtool_get_drvinfo(struct net_device *dev, void *useraddr) +{ + struct ethtool_drvinfo info; + struct ethtool_ops *ops = ethtool_ops; + + if (!ops->get_drvinfo) + return -EOPNOTSUPP; + + memset(&info, 0, sizeof(info)); + info.cmd = ETHTOOL_GDRVINFO; + ops->get_drvinfo(dev, &info); + + if (ops->self_test_count) + info.testinfo_len = ops->self_test_count(dev); + if (ops->get_stats_count) + info.n_stats = ops->get_stats_count(dev); + if (ops->get_regs_len) + info.regdump_len = ops->get_regs_len(dev); + if (ops->get_eeprom_len) + info.eedump_len = ops->get_eeprom_len(dev); + + if (copy_to_user(useraddr, &info, sizeof(info))) + return -EFAULT; + return 0; +} + +static int ethtool_get_regs(struct net_device *dev, char *useraddr) +{ + struct ethtool_regs regs; + struct ethtool_ops *ops = ethtool_ops; + void *regbuf; + int reglen, ret; + + if (!ops->get_regs || !ops->get_regs_len) + return -EOPNOTSUPP; + + if (copy_from_user(®s, useraddr, sizeof(regs))) + return -EFAULT; + + reglen = ops->get_regs_len(dev); + if (regs.len > reglen) + regs.len = reglen; + + regbuf = kmalloc(reglen, GFP_USER); + if (!regbuf) + return -ENOMEM; + + ops->get_regs(dev, ®s, regbuf); + + ret = -EFAULT; + if (copy_to_user(useraddr, ®s, sizeof(regs))) + goto out; + useraddr += offsetof(struct ethtool_regs, data); + if (copy_to_user(useraddr, regbuf, reglen)) + goto out; + ret = 0; + +out: + kfree(regbuf); + return ret; +} + +static int ethtool_get_wol(struct net_device *dev, char *useraddr) +{ + struct ethtool_wolinfo wol = { ETHTOOL_GWOL }; + + if (!ethtool_ops->get_wol) + return -EOPNOTSUPP; + + ethtool_ops->get_wol(dev, &wol); + + if (copy_to_user(useraddr, &wol, sizeof(wol))) + return -EFAULT; + return 0; +} + +static int ethtool_set_wol(struct net_device *dev, char *useraddr) +{ + struct ethtool_wolinfo wol; + + if (!ethtool_ops->set_wol) + return -EOPNOTSUPP; + + if (copy_from_user(&wol, useraddr, sizeof(wol))) + return -EFAULT; + + return ethtool_ops->set_wol(dev, &wol); +} + +static int ethtool_get_msglevel(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GMSGLVL }; + + if (!ethtool_ops->get_msglevel) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_msglevel(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_set_msglevel(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata; + + if (!ethtool_ops->set_msglevel) + return -EOPNOTSUPP; + + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + + ethtool_ops->set_msglevel(dev, edata.data); + return 0; +} + +static int ethtool_nway_reset(struct net_device *dev) +{ + if (!ethtool_ops->nway_reset) + return -EOPNOTSUPP; + + return ethtool_ops->nway_reset(dev); +} + +static int ethtool_get_link(struct net_device *dev, void *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GLINK }; + + if (!ethtool_ops->get_link) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_link(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_get_eeprom(struct net_device *dev, void *useraddr) +{ + struct ethtool_eeprom eeprom; + struct ethtool_ops *ops = ethtool_ops; + u8 *data; + int ret; + + if (!ops->get_eeprom || !ops->get_eeprom_len) + return -EOPNOTSUPP; + + if (copy_from_user(&eeprom, useraddr, sizeof(eeprom))) + return -EFAULT; + + /* Check for wrap and zero */ + if (eeprom.offset + eeprom.len <= eeprom.offset) + return -EINVAL; + + /* Check for exceeding total eeprom len */ + if (eeprom.offset + eeprom.len > ops->get_eeprom_len(dev)) + return -EINVAL; + + data = kmalloc(eeprom.len, GFP_USER); + if (!data) + return -ENOMEM; + + ret = -EFAULT; + if (copy_from_user(data, useraddr + sizeof(eeprom), eeprom.len)) + goto out; + + ret = ops->get_eeprom(dev, &eeprom, data); + if (ret) + goto out; + + ret = -EFAULT; + if (copy_to_user(useraddr, &eeprom, sizeof(eeprom))) + goto out; + if (copy_to_user(useraddr + sizeof(eeprom), data, eeprom.len)) + goto out; + ret = 0; + +out: + kfree(data); + return ret; +} + +static int ethtool_set_eeprom(struct net_device *dev, void *useraddr) +{ + struct ethtool_eeprom eeprom; + struct ethtool_ops *ops = ethtool_ops; + u8 *data; + int ret; + + if (!ops->set_eeprom || !ops->get_eeprom_len) + return -EOPNOTSUPP; + + if (copy_from_user(&eeprom, useraddr, sizeof(eeprom))) + return -EFAULT; + + /* Check for wrap and zero */ + if (eeprom.offset + eeprom.len <= eeprom.offset) + return -EINVAL; + + /* Check for exceeding total eeprom len */ + if (eeprom.offset + eeprom.len > ops->get_eeprom_len(dev)) + return -EINVAL; + + data = kmalloc(eeprom.len, GFP_USER); + if (!data) + return -ENOMEM; + + ret = -EFAULT; + if (copy_from_user(data, useraddr + sizeof(eeprom), eeprom.len)) + goto out; + + ret = ops->set_eeprom(dev, &eeprom, data); + if (ret) + goto out; + + if (copy_to_user(useraddr + sizeof(eeprom), data, eeprom.len)) + ret = -EFAULT; + +out: + kfree(data); + return ret; +} + +static int ethtool_get_coalesce(struct net_device *dev, void *useraddr) +{ + struct ethtool_coalesce coalesce = { ETHTOOL_GCOALESCE }; + + if (!ethtool_ops->get_coalesce) + return -EOPNOTSUPP; + + ethtool_ops->get_coalesce(dev, &coalesce); + + if (copy_to_user(useraddr, &coalesce, sizeof(coalesce))) + return -EFAULT; + return 0; +} + +static int ethtool_set_coalesce(struct net_device *dev, void *useraddr) +{ + struct ethtool_coalesce coalesce; + + if (!ethtool_ops->get_coalesce) + return -EOPNOTSUPP; + + if (copy_from_user(&coalesce, useraddr, sizeof(coalesce))) + return -EFAULT; + + return ethtool_ops->set_coalesce(dev, &coalesce); +} + +static int ethtool_get_ringparam(struct net_device *dev, void *useraddr) +{ + struct ethtool_ringparam ringparam = { ETHTOOL_GRINGPARAM }; + + if (!ethtool_ops->get_ringparam) + return -EOPNOTSUPP; + + ethtool_ops->get_ringparam(dev, &ringparam); + + if (copy_to_user(useraddr, &ringparam, sizeof(ringparam))) + return -EFAULT; + return 0; +} + +static int ethtool_set_ringparam(struct net_device *dev, void *useraddr) +{ + struct ethtool_ringparam ringparam; + + if (!ethtool_ops->get_ringparam) + return -EOPNOTSUPP; + + if (copy_from_user(&ringparam, useraddr, sizeof(ringparam))) + return -EFAULT; + + return ethtool_ops->set_ringparam(dev, &ringparam); +} + +static int ethtool_get_pauseparam(struct net_device *dev, void *useraddr) +{ + struct ethtool_pauseparam pauseparam = { ETHTOOL_GPAUSEPARAM }; + + if (!ethtool_ops->get_pauseparam) + return -EOPNOTSUPP; + + ethtool_ops->get_pauseparam(dev, &pauseparam); + + if (copy_to_user(useraddr, &pauseparam, sizeof(pauseparam))) + return -EFAULT; + return 0; +} + +static int ethtool_set_pauseparam(struct net_device *dev, void *useraddr) +{ + struct ethtool_pauseparam pauseparam; + + if (!ethtool_ops->get_pauseparam) + return -EOPNOTSUPP; + + if (copy_from_user(&pauseparam, useraddr, sizeof(pauseparam))) + return -EFAULT; + + return ethtool_ops->set_pauseparam(dev, &pauseparam); +} + +static int ethtool_get_rx_csum(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GRXCSUM }; + + if (!ethtool_ops->get_rx_csum) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_rx_csum(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_set_rx_csum(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata; + + if (!ethtool_ops->set_rx_csum) + return -EOPNOTSUPP; + + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + + ethtool_ops->set_rx_csum(dev, edata.data); + return 0; +} + +static int ethtool_get_tx_csum(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GTXCSUM }; + + if (!ethtool_ops->get_tx_csum) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_tx_csum(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_set_tx_csum(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata; + + if (!ethtool_ops->set_tx_csum) + return -EOPNOTSUPP; + + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + + return ethtool_ops->set_tx_csum(dev, edata.data); +} + +static int ethtool_get_sg(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GSG }; + + if (!ethtool_ops->get_sg) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_sg(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_set_sg(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata; + + if (!ethtool_ops->set_sg) + return -EOPNOTSUPP; + + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + + return ethtool_ops->set_sg(dev, edata.data); +} + +static int ethtool_get_tso(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata = { ETHTOOL_GTSO }; + + if (!ethtool_ops->get_tso) + return -EOPNOTSUPP; + + edata.data = ethtool_ops->get_tso(dev); + + if (copy_to_user(useraddr, &edata, sizeof(edata))) + return -EFAULT; + return 0; +} + +static int ethtool_set_tso(struct net_device *dev, char *useraddr) +{ + struct ethtool_value edata; + + if (!ethtool_ops->set_tso) + return -EOPNOTSUPP; + + if (copy_from_user(&edata, useraddr, sizeof(edata))) + return -EFAULT; + + return ethtool_ops->set_tso(dev, edata.data); +} + +static int ethtool_self_test(struct net_device *dev, char *useraddr) +{ + struct ethtool_test test; + struct ethtool_ops *ops = ethtool_ops; + u64 *data; + int ret; + + if (!ops->self_test || !ops->self_test_count) + return -EOPNOTSUPP; + + if (copy_from_user(&test, useraddr, sizeof(test))) + return -EFAULT; + + test.len = ops->self_test_count(dev); + data = kmalloc(test.len * sizeof(u64), GFP_USER); + if (!data) + return -ENOMEM; + + ops->self_test(dev, &test, data); + + ret = -EFAULT; + if (copy_to_user(useraddr, &test, sizeof(test))) + goto out; + useraddr += sizeof(test); + if (copy_to_user(useraddr, data, test.len * sizeof(u64))) + goto out; + ret = 0; + +out: + kfree(data); + return ret; +} + +static int ethtool_get_strings(struct net_device *dev, void *useraddr) +{ + struct ethtool_gstrings gstrings; + struct ethtool_ops *ops = ethtool_ops; + u8 *data; + int ret; + + if (!ops->get_strings) + return -EOPNOTSUPP; + + if (copy_from_user(&gstrings, useraddr, sizeof(gstrings))) + return -EFAULT; + + switch (gstrings.string_set) { + case ETH_SS_TEST: + if (!ops->self_test_count) + return -EOPNOTSUPP; + gstrings.len = ops->self_test_count(dev); + break; + case ETH_SS_STATS: + if (!ops->get_stats_count) + return -EOPNOTSUPP; + gstrings.len = ops->get_stats_count(dev); + break; + default: + return -EINVAL; + } + + data = kmalloc(gstrings.len * ETH_GSTRING_LEN, GFP_USER); + if (!data) + return -ENOMEM; + + ops->get_strings(dev, gstrings.string_set, data); + + ret = -EFAULT; + if (copy_to_user(useraddr, &gstrings, sizeof(gstrings))) + goto out; + useraddr += sizeof(gstrings); + if (copy_to_user(useraddr, data, gstrings.len * ETH_GSTRING_LEN)) + goto out; + ret = 0; + +out: + kfree(data); + return ret; +} + +static int ethtool_phys_id(struct net_device *dev, void *useraddr) +{ + struct ethtool_value id; + + if (!ethtool_ops->phys_id) + return -EOPNOTSUPP; + + if (copy_from_user(&id, useraddr, sizeof(id))) + return -EFAULT; + + return ethtool_ops->phys_id(dev, id.data); +} + +static int ethtool_get_stats(struct net_device *dev, void *useraddr) +{ + struct ethtool_stats stats; + struct ethtool_ops *ops = ethtool_ops; + u64 *data; + int ret; + + if (!ops->get_ethtool_stats || !ops->get_stats_count) + return -EOPNOTSUPP; + + if (copy_from_user(&stats, useraddr, sizeof(stats))) + return -EFAULT; + + stats.n_stats = ops->get_stats_count(dev); + data = kmalloc(stats.n_stats * sizeof(u64), GFP_USER); + if (!data) + return -ENOMEM; + + ops->get_ethtool_stats(dev, &stats, data); + + ret = -EFAULT; + if (copy_to_user(useraddr, &stats, sizeof(stats))) + goto out; + useraddr += sizeof(stats); + if (copy_to_user(useraddr, data, stats.n_stats * sizeof(u64))) + goto out; + ret = 0; + +out: + kfree(data); + return ret; +} + +static int ethtool_ioctl(struct ifreq *ifr) +{ + struct net_device *dev = __dev_get_by_name(ifr->ifr_name); + void *useraddr = (void *) ifr->ifr_data; + u32 ethcmd; + + /* + * XXX: This can be pushed down into the ethtool_* handlers that + * need it. Keep existing behaviour for the moment. + */ + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + if (!dev || !netif_device_present(dev)) + return -ENODEV; + + if (copy_from_user(ðcmd, useraddr, sizeof (ethcmd))) + return -EFAULT; + + switch (ethcmd) { + case ETHTOOL_GSET: + return ethtool_get_settings(dev, useraddr); + case ETHTOOL_SSET: + return ethtool_set_settings(dev, useraddr); + case ETHTOOL_GDRVINFO: + return ethtool_get_drvinfo(dev, useraddr); + case ETHTOOL_GREGS: + return ethtool_get_regs(dev, useraddr); + case ETHTOOL_GWOL: + return ethtool_get_wol(dev, useraddr); + case ETHTOOL_SWOL: + return ethtool_set_wol(dev, useraddr); + case ETHTOOL_GMSGLVL: + return ethtool_get_msglevel(dev, useraddr); + case ETHTOOL_SMSGLVL: + return ethtool_set_msglevel(dev, useraddr); + case ETHTOOL_NWAY_RST: + return ethtool_nway_reset(dev); + case ETHTOOL_GLINK: + return ethtool_get_link(dev, useraddr); + case ETHTOOL_GEEPROM: + return ethtool_get_eeprom(dev, useraddr); + case ETHTOOL_SEEPROM: + return ethtool_set_eeprom(dev, useraddr); + case ETHTOOL_GCOALESCE: + return ethtool_get_coalesce(dev, useraddr); + case ETHTOOL_SCOALESCE: + return ethtool_set_coalesce(dev, useraddr); + case ETHTOOL_GRINGPARAM: + return ethtool_get_ringparam(dev, useraddr); + case ETHTOOL_SRINGPARAM: + return ethtool_set_ringparam(dev, useraddr); + case ETHTOOL_GPAUSEPARAM: + return ethtool_get_pauseparam(dev, useraddr); + case ETHTOOL_SPAUSEPARAM: + return ethtool_set_pauseparam(dev, useraddr); + case ETHTOOL_GRXCSUM: + return ethtool_get_rx_csum(dev, useraddr); + case ETHTOOL_SRXCSUM: + return ethtool_set_rx_csum(dev, useraddr); + case ETHTOOL_GTXCSUM: + return ethtool_get_tx_csum(dev, useraddr); + case ETHTOOL_STXCSUM: + return ethtool_set_tx_csum(dev, useraddr); + case ETHTOOL_GSG: + return ethtool_get_sg(dev, useraddr); + case ETHTOOL_SSG: + return ethtool_set_sg(dev, useraddr); + case ETHTOOL_GTSO: + return ethtool_get_tso(dev, useraddr); + case ETHTOOL_STSO: + return ethtool_set_tso(dev, useraddr); + case ETHTOOL_TEST: + return ethtool_self_test(dev, useraddr); + case ETHTOOL_GSTRINGS: + return ethtool_get_strings(dev, useraddr); + case ETHTOOL_PHYS_ID: + return ethtool_phys_id(dev, useraddr); + case ETHTOOL_GSTATS: + return ethtool_get_stats(dev, useraddr); + default: + return -EOPNOTSUPP; + } + + return -EOPNOTSUPP; +} +#endif //ETHTOOL_OPS_COMPAT + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,15,0) +static int rtl8127_siocdevprivate(struct net_device *dev, struct ifreq *ifr, + void __user *data, int cmd) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret = 0; + + switch (cmd) { +#ifdef ENABLE_DASH_SUPPORT + case SIOCDEVPRIVATE_RTLDASH: + if (!netif_running(dev)) { + ret = -ENODEV; + break; + } + if (!capable(CAP_NET_ADMIN)) { + ret = -EPERM; + break; + } + + ret = rtl8127_dash_ioctl(dev, ifr); + break; +#endif + +#ifdef ENABLE_REALWOW_SUPPORT + case SIOCDEVPRIVATE_RTLREALWOW: + if (!netif_running(dev)) { + ret = -ENODEV; + break; + } + + ret = rtl8127_realwow_ioctl(dev, ifr); + break; +#endif + + case SIOCRTLTOOL: + if (!capable(CAP_NET_ADMIN)) { + ret = -EPERM; + break; + } + + ret = rtl8127_tool_ioctl(tp, ifr); + break; + + default: + ret = -EOPNOTSUPP; + } + + return ret; +} +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,15,0) + +static int +rtl8127_do_ioctl(struct net_device *dev, + struct ifreq *ifr, + int cmd) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct mii_ioctl_data *data = if_mii(ifr); + int ret = 0; + + switch (cmd) { + case SIOCGMIIPHY: + data->phy_id = 32; /* Internal PHY */ + break; + + case SIOCGMIIREG: + rtl8127_mdio_write(tp, 0x1F, 0x0000); + data->val_out = rtl8127_mdio_read(tp, data->reg_num); + break; + + case SIOCSMIIREG: + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + rtl8127_mdio_write(tp, 0x1F, 0x0000); + rtl8127_mdio_write(tp, data->reg_num, data->val_in); + break; + +#ifdef ETHTOOL_OPS_COMPAT + case SIOCETHTOOL: + ret = ethtool_ioctl(ifr); + break; +#endif + +#ifdef ENABLE_PTP_SUPPORT + case SIOCSHWTSTAMP: + case SIOCGHWTSTAMP: + if (tp->EnablePtp) + ret = rtl8127_ptp_ioctl(dev, ifr, cmd); + else + ret = -EOPNOTSUPP; + break; +#endif +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,15,0) +#ifdef ENABLE_DASH_SUPPORT + case SIOCDEVPRIVATE_RTLDASH: + if (!netif_running(dev)) { + ret = -ENODEV; + break; + } + if (!capable(CAP_NET_ADMIN)) { + ret = -EPERM; + break; + } + + ret = rtl8127_dash_ioctl(dev, ifr); + break; +#endif + +#ifdef ENABLE_REALWOW_SUPPORT + case SIOCDEVPRIVATE_RTLREALWOW: + if (!netif_running(dev)) { + ret = -ENODEV; + break; + } + + if (!capable(CAP_NET_ADMIN)) { + ret = -EPERM; + break; + } + + ret = rtl8127_realwow_ioctl(dev, ifr); + break; +#endif + + case SIOCRTLTOOL: + if (!capable(CAP_NET_ADMIN)) { + ret = -EPERM; + break; + } + + ret = rtl8127_tool_ioctl(tp, ifr); + break; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(5,15,0) + + default: + ret = -EOPNOTSUPP; + break; + } + + return ret; +} + +static void +rtl8127_phy_power_up(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + + if (rtl8127_is_in_phy_disable_mode(dev)) + return; + + spin_lock_irqsave(&tp->phy_lock, flags); + + rtl8127_mdio_write(tp, 0x1F, 0x0000); + rtl8127_mdio_write(tp, MII_BMCR, BMCR_ANENABLE); + + //wait ups resume (phy state 3) + rtl8127_wait_phy_ups_resume(dev, 3); + + spin_unlock_irqrestore(&tp->phy_lock, flags); +} + +static void +rtl8127_phy_power_down(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + rtl8127_mdio_write(tp, 0x1F, 0x0000); + rtl8127_mdio_write(tp, MII_BMCR, BMCR_ANENABLE | BMCR_PDOWN); + spin_unlock_irqrestore(&tp->phy_lock, flags); +} + +static int __devinit +rtl8127_init_board(struct pci_dev *pdev, + struct net_device **dev_out, + void __iomem **ioaddr_out) +{ + void __iomem *ioaddr; + struct net_device *dev; + struct rtl8127_private *tp; + int rc = -ENOMEM, i, pm_cap; + + assert(ioaddr_out != NULL); + + /* dev zeroed in alloc_etherdev */ + dev = alloc_etherdev_mq(sizeof (*tp), R8127_MAX_QUEUES); + if (dev == NULL) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_drv(&debug)) + dev_err(&pdev->dev, "unable to alloc new ethernet\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + goto err_out; + } + + SET_MODULE_OWNER(dev); + SET_NETDEV_DEV(dev, &pdev->dev); + tp = netdev_priv(dev); + tp->dev = dev; + tp->pci_dev = pdev; + tp->msg_enable = netif_msg_init(debug.msg_enable, R8127_MSG_DEFAULT); + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,26) + if (!aspm) + pci_disable_link_state(pdev, PCIE_LINK_STATE_L0S | PCIE_LINK_STATE_L1 | + PCIE_LINK_STATE_CLKPM); +#endif + + /* enable device (incl. PCI PM wakeup and hotplug setup) */ + rc = pci_enable_device(pdev); + if (rc < 0) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "enable failure\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + goto err_out_free_dev; + } + + if (pci_set_mwi(pdev) < 0) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_drv(&debug)) + dev_info(&pdev->dev, "Mem-Wr-Inval unavailable.\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + } + + /* save power state before pci_enable_device overwrites it */ + pm_cap = pci_find_capability(pdev, PCI_CAP_ID_PM); + if (pm_cap) { + u16 pwr_command; + + pci_read_config_word(pdev, pm_cap + PCI_PM_CTRL, &pwr_command); + } else { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "PowerManagement capability not found.\n"); +#else + printk("PowerManagement capability not found.\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + + } + + /* make sure PCI base addr 1 is MMIO */ + if (!(pci_resource_flags(pdev, 2) & IORESOURCE_MEM)) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "region #1 not an MMIO resource, aborting\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + rc = -ENODEV; + goto err_out_mwi; + } + /* check for weird/broken PCI region reporting */ + if (pci_resource_len(pdev, 2) < R8127_REGS_SIZE) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "Invalid PCI region size(s), aborting\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + rc = -ENODEV; + goto err_out_mwi; + } + + rc = pci_request_regions(pdev, MODULENAME); + if (rc < 0) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "could not request regions.\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + goto err_out_mwi; + } + + if ((sizeof(dma_addr_t) > 4) && + use_dac && + !dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)) && + !dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64))) { + dev->features |= NETIF_F_HIGHDMA; + } else { + rc = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32)); + if (rc < 0) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "DMA configuration failed.\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + goto err_out_free_res; + } + } + + /* ioremap MMIO region */ + ioaddr = ioremap(pci_resource_start(pdev, 2), pci_resource_len(pdev, 2)); + if (ioaddr == NULL) { +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_err(&pdev->dev, "cannot remap MMIO, aborting\n"); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + rc = -EIO; + goto err_out_free_res; + } + + tp->mmio_addr = ioaddr; + + /* Identify chip attached to board */ + rtl8127_get_mac_version(tp); + + rtl8127_print_mac_version(tp); + + for (i = ARRAY_SIZE(rtl_chip_info) - 1; i >= 0; i--) { + if (tp->mcfg == rtl_chip_info[i].mcfg) + break; + } + + if (i < 0) { + /* Unknown chip: assume array element #0, original RTL-8125 */ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (netif_msg_probe(tp)) + dev_printk(KERN_DEBUG, &pdev->dev, "unknown chip version, assuming %s\n", rtl_chip_info[0].name); +#else + printk("Realtek unknown chip version, assuming %s\n", rtl_chip_info[0].name); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) + i++; + } + + tp->chipset = i; + + *ioaddr_out = ioaddr; + *dev_out = dev; +out: + return rc; + +err_out_free_res: + pci_release_regions(pdev); +err_out_mwi: + pci_clear_mwi(pdev); + pci_disable_device(pdev); +err_out_free_dev: + free_netdev(dev); +err_out: + *ioaddr_out = NULL; + *dev_out = NULL; + goto out; +} + +static void +rtl8127_esd_checker(struct rtl8127_private *tp) +{ + struct net_device *dev = tp->dev; + struct pci_dev *pdev = tp->pci_dev; + u8 cmd; + u16 io_base_l; + u16 mem_base_l; + u16 mem_base_h; + u8 ilr; + u16 resv_0x1c_h; + u16 resv_0x1c_l; + u16 resv_0x20_l; + u16 resv_0x20_h; + u16 resv_0x24_l; + u16 resv_0x24_h; + u16 resv_0x2c_h; + u16 resv_0x2c_l; + u32 pci_sn_l; + u32 pci_sn_h; + + if (unlikely(tp->rtk_enable_diag)) + goto exit; + + tp->esd_flag = 0; + + pci_read_config_byte(pdev, PCI_COMMAND, &cmd); + if (cmd != tp->pci_cfg_space.cmd) { + printk(KERN_ERR "%s: cmd = 0x%02x, should be 0x%02x \n.", dev->name, cmd, tp->pci_cfg_space.cmd); + pci_write_config_byte(pdev, PCI_COMMAND, tp->pci_cfg_space.cmd); + tp->esd_flag |= BIT_0; + + pci_read_config_byte(pdev, PCI_COMMAND, &cmd); + if (cmd == 0xff) { + printk(KERN_ERR "%s: pci link is down \n.", dev->name); + goto exit; + } + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_0, &io_base_l); + if (io_base_l != tp->pci_cfg_space.io_base_l) { + printk(KERN_ERR "%s: io_base_l = 0x%04x, should be 0x%04x \n.", dev->name, io_base_l, tp->pci_cfg_space.io_base_l); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_0, tp->pci_cfg_space.io_base_l); + tp->esd_flag |= BIT_1; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_2, &mem_base_l); + if (mem_base_l != tp->pci_cfg_space.mem_base_l) { + printk(KERN_ERR "%s: mem_base_l = 0x%04x, should be 0x%04x \n.", dev->name, mem_base_l, tp->pci_cfg_space.mem_base_l); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_2, tp->pci_cfg_space.mem_base_l); + tp->esd_flag |= BIT_2; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_2 + 2, &mem_base_h); + if (mem_base_h!= tp->pci_cfg_space.mem_base_h) { + printk(KERN_ERR "%s: mem_base_h = 0x%04x, should be 0x%04x \n.", dev->name, mem_base_h, tp->pci_cfg_space.mem_base_h); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_2 + 2, tp->pci_cfg_space.mem_base_h); + tp->esd_flag |= BIT_3; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_3, &resv_0x1c_l); + if (resv_0x1c_l != tp->pci_cfg_space.resv_0x1c_l) { + printk(KERN_ERR "%s: resv_0x1c_l = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x1c_l, tp->pci_cfg_space.resv_0x1c_l); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_3, tp->pci_cfg_space.resv_0x1c_l); + tp->esd_flag |= BIT_4; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_3 + 2, &resv_0x1c_h); + if (resv_0x1c_h != tp->pci_cfg_space.resv_0x1c_h) { + printk(KERN_ERR "%s: resv_0x1c_h = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x1c_h, tp->pci_cfg_space.resv_0x1c_h); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_3 + 2, tp->pci_cfg_space.resv_0x1c_h); + tp->esd_flag |= BIT_5; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_4, &resv_0x20_l); + if (resv_0x20_l != tp->pci_cfg_space.resv_0x20_l) { + printk(KERN_ERR "%s: resv_0x20_l = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x20_l, tp->pci_cfg_space.resv_0x20_l); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_4, tp->pci_cfg_space.resv_0x20_l); + tp->esd_flag |= BIT_6; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_4 + 2, &resv_0x20_h); + if (resv_0x20_h != tp->pci_cfg_space.resv_0x20_h) { + printk(KERN_ERR "%s: resv_0x20_h = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x20_h, tp->pci_cfg_space.resv_0x20_h); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_4 + 2, tp->pci_cfg_space.resv_0x20_h); + tp->esd_flag |= BIT_7; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_5, &resv_0x24_l); + if (resv_0x24_l != tp->pci_cfg_space.resv_0x24_l) { + printk(KERN_ERR "%s: resv_0x24_l = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x24_l, tp->pci_cfg_space.resv_0x24_l); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_5, tp->pci_cfg_space.resv_0x24_l); + tp->esd_flag |= BIT_8; + } + + pci_read_config_word(pdev, PCI_BASE_ADDRESS_5 + 2, &resv_0x24_h); + if (resv_0x24_h != tp->pci_cfg_space.resv_0x24_h) { + printk(KERN_ERR "%s: resv_0x24_h = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x24_h, tp->pci_cfg_space.resv_0x24_h); + pci_write_config_word(pdev, PCI_BASE_ADDRESS_5 + 2, tp->pci_cfg_space.resv_0x24_h); + tp->esd_flag |= BIT_9; + } + + pci_read_config_byte(pdev, PCI_INTERRUPT_LINE, &ilr); + if (ilr != tp->pci_cfg_space.ilr) { + printk(KERN_ERR "%s: ilr = 0x%02x, should be 0x%02x \n.", dev->name, ilr, tp->pci_cfg_space.ilr); + pci_write_config_byte(pdev, PCI_INTERRUPT_LINE, tp->pci_cfg_space.ilr); + tp->esd_flag |= BIT_10; + } + + pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &resv_0x2c_l); + if (resv_0x2c_l != tp->pci_cfg_space.resv_0x2c_l) { + printk(KERN_ERR "%s: resv_0x2c_l = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x2c_l, tp->pci_cfg_space.resv_0x2c_l); + pci_write_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, tp->pci_cfg_space.resv_0x2c_l); + tp->esd_flag |= BIT_11; + } + + pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID + 2, &resv_0x2c_h); + if (resv_0x2c_h != tp->pci_cfg_space.resv_0x2c_h) { + printk(KERN_ERR "%s: resv_0x2c_h = 0x%04x, should be 0x%04x \n.", dev->name, resv_0x2c_h, tp->pci_cfg_space.resv_0x2c_h); + pci_write_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID + 2, tp->pci_cfg_space.resv_0x2c_h); + tp->esd_flag |= BIT_12; + } + + if (tp->HwPcieSNOffset > 0) { + pci_sn_l = rtl8127_csi_read(tp, tp->HwPcieSNOffset); + if (pci_sn_l != tp->pci_cfg_space.pci_sn_l) { + printk(KERN_ERR "%s: pci_sn_l = 0x%08x, should be 0x%08x \n.", dev->name, pci_sn_l, tp->pci_cfg_space.pci_sn_l); + rtl8127_csi_write(tp, tp->HwPcieSNOffset, tp->pci_cfg_space.pci_sn_l); + tp->esd_flag |= BIT_13; + } + + pci_sn_h = rtl8127_csi_read(tp, tp->HwPcieSNOffset + 4); + if (pci_sn_h != tp->pci_cfg_space.pci_sn_h) { + printk(KERN_ERR "%s: pci_sn_h = 0x%08x, should be 0x%08x \n.", dev->name, pci_sn_h, tp->pci_cfg_space.pci_sn_h); + rtl8127_csi_write(tp, tp->HwPcieSNOffset + 4, tp->pci_cfg_space.pci_sn_h); + tp->esd_flag |= BIT_14; + } + } + + if (tp->esd_flag != 0) { + printk(KERN_ERR "%s: esd_flag = 0x%04x\n.\n", dev->name, tp->esd_flag); + netif_carrier_off(dev); + netif_tx_disable(dev); + rtl8127_hw_reset(dev); + rtl8127_tx_clear(tp); + rtl8127_rx_clear(tp); + rtl8127_init_ring(dev); + rtl8127_up(dev); + rtl8127_enable_hw_linkchg_interrupt(tp); + rtl8127_set_speed(dev, tp->autoneg, tp->speed, tp->duplex, tp->advertising); + tp->esd_flag = 0; + } +exit: + return; +} +/* +static void +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) +rtl8127_esd_timer(unsigned long __opaque) +#else +rtl8127_esd_timer(struct timer_list *t) +#endif +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) + struct net_device *dev = (struct net_device *)__opaque; + struct rtl8127_private *tp = netdev_priv(dev); + struct timer_list *timer = &tp->esd_timer; +#else + struct rtl8127_private *tp = from_timer(tp, t, esd_timer); + //struct net_device *dev = tp->dev; + struct timer_list *timer = t; +#endif + rtl8127_esd_checker(tp); + + mod_timer(timer, jiffies + timeout); +} +*/ + +/* +static void +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) +rtl8127_link_timer(unsigned long __opaque) +#else +rtl8127_link_timer(struct timer_list *t) +#endif +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0) + struct net_device *dev = (struct net_device *)__opaque; + struct rtl8127_private *tp = netdev_priv(dev); + struct timer_list *timer = &tp->link_timer; +#else + struct rtl8127_private *tp = from_timer(tp, t, link_timer); + struct net_device *dev = tp->dev; + struct timer_list *timer = t; +#endif + rtl8127_check_link_status(dev); + + mod_timer(timer, jiffies + RTL8127_LINK_TIMEOUT); +} +*/ + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) +static int pci_enable_msix_range(struct pci_dev *dev, struct msix_entry *entries, + int minvec, int maxvec) +{ + int nvec = maxvec; + int rc; + + if (maxvec < minvec) + return -ERANGE; + + do { + rc = pci_enable_msix(dev, entries, nvec); + if (rc < 0) { + return rc; + } else if (rc > 0) { + if (rc < minvec) + return -ENOSPC; + nvec = rc; + } + } while (rc); + + return nvec; +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) */ + +static int rtl8127_enable_msix(struct rtl8127_private *tp) +{ + int i, nvecs = 0; + struct msix_entry msix_ent[R8127_MAX_MSIX_VEC]; + //struct net_device *dev = tp->dev; + //const int len = sizeof(tp->irq_tbl[0].name); + + for (i = 0; i < R8127_MAX_MSIX_VEC; i++) { + msix_ent[i].entry = i; + msix_ent[i].vector = 0; + } + + nvecs = pci_enable_msix_range(tp->pci_dev, msix_ent, + tp->min_irq_nvecs, tp->max_irq_nvecs); + if (nvecs < 0) + goto out; + + for (i = 0; i < nvecs; i++) { + struct r8127_irq *irq = &tp->irq_tbl[i]; + irq->vector = msix_ent[i].vector; + //snprintf(irq->name, len, "%s-%d", dev->name, i); + //irq->handler = rtl8127_interrupt_msix; + } + +out: + return nvecs; +} + +/* Cfg9346_Unlock assumed. */ +static int rtl8127_try_msi(struct rtl8127_private *tp) +{ + struct pci_dev *pdev = tp->pci_dev; + unsigned int hw_supp_irq_nvecs; + unsigned msi = 0; + int nvecs = 1; + + hw_supp_irq_nvecs = R8127_MAX_MSIX_VEC_8125B; + tp->hw_supp_irq_nvecs = clamp_val(hw_supp_irq_nvecs, 1, + R8127_MAX_MSIX_VEC); + + tp->max_irq_nvecs = tp->hw_supp_irq_nvecs; + tp->min_irq_nvecs = R8127_MIN_MSIX_VEC_8127; +#ifdef DISABLE_MULTI_MSIX_VECTOR + tp->max_irq_nvecs = 1; +#endif + +#if defined(RTL_USE_NEW_INTR_API) + if ((nvecs = pci_alloc_irq_vectors(pdev, tp->min_irq_nvecs, tp->max_irq_nvecs, PCI_IRQ_MSIX)) > 0) + msi |= RTL_FEATURE_MSIX; + else if ((nvecs = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES)) > 0 && + pci_dev_msi_enabled(pdev)) + msi |= RTL_FEATURE_MSI; +#elif LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + if ((nvecs = rtl8127_enable_msix(tp)) > 0) + msi |= RTL_FEATURE_MSIX; + else if (!pci_enable_msi(pdev)) + msi |= RTL_FEATURE_MSI; +#endif + if (!(msi & (RTL_FEATURE_MSI | RTL_FEATURE_MSIX))) + dev_info(&pdev->dev, "no MSI/MSI-X. Back to INTx.\n"); + + if (!(msi & RTL_FEATURE_MSIX) || nvecs < 1) + nvecs = 1; + + tp->irq_nvecs = nvecs; + + tp->features |= msi; + + return nvecs; +} + +static void rtl8127_disable_msi(struct pci_dev *pdev, struct rtl8127_private *tp) +{ +#if defined(RTL_USE_NEW_INTR_API) + if (tp->features & (RTL_FEATURE_MSI | RTL_FEATURE_MSIX)) + pci_free_irq_vectors(pdev); +#elif LINUX_VERSION_CODE > KERNEL_VERSION(2,6,13) + if (tp->features & (RTL_FEATURE_MSIX)) + pci_disable_msix(pdev); + else if (tp->features & (RTL_FEATURE_MSI)) + pci_disable_msi(pdev); +#endif + tp->features &= ~(RTL_FEATURE_MSI | RTL_FEATURE_MSIX); +} + +static int rtl8127_get_irq(struct pci_dev *pdev) +{ +#if defined(RTL_USE_NEW_INTR_API) + return pci_irq_vector(pdev, 0); +#else + return pdev->irq; +#endif +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,11,0) +static void +rtl8127_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct rtl8127_counters *counters = tp->tally_vaddr; + dma_addr_t paddr = tp->tally_paddr; + + if (!counters) + return; + + netdev_stats_to_stats64(stats, &dev->stats); + dev_fetch_sw_netstats(stats, dev->tstats); + + /* + * Fetch additional counter values missing in stats collected by driver + * from tally counters. + */ + rtl8127_dump_tally_counter(tp, paddr); + + stats->tx_errors = le64_to_cpu(counters->tx_errors); + stats->collisions = le32_to_cpu(counters->tx_multi_collision); + stats->tx_aborted_errors = le16_to_cpu(counters->tx_aborted); + stats->rx_missed_errors = le16_to_cpu(counters->rx_missed); +} +#else +/** + * rtl8127_get_stats - Get rtl8127 read/write statistics + * @dev: The Ethernet Device to get statistics for + * + * Get TX/RX statistics for rtl8127 + */ +static struct +net_device_stats *rtl8127_get_stats(struct net_device *dev) +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) + struct rtl8127_private *tp = netdev_priv(dev); +#endif + return &RTLDEV->stats; +} +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,36) + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29) +static const struct net_device_ops rtl8127_netdev_ops = { + .ndo_open = rtl8127_open, + .ndo_stop = rtl8127_close, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,11,0) + .ndo_get_stats64 = rtl8127_get_stats64, +#else + .ndo_get_stats = rtl8127_get_stats, +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,11,0) + .ndo_start_xmit = rtl8127_start_xmit, + .ndo_tx_timeout = rtl8127_tx_timeout, + .ndo_change_mtu = rtl8127_change_mtu, + .ndo_set_mac_address = rtl8127_set_mac_address, +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,15,0) + .ndo_do_ioctl = rtl8127_do_ioctl, +#else + .ndo_siocdevprivate = rtl8127_siocdevprivate, + .ndo_eth_ioctl = rtl8127_do_ioctl, +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(5,15,0) +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,1,0) + .ndo_set_multicast_list = rtl8127_set_rx_mode, +#else + .ndo_set_rx_mode = rtl8127_set_rx_mode, +#endif +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) +#ifdef CONFIG_R8127_VLAN + .ndo_vlan_rx_register = rtl8127_vlan_rx_register, +#endif +#else + .ndo_fix_features = rtl8127_fix_features, + .ndo_set_features = rtl8127_set_features, +#endif +#ifdef CONFIG_NET_POLL_CONTROLLER + .ndo_poll_controller = rtl8127_netpoll, +#endif +}; +#endif + + +#ifdef CONFIG_R8127_NAPI + +static int rtl8127_poll(napi_ptr napi, napi_budget budget) +{ + struct r8127_napi *r8127napi = RTL_GET_PRIV(napi, struct r8127_napi); + struct rtl8127_private *tp = r8127napi->priv; + RTL_GET_NETDEV(tp) + unsigned int work_to_do = RTL_NAPI_QUOTA(budget, dev); + unsigned int work_done = 0; + int i; + + for (i = 0; i < tp->num_tx_rings; i++) + rtl8127_tx_interrupt(&tp->tx_ring[i], budget); + + for (i = 0; i < tp->num_rx_rings; i++) + work_done += rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[i], budget); + + work_done = min(work_done, work_to_do); + + RTL_NAPI_QUOTA_UPDATE(dev, work_done, budget); + + if (work_done < work_to_do) { +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + HandleDashInterrupt(tp->dev); +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + if (RTL_NETIF_RX_COMPLETE(dev, napi, work_done) == FALSE) + return RTL_NAPI_RETURN_VALUE; +#else + RTL_NETIF_RX_COMPLETE(dev, napi, work_done); +#endif + /* + * 20040426: the barrier is not strictly required but the + * behavior of the irq handler could be less predictable + * without it. Btw, the lack of flush for the posted pci + * write is safe - FR + */ + smp_wmb(); + + rtl8127_switch_to_timer_interrupt(tp); + } + + return RTL_NAPI_RETURN_VALUE; +} + +static int rtl8127_poll_msix_ring(napi_ptr napi, napi_budget budget) +{ + struct r8127_napi *r8127napi = RTL_GET_PRIV(napi, struct r8127_napi); + struct rtl8127_private *tp = r8127napi->priv; + RTL_GET_NETDEV(tp) + unsigned int work_to_do = RTL_NAPI_QUOTA(budget, dev); + unsigned int work_done = 0; + const int message_id = r8127napi->index; + + if (message_id < tp->num_tx_rings) + rtl8127_tx_interrupt_with_vector(tp, message_id, budget); + + if (message_id < tp->num_rx_rings) + work_done += rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[message_id], budget); + + RTL_NAPI_QUOTA_UPDATE(dev, work_done, budget); + + if (work_done < work_to_do) { +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH && message_id == 0) + HandleDashInterrupt(tp->dev); +#endif + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + if (RTL_NETIF_RX_COMPLETE(dev, napi, work_done) == FALSE) + return RTL_NAPI_RETURN_VALUE; +#else + RTL_NETIF_RX_COMPLETE(dev, napi, work_done); +#endif + /* + * 20040426: the barrier is not strictly required but the + * behavior of the irq handler could be less predictable + * without it. Btw, the lack of flush for the posted pci + * write is safe - FR + */ + smp_wmb(); + + rtl8127_enable_hw_interrupt_v2(tp, message_id); + } + + return RTL_NAPI_RETURN_VALUE; +} + +static int rtl8127_poll_msix_tx(napi_ptr napi, napi_budget budget) +{ + struct r8127_napi *r8127napi = RTL_GET_PRIV(napi, struct r8127_napi); + struct rtl8127_private *tp = r8127napi->priv; + RTL_GET_NETDEV(tp) + unsigned int work_to_do = RTL_NAPI_QUOTA(budget, dev); + unsigned int work_done = 0; + const int message_id = r8127napi->index; + + //suppress unused variable + (void)(dev); + + rtl8127_tx_interrupt_with_vector(tp, message_id, budget); + + RTL_NAPI_QUOTA_UPDATE(dev, work_done, budget); + + if (work_done < work_to_do) { +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + if (RTL_NETIF_RX_COMPLETE(dev, napi, work_done) == FALSE) + return RTL_NAPI_RETURN_VALUE; +#else + RTL_NETIF_RX_COMPLETE(dev, napi, work_done); +#endif + /* + * 20040426: the barrier is not strictly required but the + * behavior of the irq handler could be less predictable + * without it. Btw, the lack of flush for the posted pci + * write is safe - FR + */ + smp_wmb(); + + rtl8127_enable_hw_interrupt_v2(tp, message_id); + } + + return RTL_NAPI_RETURN_VALUE; +} + +static int rtl8127_poll_msix_other(napi_ptr napi, napi_budget budget) +{ + struct r8127_napi *r8127napi = RTL_GET_PRIV(napi, struct r8127_napi); + struct rtl8127_private *tp = r8127napi->priv; + RTL_GET_NETDEV(tp) + unsigned int work_to_do = RTL_NAPI_QUOTA(budget, dev); + const int message_id = r8127napi->index; + + //suppress unused variable + (void)(dev); + (void)(work_to_do); + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + RTL_NETIF_RX_COMPLETE(dev, napi, work_to_do); +#else + RTL_NETIF_RX_COMPLETE(dev, napi, work_to_do); +#endif + + rtl8127_enable_hw_interrupt_v2(tp, message_id); + + return 1; +} + +static int rtl8127_poll_msix_rx(napi_ptr napi, napi_budget budget) +{ + struct r8127_napi *r8127napi = RTL_GET_PRIV(napi, struct r8127_napi); + struct rtl8127_private *tp = r8127napi->priv; + RTL_GET_NETDEV(tp) + unsigned int work_to_do = RTL_NAPI_QUOTA(budget, dev); + unsigned int work_done = 0; + const int message_id = r8127napi->index; + + if (message_id < tp->num_rx_rings) + work_done += rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[message_id], budget); + + RTL_NAPI_QUOTA_UPDATE(dev, work_done, budget); + + if (work_done < work_to_do) { +#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,10,0) + if (RTL_NETIF_RX_COMPLETE(dev, napi, work_done) == FALSE) + return RTL_NAPI_RETURN_VALUE; +#else + RTL_NETIF_RX_COMPLETE(dev, napi, work_done); +#endif + /* + * 20040426: the barrier is not strictly required but the + * behavior of the irq handler could be less predictable + * without it. Btw, the lack of flush for the posted pci + * write is safe - FR + */ + smp_wmb(); + + rtl8127_enable_hw_interrupt_v2(tp, message_id); + } + + return RTL_NAPI_RETURN_VALUE; +} + +void rtl8127_enable_napi(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + int i; + + for (i = 0; i < tp->irq_nvecs; i++) + RTL_NAPI_ENABLE(tp->dev, &tp->r8127napi[i].napi); +#endif +} + +static void rtl8127_disable_napi(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + int i; + + for (i = 0; i < tp->irq_nvecs; i++) + RTL_NAPI_DISABLE(tp->dev, &tp->r8127napi[i].napi); +#endif +} + +static void rtl8127_del_napi(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + int i; + + for (i = 0; i < tp->irq_nvecs; i++) + RTL_NAPI_DEL((&tp->r8127napi[i])); +#endif +} +#endif //CONFIG_R8127_NAPI + +static void rtl8127_init_napi(struct rtl8127_private *tp) +{ + int i; + + for (i=0; iirq_nvecs; i++) { + struct r8127_napi *r8127napi = &tp->r8127napi[i]; +#ifdef CONFIG_R8127_NAPI + int (*poll)(struct napi_struct *, int); + + poll = rtl8127_poll; + if (tp->features & RTL_FEATURE_MSIX) { + switch (tp->HwCurrIsrVer) { + case 6: + if (i < R8127_MAX_RX_QUEUES_VEC_V4) + poll = rtl8127_poll_msix_rx; + else if (i == 8 || i == 9) + poll = rtl8127_poll_msix_tx; + else + poll = rtl8127_poll_msix_other; + break; + case 5: + if (i < R8127_MAX_RX_QUEUES_VEC_V3) + poll = rtl8127_poll_msix_rx; + else if (i == 16 || i == 17) + poll = rtl8127_poll_msix_tx; + else + poll = rtl8127_poll_msix_other; + break; + case 2: + if (i < R8127_MAX_RX_QUEUES_VEC_V3) + poll = rtl8127_poll_msix_rx; + else if (i == 16 || i == 18) + poll = rtl8127_poll_msix_tx; + else + poll = rtl8127_poll_msix_other; + break; + case 3: + case 4: + if (i < R8127_MAX_RX_QUEUES_VEC_V3) + poll = rtl8127_poll_msix_ring; + else + poll = rtl8127_poll_msix_other; + break; + } + } + + RTL_NAPI_CONFIG(tp->dev, r8127napi, poll, R8127_NAPI_WEIGHT); +#endif + + r8127napi->priv = tp; + r8127napi->index = i; + } +} + +static int +rtl8127_set_real_num_queue(struct rtl8127_private *tp) +{ + int retval = 0; + + retval = netif_set_real_num_tx_queues(tp->dev, tp->num_tx_rings); + if (retval < 0) + goto exit; + + retval = netif_set_real_num_rx_queues(tp->dev, tp->num_rx_rings); + if (retval < 0) + goto exit; + +exit: + return retval; +} + +static int __devinit +rtl8127_init_one(struct pci_dev *pdev, + const struct pci_device_id *ent) +{ + struct net_device *dev = NULL; + struct rtl8127_private *tp; + void __iomem *ioaddr = NULL; + static int board_idx = -1; + + int rc; + + assert(pdev != NULL); + assert(ent != NULL); + + board_idx++; + + if (netif_msg_drv(&debug)) + printk(KERN_INFO "%s Ethernet controller driver %s loaded\n", + MODULENAME, RTL8127_VERSION); + + rc = rtl8127_init_board(pdev, &dev, &ioaddr); + if (rc) + goto out; + + tp = netdev_priv(dev); + assert(ioaddr != NULL); + + spin_lock_init(&tp->phy_lock); + + tp->set_speed = rtl8127_set_speed_xmii; + tp->get_settings = rtl8127_gset_xmii; + tp->phy_reset_enable = rtl8127_xmii_reset_enable; + tp->phy_reset_pending = rtl8127_xmii_reset_pending; + tp->link_ok = rtl8127_xmii_link_ok; + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,11,0) + dev->tstats = devm_netdev_alloc_pcpu_stats(&pdev->dev, + struct pcpu_sw_netstats); + if (!dev->tstats) + goto err_out_1; +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,11,0) + + rc = rtl8127_try_msi(tp); + if (rc < 0) { + dev_err(&pdev->dev, "Can't allocate interrupt\n"); + goto err_out_1; + } + + rtl8127_init_software_variable(dev); + + RTL_NET_DEVICE_OPS(rtl8127_netdev_ops); + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,4,22) + SET_ETHTOOL_OPS(dev, &rtl8127_ethtool_ops); +#endif + + dev->watchdog_timeo = RTL8127_TX_TIMEOUT; + dev->irq = rtl8127_get_irq(pdev); + dev->base_addr = (unsigned long) ioaddr; + + rtl8127_init_napi(tp); + +#ifdef CONFIG_R8127_VLAN + if (tp->mcfg != CFG_METHOD_DEFAULT) { + dev->features |= NETIF_F_HW_VLAN_TX | NETIF_F_HW_VLAN_RX; +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) + dev->vlan_rx_kill_vid = rtl8127_vlan_rx_kill_vid; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) + } +#endif + + /* There has been a number of reports that using SG/TSO results in + * tx timeouts. However for a lot of people SG/TSO works fine. + * Therefore disable both features by default, but allow users to + * enable them. Use at own risk! + */ + tp->cp_cmd |= RTL_R16(tp, CPlusCmd); + if (tp->mcfg != CFG_METHOD_DEFAULT) { + dev->features |= NETIF_F_IP_CSUM; +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + tp->cp_cmd |= RxChkSum; +#else + dev->features |= NETIF_F_RXCSUM; + dev->features |= NETIF_F_SG | NETIF_F_TSO; + dev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO | + NETIF_F_RXCSUM | NETIF_F_HW_VLAN_TX | NETIF_F_HW_VLAN_RX; + dev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO | + NETIF_F_HIGHDMA; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,15,0) + dev->priv_flags |= IFF_LIVE_ADDR_CHANGE; +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,15,0) + dev->hw_features |= NETIF_F_RXALL; + dev->hw_features |= NETIF_F_RXFCS; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22) + dev->hw_features |= NETIF_F_IPV6_CSUM | NETIF_F_TSO6; + dev->features |= NETIF_F_IPV6_CSUM; + dev->features |= NETIF_F_TSO6; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,19,0) + netif_set_tso_max_size(dev, LSO_64K); + netif_set_tso_max_segs(dev, NIC_MAX_PHYS_BUF_COUNT_LSO2); +#else //LINUX_VERSION_CODE >= KERNEL_VERSION(5,19,0) + netif_set_gso_max_size(dev, LSO_64K); +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,0) + dev->gso_max_segs = NIC_MAX_PHYS_BUF_COUNT_LSO2; +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,7,0) + dev->gso_min_segs = NIC_MIN_PHYS_BUF_COUNT; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,7,0) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,0) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(5,19,0) + +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22) +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + +#ifdef ENABLE_RSS_SUPPORT + if (tp->EnableRss) { + dev->hw_features |= NETIF_F_RXHASH; + dev->features |= NETIF_F_RXHASH; + } +#endif + } + + netdev_sw_irq_coalesce_default_on(dev); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) + AllocateDashShareMemory(dev); +#endif + +#ifdef ENABLE_LIB_SUPPORT + BLOCKING_INIT_NOTIFIER_HEAD(&tp->lib_nh); +#endif + rtl8127_init_all_schedule_work(tp); + + rc = rtl8127_set_real_num_queue(tp); + if (rc < 0) + goto err_out; + + rtl8127_exit_oob(dev); + + rtl8127_powerup_pll(dev); + + rtl8127_hw_init(dev); + + rtl8127_hw_reset(dev); + + /* Get production from EEPROM */ + rtl8127_eeprom_type(tp); + + if (tp->eeprom_type == EEPROM_TYPE_93C46 || tp->eeprom_type == EEPROM_TYPE_93C56) + rtl8127_set_eeprom_sel_low(tp); + + rtl8127_get_mac_address(dev); + + tp->fw_name = rtl_chip_fw_infos[tp->mcfg].fw_name; + + tp->tally_vaddr = dma_alloc_coherent(&pdev->dev, sizeof(*tp->tally_vaddr), + &tp->tally_paddr, GFP_KERNEL); + if (!tp->tally_vaddr) { + rc = -ENOMEM; + goto err_out; + } + + rtl8127_tally_counter_clear(tp); + + pci_set_drvdata(pdev, dev); + + rc = register_netdev(dev); + if (rc) + goto err_out; + + printk(KERN_INFO "%s: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.\n", MODULENAME); + + rtl8127_disable_rxdvgate(dev); + + device_set_wakeup_enable(&pdev->dev, tp->wol_enabled); + + netif_carrier_off(dev); + +#ifdef ENABLE_R8127_SYSFS + rtl8127_sysfs_init(dev); +#endif /* ENABLE_R8127_SYSFS */ + + printk("%s", GPL_CLAIM); + +out: + return rc; + +err_out: + if (tp->tally_vaddr != NULL) { + dma_free_coherent(&pdev->dev, sizeof(*tp->tally_vaddr), tp->tally_vaddr, + tp->tally_paddr); + + tp->tally_vaddr = NULL; + } +#ifdef CONFIG_R8127_NAPI + rtl8127_del_napi(tp); +#endif + rtl8127_disable_msi(pdev, tp); + +err_out_1: + rtl8127_release_board(pdev, dev); + + goto out; +} + +static void __devexit +rtl8127_remove_one(struct pci_dev *pdev) +{ + struct net_device *dev = pci_get_drvdata(pdev); + struct rtl8127_private *tp = netdev_priv(dev); + + assert(dev != NULL); + assert(tp != NULL); + + set_bit(R8127_FLAG_DOWN, tp->task_flags); + + rtl8127_cancel_all_schedule_work(tp); + +#ifdef CONFIG_R8127_NAPI + rtl8127_del_napi(tp); +#endif + if (HW_DASH_SUPPORT_DASH(tp)) + rtl8127_driver_stop(tp); + + rtl8127_disable_pci_offset_180(tp); + +#ifdef ENABLE_R8127_SYSFS + rtl8127_sysfs_remove(dev); +#endif //ENABLE_R8127_SYSFS + + unregister_netdev(dev); + rtl8127_disable_msi(pdev, tp); +#ifdef ENABLE_R8127_PROCFS + rtl8127_proc_remove(dev); +#endif + if (tp->tally_vaddr != NULL) { + dma_free_coherent(&pdev->dev, sizeof(*tp->tally_vaddr), tp->tally_vaddr, tp->tally_paddr); + tp->tally_vaddr = NULL; + } + + rtl8127_release_board(pdev, dev); + +#ifdef ENABLE_USE_FIRMWARE_FILE + rtl8127_release_firmware(tp); +#endif + + pci_set_drvdata(pdev, NULL); +} + +#ifdef ENABLE_PAGE_REUSE +static inline unsigned int rtl8127_rx_page_order(unsigned rx_buf_sz, unsigned page_size) +{ + unsigned truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) + + SKB_DATA_ALIGN(rx_buf_sz + R8127_RX_ALIGN); + + return get_order(truesize * 2); +} +#endif //ENABLE_PAGE_REUSE + +static void +rtl8127_set_rxbufsize(struct rtl8127_private *tp, + struct net_device *dev) +{ + unsigned int mtu = dev->mtu; + + tp->rms = (mtu > ETH_DATA_LEN) ? + mtu + ETH_HLEN + RT_VALN_HLEN + ETH_FCS_LEN: + RX_BUF_SIZE; + tp->rx_buf_sz = tp->rms; +#ifdef ENABLE_RX_PACKET_FRAGMENT + tp->rx_buf_sz = SKB_DATA_ALIGN(RX_BUF_SIZE); +#endif //ENABLE_RX_PACKET_FRAGMENT +#ifdef ENABLE_PAGE_REUSE + tp->rx_buf_page_order = rtl8127_rx_page_order(tp->rx_buf_sz, PAGE_SIZE); + tp->rx_buf_page_size = rtl8127_rx_page_size(tp->rx_buf_page_order); +#endif //ENABLE_PAGE_REUSE +} + +static void +rtl8127_set_rms(struct rtl8127_private *tp, u16 rms) +{ + RTL_W16(tp, RxMaxSize, rms | AcceppVlanPhys); +} + +static void rtl8127_free_irq(struct rtl8127_private *tp) +{ + int i; + + for (i=0; iirq_nvecs; i++) { + struct r8127_irq *irq = &tp->irq_tbl[i]; + struct r8127_napi *r8127napi = &tp->r8127napi[i]; + + if (irq->requested) { + irq->requested = 0; +#if defined(RTL_USE_NEW_INTR_API) + pci_free_irq(tp->pci_dev, i, r8127napi); +#else + free_irq(irq->vector, r8127napi); +#endif + } + } +} + +static int rtl8127_alloc_irq(struct rtl8127_private *tp) +{ + struct net_device *dev = tp->dev; + int rc = 0; + struct r8127_irq *irq; + struct r8127_napi *r8127napi; + int i = 0; + const int len = sizeof(tp->irq_tbl[0].name); + +#if defined(RTL_USE_NEW_INTR_API) + for (i=0; iirq_nvecs; i++) { + irq = &tp->irq_tbl[i]; + if (tp->features & RTL_FEATURE_MSIX && + tp->HwCurrIsrVer > 1) + irq->handler = rtl8127_interrupt_msix; + else + irq->handler = rtl8127_interrupt; + + r8127napi = &tp->r8127napi[i]; + snprintf(irq->name, len, "%s-%d", dev->name, i); + rc = pci_request_irq(tp->pci_dev, i, irq->handler, NULL, r8127napi, + irq->name); + if (rc) + break; + + irq->vector = pci_irq_vector(tp->pci_dev, i); + irq->requested = 1; + } +#else + unsigned long irq_flags = 0; +#ifdef ENABLE_LIB_SUPPORT + irq_flags |= IRQF_NO_SUSPEND; +#endif + if (tp->features & RTL_FEATURE_MSIX && + tp->HwCurrIsrVer > 1) { + for (i=0; iirq_nvecs; i++) { + irq = &tp->irq_tbl[i]; + irq->handler = rtl8127_interrupt_msix; + r8127napi = &tp->r8127napi[i]; + snprintf(irq->name, len, "%s-%d", dev->name, i); + rc = request_irq(irq->vector, irq->handler, irq_flags, irq->name, r8127napi); + + if (rc) + break; + + irq->requested = 1; + } + } else { + irq = &tp->irq_tbl[0]; + irq->handler = rtl8127_interrupt; + r8127napi = &tp->r8127napi[0]; + snprintf(irq->name, len, "%s-0", dev->name); + if (!(tp->features & RTL_FEATURE_MSIX)) + irq->vector = dev->irq; + irq_flags |= (tp->features & (RTL_FEATURE_MSI | RTL_FEATURE_MSIX)) ? 0 : SA_SHIRQ; + rc = request_irq(irq->vector, irq->handler, irq_flags, irq->name, r8127napi); + + if (rc == 0) + irq->requested = 1; + } +#endif + if (rc) + rtl8127_free_irq(tp); + + return rc; +} + +static int rtl8127_alloc_tx_desc(struct rtl8127_private *tp) +{ + struct rtl8127_tx_ring *ring; + struct pci_dev *pdev = tp->pci_dev; + int i; + + for (i = 0; i < tp->num_tx_rings; i++) { + ring = &tp->tx_ring[i]; + ring->TxDescAllocSize = (ring->num_tx_desc + 1) * sizeof(struct TxDesc); + ring->TxDescArray = dma_alloc_coherent(&pdev->dev, + ring->TxDescAllocSize, + &ring->TxPhyAddr, + GFP_KERNEL); + + if (!ring->TxDescArray) + return -1; + } + + return 0; +} + +static int rtl8127_alloc_rx_desc(struct rtl8127_private *tp) +{ + struct rtl8127_rx_ring *ring; + struct pci_dev *pdev = tp->pci_dev; + int i; + + for (i = 0; i < tp->num_rx_rings; i++) { + ring = &tp->rx_ring[i]; + ring->RxDescAllocSize = (ring->num_rx_desc + 1) * tp->RxDescLength; + ring->RxDescArray = dma_alloc_coherent(&pdev->dev, + ring->RxDescAllocSize, + &ring->RxPhyAddr, + GFP_KERNEL); + + if (!ring->RxDescArray) + return -1; + } + + return 0; +} + +static void rtl8127_free_tx_desc(struct rtl8127_private *tp) +{ + struct rtl8127_tx_ring *ring; + struct pci_dev *pdev = tp->pci_dev; + int i; + + for (i = 0; i < tp->num_tx_rings; i++) { + ring = &tp->tx_ring[i]; + if (ring->TxDescArray) { + dma_free_coherent(&pdev->dev, + ring->TxDescAllocSize, + ring->TxDescArray, + ring->TxPhyAddr); + ring->TxDescArray = NULL; + } + } +} + +static void rtl8127_free_rx_desc(struct rtl8127_private *tp) +{ + struct rtl8127_rx_ring *ring; + struct pci_dev *pdev = tp->pci_dev; + int i; + + for (i = 0; i < tp->num_rx_rings; i++) { + ring = &tp->rx_ring[i]; + if (ring->RxDescArray) { + dma_free_coherent(&pdev->dev, + ring->RxDescAllocSize, + ring->RxDescArray, + ring->RxPhyAddr); + ring->RxDescArray = NULL; + } + } +} + +static void rtl8127_free_alloc_resources(struct rtl8127_private *tp) +{ + rtl8127_free_rx_desc(tp); + + rtl8127_free_tx_desc(tp); +} + +#ifdef ENABLE_USE_FIRMWARE_FILE +static void rtl8127_request_firmware(struct rtl8127_private *tp) +{ + struct rtl8127_fw *rtl_fw; + + /* firmware loaded already or no firmware available */ + if (tp->rtl_fw || !tp->fw_name) + return; + + rtl_fw = kzalloc(sizeof(*rtl_fw), GFP_KERNEL); + if (!rtl_fw) + return; + + rtl_fw->phy_write = rtl8127_mdio_write; + rtl_fw->phy_read = rtl8127_mdio_read; + rtl_fw->mac_mcu_write = mac_mcu_write; + rtl_fw->mac_mcu_read = mac_mcu_read; + rtl_fw->fw_name = tp->fw_name; + rtl_fw->dev = tp_to_dev(tp); + + if (rtl8127_fw_request_firmware(rtl_fw)) + kfree(rtl_fw); + else + tp->rtl_fw = rtl_fw; +} +#endif + +int rtl8127_open(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int retval; + + retval = -ENOMEM; + +#ifdef ENABLE_R8127_PROCFS + rtl8127_proc_init(dev); +#endif + rtl8127_set_rxbufsize(tp, dev); + /* + * Rx and Tx descriptors needs 256 bytes alignment. + * pci_alloc_consistent provides more. + */ + if (rtl8127_alloc_tx_desc(tp) < 0 || rtl8127_alloc_rx_desc(tp) < 0) + goto err_free_all_allocated_mem; + + retval = rtl8127_init_ring(dev); + if (retval < 0) + goto err_free_all_allocated_mem; + + retval = rtl8127_alloc_irq(tp); + if (retval < 0) + goto err_free_all_allocated_mem; + + if (netif_msg_probe(tp)) { + printk(KERN_INFO "%s: 0x%lx, " + "%2.2x:%2.2x:%2.2x:%2.2x:%2.2x:%2.2x, " + "IRQ %d\n", + dev->name, + dev->base_addr, + dev->dev_addr[0], dev->dev_addr[1], + dev->dev_addr[2], dev->dev_addr[3], + dev->dev_addr[4], dev->dev_addr[5], dev->irq); + } + +#ifdef ENABLE_USE_FIRMWARE_FILE + rtl8127_request_firmware(tp); +#endif + pci_set_master(tp->pci_dev); + +#ifdef CONFIG_R8127_NAPI + rtl8127_enable_napi(tp); +#endif + + rtl8127_exit_oob(dev); + + rtl8127_up(dev); + +#ifdef ENABLE_PTP_SUPPORT + if (tp->EnablePtp) + rtl8127_ptp_init(tp); +#endif + clear_bit(R8127_FLAG_DOWN, tp->task_flags); + + if (tp->resume_not_chg_speed) + _rtl8127_check_link_status(dev, R8127_LINK_STATE_UNKNOWN); + else + rtl8127_set_speed(dev, tp->autoneg, tp->speed, tp->duplex, tp->advertising); + + if (tp->esd_flag == 0) { + //rtl8127_request_esd_timer(dev); + + rtl8127_schedule_esd_work(tp); + } + + //rtl8127_request_link_timer(dev); + + rtl8127_enable_hw_linkchg_interrupt(tp); + +out: + + return retval; + +err_free_all_allocated_mem: + rtl8127_free_alloc_resources(tp); + + goto out; +} + +static void +_rtl8127_set_l1_l0s_entry_latency(struct rtl8127_private *tp, u8 setting) +{ + u32 csi_tmp; + u32 temp; + + temp = setting & 0x3f; + temp <<= 24; + /*set PCI configuration space offset 0x70F to setting*/ + /*When the register offset of PCI configuration space larger than 0xff, use CSI to access it.*/ + + csi_tmp = rtl8127_csi_read(tp, 0x70c) & 0xc0ffffff; + rtl8127_csi_write(tp, 0x70c, csi_tmp | temp); +} + +static void +rtl8127_set_l1_l0s_entry_latency(struct rtl8127_private *tp) +{ + _rtl8127_set_l1_l0s_entry_latency(tp, 0x27); +} + +static void +_rtl8127_set_mrrs(struct rtl8127_private *tp, u8 setting) +{ + //Set PCI configuration space offset 0x79 to setting + + struct pci_dev *pdev = tp->pci_dev; + u8 device_control; + + pci_read_config_byte(pdev, 0x79, &device_control); + device_control &= ~0x70; + device_control |= setting; + pci_write_config_byte(pdev, 0x79, device_control); +} + +static void +rtl8127_set_mrrs(struct rtl8127_private *tp) +{ + if (hwoptimize & HW_PATCH_SOC_LAN) + return; + + _rtl8127_set_mrrs(tp, 0x40); +} + +static void +rtl8127_disable_l1_timeout(struct rtl8127_private *tp) +{ + rtl8127_csi_write(tp, 0x890, rtl8127_csi_read(tp, 0x890) & ~BIT(0)); +} + +void +rtl8127_hw_set_rx_packet_filter(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + u32 mc_filter[2]; /* Multicast hash filter */ + int rx_mode; + u32 tmp = 0; + + if (dev->flags & IFF_PROMISC) { + /* Unconditionally log net taps. */ + if (netif_msg_link(tp)) + printk(KERN_NOTICE "%s: Promiscuous mode enabled.\n", + dev->name); + + rx_mode = + AcceptBroadcast | AcceptMulticast | AcceptMyPhys | + AcceptAllPhys; + mc_filter[1] = mc_filter[0] = 0xffffffff; + } else if (dev->flags & IFF_ALLMULTI) { + /* accept all multicasts. */ + rx_mode = AcceptBroadcast | AcceptMulticast | AcceptMyPhys; + mc_filter[1] = mc_filter[0] = 0xffffffff; + } else { +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,35) + struct dev_mc_list *mclist; + unsigned int i; + + rx_mode = AcceptBroadcast | AcceptMyPhys; + mc_filter[1] = mc_filter[0] = 0; + for (i = 0, mclist = dev->mc_list; mclist && i < dev->mc_count; + i++, mclist = mclist->next) { + int bit_nr = ether_crc(ETH_ALEN, mclist->dmi_addr) >> 26; + mc_filter[bit_nr >> 5] |= 1 << (bit_nr & 31); + rx_mode |= AcceptMulticast; + } +#else + struct netdev_hw_addr *ha; + + rx_mode = AcceptBroadcast | AcceptMyPhys; + mc_filter[1] = mc_filter[0] = 0; + netdev_for_each_mc_addr(ha, dev) { + int bit_nr = ether_crc(ETH_ALEN, ha->addr) >> 26; + mc_filter[bit_nr >> 5] |= 1 << (bit_nr & 31); + rx_mode |= AcceptMulticast; + } +#endif + } + + if (dev->features & NETIF_F_RXALL) + rx_mode |= (AcceptErr | AcceptRunt); + + tmp = mc_filter[0]; + mc_filter[0] = swab32(mc_filter[1]); + mc_filter[1] = swab32(tmp); + + tmp = tp->rtl8127_rx_config | rx_mode | (RTL_R32(tp, RxConfig) & rtl_chip_info[tp->chipset].RxConfigMask); + + RTL_W32(tp, RxConfig, tmp); + RTL_W32(tp, MAR0 + 0, mc_filter[0]); + RTL_W32(tp, MAR0 + 4, mc_filter[1]); +} + +static void +rtl8127_set_rx_mode(struct net_device *dev) +{ + rtl8127_hw_set_rx_packet_filter(dev); +} + +void +rtl8127_set_rx_q_num(struct rtl8127_private *tp, + unsigned int num_rx_queues) +{ + u16 q_ctrl; + u16 rx_q_num; + + rx_q_num = (u16)ilog2(num_rx_queues); + rx_q_num &= (BIT_0 | BIT_1 | BIT_2); + rx_q_num <<= 2; + q_ctrl = RTL_R16(tp, Q_NUM_CTRL_8125); + q_ctrl &= ~(BIT_2 | BIT_3 | BIT_4); + q_ctrl |= rx_q_num; + RTL_W16(tp, Q_NUM_CTRL_8125, q_ctrl); +} + +void +rtl8127_set_tx_q_num(struct rtl8127_private *tp, + unsigned int num_tx_queues) +{ + u16 mac_ocp_data; + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xE63E); + mac_ocp_data &= ~(BIT_11 | BIT_10); + mac_ocp_data |= ((ilog2(num_tx_queues) & 0x03) << 10); + rtl8127_mac_ocp_write(tp, 0xE63E, mac_ocp_data); +} + +void +rtl8127_enable_mcu(struct rtl8127_private *tp, bool enable) +{ + if (FALSE == HW_SUPPORT_MAC_MCU(tp)) + return; + + if (enable) + rtl8127_set_mac_ocp_bit(tp, 0xC0B4, BIT_0); + else + rtl8127_clear_mac_ocp_bit(tp, 0xC0B4, BIT_0); +} + +static void +rtl8127_clear_tcam_entries(struct rtl8127_private *tp) +{ + if (FALSE == HW_SUPPORT_TCAM(tp)) + return; + + rtl8127_set_mac_ocp_bit(tp, 0xEB54, BIT_0); + fsleep(1); + rtl8127_clear_mac_ocp_bit(tp, 0xEB54, BIT_0); +} + +static u8 +rtl8127_get_l1off_cap_bits(struct rtl8127_private *tp) +{ + u8 l1offCapBits = 0; + + l1offCapBits = (BIT_0 | BIT_1); + l1offCapBits |= (BIT_2 | BIT_3); + + return l1offCapBits; +} + +void +rtl8127_hw_config(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + struct pci_dev *pdev = tp->pci_dev; + u16 mac_ocp_data; + + rtl8127_disable_rx_packet_filter(tp); + + rtl8127_hw_reset(dev); + + rtl8127_enable_cfg9346_write(tp); + rtl8127_enable_force_clkreq(tp, 0); + rtl8127_enable_aspm_clkreq_lock(tp, 0); + + rtl8127_set_eee_lpi_timer(tp); + + //keep magic packet only + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xC0B6); + mac_ocp_data &= BIT_0; + rtl8127_mac_ocp_write(tp, 0xC0B6, mac_ocp_data); + + rtl8127_tally_counter_addr_fill(tp); + + rtl8127_enable_extend_tally_couter(tp); + + rtl8127_desc_addr_fill(tp); + + /* Set DMA burst size and Interframe Gap Time */ + RTL_W32(tp, TxConfig, (TX_DMA_BURST_unlimited << TxDMAShift) | + (InterFrameGap << TxInterFrameGapShift)); + + if (tp->EnableTxNoClose) + RTL_W32(tp, TxConfig, (RTL_R32(tp, TxConfig) | BIT_6)); + + if (enable_double_vlan) + rtl8127_enable_double_vlan(tp); + else + rtl8127_disable_double_vlan(tp); + + rtl8127_set_l1_l0s_entry_latency(tp); + + rtl8127_set_mrrs(tp); + + rtl8127_disable_l1_timeout(tp); + +#ifdef ENABLE_RSS_SUPPORT + rtl8127_config_rss(tp); +#else + RTL_W32(tp, RSS_CTRL_8125, 0x00); +#endif + rtl8127_set_rx_q_num(tp, rtl8127_tot_rx_rings(tp)); + + RTL_W8(tp, Config1, RTL_R8(tp, Config1) & ~0x10); + + rtl8127_mac_ocp_write(tp, 0xC140, 0xFFFF); + rtl8127_mac_ocp_write(tp, 0xC142, 0xFFFF); + + //new tx desc format + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xEB58); + mac_ocp_data &= ~(BIT_0 | BIT_1); + mac_ocp_data |= (BIT_0); + rtl8127_mac_ocp_write(tp, 0xEB58, mac_ocp_data); + + if (tp->EnableTxNoClose) + RTL_W8(tp, 0x20E4, RTL_R8(tp, 0x20E4) | BIT_2); + else + RTL_W8(tp, 0x20E4, RTL_R8(tp, 0x20E4) & ~BIT_2); + + if (tp->HwSuppRxDescType == RX_DESC_RING_TYPE_4) { + if (tp->InitRxDescType == RX_DESC_RING_TYPE_4) + RTL_W8(tp, 0xd8, RTL_R8(tp, 0xd8) | + EnableRxDescV4_0); + else + RTL_W8(tp, 0xd8, RTL_R8(tp, 0xd8) & + ~EnableRxDescV4_0); + } + + if (tp->mcfg == CFG_METHOD_2) { + rtl8127_clear_mac_ocp_bit(tp, 0xE00C, BIT_12); + + rtl8127_clear_mac_ocp_bit(tp, 0xC0C2, BIT_6); + } + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xE614); + mac_ocp_data &= ~(BIT_11 | BIT_10 | BIT_9 | BIT_8); + mac_ocp_data |= (15 << 8); + rtl8127_mac_ocp_write(tp, 0xE614, mac_ocp_data); + + rtl8127_set_tx_q_num(tp, rtl8127_tot_tx_rings(tp)); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xE63E); + mac_ocp_data &= ~(BIT_5 | BIT_4); + mac_ocp_data |= ((0x02 & 0x03) << 4); + rtl8127_mac_ocp_write(tp, 0xE63E, mac_ocp_data); + + rtl8127_enable_mcu(tp, 0); + rtl8127_enable_mcu(tp, 1); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xC0B4); + mac_ocp_data |= (BIT_3 | BIT_2); + rtl8127_mac_ocp_write(tp, 0xC0B4, mac_ocp_data); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xEB6A); + mac_ocp_data &= ~(BIT_7 | BIT_6 | BIT_5 | BIT_4 | BIT_3 | BIT_2 | BIT_1 | BIT_0); + mac_ocp_data |= (BIT_5 | BIT_4 | BIT_1 | BIT_0); + rtl8127_mac_ocp_write(tp, 0xEB6A, mac_ocp_data); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xEB50); + mac_ocp_data &= ~(BIT_9 | BIT_8 | BIT_7 | BIT_6 | BIT_5); + mac_ocp_data |= (BIT_6); + rtl8127_mac_ocp_write(tp, 0xEB50, mac_ocp_data); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xE056); + mac_ocp_data &= ~(BIT_7 | BIT_6 | BIT_5 | BIT_4); + //mac_ocp_data |= (BIT_4 | BIT_5); + rtl8127_mac_ocp_write(tp, 0xE056, mac_ocp_data); + + RTL_W8(tp, TDFNR, 0x10); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xE040); + mac_ocp_data &= ~(BIT_12); + rtl8127_mac_ocp_write(tp, 0xE040, mac_ocp_data); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xEA1C); + mac_ocp_data &= ~(BIT_1 | BIT_0); + mac_ocp_data |= (BIT_0); + rtl8127_mac_ocp_write(tp, 0xEA1C, mac_ocp_data); + + rtl8127_mac_ocp_write(tp, 0xE0C0, 0x4000); + + rtl8127_set_mac_ocp_bit(tp, 0xE052, (BIT_6 | BIT_5)); + rtl8127_clear_mac_ocp_bit(tp, 0xE052, BIT_3 | BIT_7); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xD430); + mac_ocp_data &= ~(BIT_11 | BIT_10 | BIT_9 | BIT_8 | BIT_7 | BIT_6 | BIT_5 | BIT_4 | BIT_3 | BIT_2 | BIT_1 | BIT_0); + mac_ocp_data |= 0x45F; + rtl8127_mac_ocp_write(tp, 0xD430, mac_ocp_data); + + //rtl8127_mac_ocp_write(tp, 0xE0C0, 0x4F87); + if (!tp->DASH) + RTL_W8(tp, 0xD0, RTL_R8(tp, 0xD0) | BIT_6 | BIT_7); + else + RTL_W8(tp, 0xD0, RTL_R8(tp, 0xD0) & ~(BIT_6 | BIT_7)); + + rtl8127_disable_eee_plus(tp); + + mac_ocp_data = rtl8127_mac_ocp_read(tp, 0xEA1C); + mac_ocp_data &= ~(BIT_2); + mac_ocp_data &= ~(BIT_9 | BIT_8); + rtl8127_mac_ocp_write(tp, 0xEA1C, mac_ocp_data); + + rtl8127_clear_tcam_entries(tp); + + RTL_W16(tp, 0x1880, RTL_R16(tp, 0x1880) & ~(BIT_4 | BIT_5)); + + rtl8127_clear_set_mac_ocp_bit(tp, 0xD40C, 0xE038, 0x8020); + + /* csum offload command for RTL8125 */ + tp->tx_tcp_csum_cmd = TxTCPCS_C; + tp->tx_udp_csum_cmd = TxUDPCS_C; + tp->tx_ip_csum_cmd = TxIPCS_C; + tp->tx_ipv6_csum_cmd = TxIPV6F_C; + + /* config interrupt type for RTL8125B */ + if (tp->HwSuppIsrVer > 1) + rtl8127_hw_set_interrupt_type(tp, tp->HwCurrIsrVer); + + //other hw parameters + rtl8127_hw_clear_timer_int(dev); + + rtl8127_hw_clear_int_miti(dev); + + if (tp->use_timer_interrupt && + (tp->HwCurrIsrVer > 1) && + (tp->HwSuppIntMitiVer > 3) && + (tp->features & RTL_FEATURE_MSIX)) { + int i; + for (i = 0; i < tp->irq_nvecs; i++) + rtl8127_hw_set_timer_int(tp, i, timer_count_v2); + } + + rtl8127_enable_exit_l1_mask(tp); + + rtl8127_mac_ocp_write(tp, 0xE098, 0xC302); + + if (aspm && (tp->org_pci_offset_99 & (BIT_2 | BIT_5 | BIT_6))) + rtl8127_init_pci_offset_99(tp); + else + rtl8127_disable_pci_offset_99(tp); + + if (aspm && (tp->org_pci_offset_180 & rtl8127_get_l1off_cap_bits(tp))) + rtl8127_init_pci_offset_180(tp); + else + rtl8127_disable_pci_offset_180(tp); + + tp->cp_cmd &= ~(EnableBist | Macdbgo_oe | Force_halfdup | + Force_rxflow_en | Force_txflow_en | Cxpl_dbg_sel | + ASF | Macdbgo_sel); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,0,0) + RTL_W16(tp, CPlusCmd, tp->cp_cmd); +#else + rtl8127_hw_set_features(dev, dev->features); +#endif + rtl8127_set_rms(tp, tp->rms); + + rtl8127_disable_rxdvgate(dev); + + if (!tp->pci_cfg_is_read) { + pci_read_config_byte(pdev, PCI_COMMAND, &tp->pci_cfg_space.cmd); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_0, &tp->pci_cfg_space.io_base_l); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_0 + 2, &tp->pci_cfg_space.io_base_h); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_2, &tp->pci_cfg_space.mem_base_l); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_2 + 2, &tp->pci_cfg_space.mem_base_h); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_3, &tp->pci_cfg_space.resv_0x1c_l); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_3 + 2, &tp->pci_cfg_space.resv_0x1c_h); + pci_read_config_byte(pdev, PCI_INTERRUPT_LINE, &tp->pci_cfg_space.ilr); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_4, &tp->pci_cfg_space.resv_0x20_l); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_4 + 2, &tp->pci_cfg_space.resv_0x20_h); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_5, &tp->pci_cfg_space.resv_0x24_l); + pci_read_config_word(pdev, PCI_BASE_ADDRESS_5 + 2, &tp->pci_cfg_space.resv_0x24_h); + pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &tp->pci_cfg_space.resv_0x2c_l); + pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID + 2, &tp->pci_cfg_space.resv_0x2c_h); + if (tp->HwPcieSNOffset > 0) { + tp->pci_cfg_space.pci_sn_l = rtl8127_csi_read(tp, tp->HwPcieSNOffset); + tp->pci_cfg_space.pci_sn_h = rtl8127_csi_read(tp, tp->HwPcieSNOffset + 4); + } + + tp->pci_cfg_is_read = 1; + } + + /* Set Rx packet filter */ + rtl8127_hw_set_rx_packet_filter(dev); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH && !tp->dash_printer_enabled) + NICChkTypeEnableDashInterrupt(tp); +#endif + + rtl8127_enable_aspm_clkreq_lock(tp, aspm ? 1 : 0); + + rtl8127_disable_cfg9346_write(tp); + + fsleep(10); +} + +void +rtl8127_hw_start(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + +#ifdef ENABLE_LIB_SUPPORT + rtl8127_init_lib_ring(tp); +#endif + + RTL_W8(tp, ChipCmd, CmdTxEnb | CmdRxEnb); + + rtl8127_enable_hw_interrupt(tp); + + rtl8127_lib_reset_complete(tp); +} + +static int +rtl8127_change_mtu(struct net_device *dev, + int new_mtu) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret = 0; + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,10,0) + if (new_mtu < ETH_MIN_MTU) + return -EINVAL; + else if (new_mtu > tp->max_jumbo_frame_size) + new_mtu = tp->max_jumbo_frame_size; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,10,0) + + dev->mtu = new_mtu; + + tp->eee.tx_lpi_timer = dev->mtu + ETH_HLEN + 0x20; + + if (!netif_running(dev)) + goto out; + + rtl8127_down(dev); + + rtl8127_set_rxbufsize(tp, dev); + + ret = rtl8127_init_ring(dev); + + if (ret < 0) + goto err_out; + +#ifdef CONFIG_R8127_NAPI + rtl8127_enable_napi(tp); +#endif//CONFIG_R8127_NAPI + + if (tp->link_ok(dev)) + rtl8127_link_on_patch(dev); + else + rtl8127_link_down_patch(dev); + + //mod_timer(&tp->esd_timer, jiffies + RTL8127_ESD_TIMEOUT); + //mod_timer(&tp->link_timer, jiffies + RTL8127_LINK_TIMEOUT); +out: +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,0,0) + netdev_update_features(dev); +#endif + +err_out: + return ret; +} + +static inline void +rtl8127_set_desc_dma_addr(struct rtl8127_private *tp, + struct RxDesc *desc, + dma_addr_t mapping) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + ((struct RxDescV3 *)desc)->addr = cpu_to_le64(mapping); + break; + case RX_DESC_RING_TYPE_4: + ((struct RxDescV4 *)desc)->addr = cpu_to_le64(mapping); + break; + default: + desc->addr = cpu_to_le64(mapping); + break; + } +} + +static inline void +rtl8127_mark_to_asic_v1(struct RxDesc *desc, + u32 rx_buf_sz) +{ + u32 eor = le32_to_cpu(desc->opts1) & RingEnd; + + WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | rx_buf_sz)); +} + +static inline void +rtl8127_mark_to_asic_v3(struct RxDescV3 *descv3, + u32 rx_buf_sz) +{ + u32 eor = le32_to_cpu(descv3->RxDescNormalDDWord4.opts1) & RingEnd; + + WRITE_ONCE(descv3->RxDescNormalDDWord4.opts1, cpu_to_le32(DescOwn | eor | rx_buf_sz)); +} + +static inline void +rtl8127_mark_to_asic_v4(struct RxDescV4 *descv4, + u32 rx_buf_sz) +{ + u32 eor = le32_to_cpu(descv4->RxDescNormalDDWord2.opts1) & RingEnd; + + WRITE_ONCE(descv4->RxDescNormalDDWord2.opts1, cpu_to_le32(DescOwn | eor | rx_buf_sz)); +} + +void +rtl8127_mark_to_asic(struct rtl8127_private *tp, + struct RxDesc *desc, + u32 rx_buf_sz) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + rtl8127_mark_to_asic_v3((struct RxDescV3 *)desc, rx_buf_sz); + break; + case RX_DESC_RING_TYPE_4: + rtl8127_mark_to_asic_v4((struct RxDescV4 *)desc, rx_buf_sz); + break; + default: + rtl8127_mark_to_asic_v1(desc, rx_buf_sz); + break; + } +} + +static inline void +rtl8127_map_to_asic(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct RxDesc *desc, + dma_addr_t mapping, + u32 rx_buf_sz, + const u32 cur_rx) +{ + ring->RxDescPhyAddr[cur_rx] = mapping; + rtl8127_set_desc_dma_addr(tp, desc, mapping); + wmb(); + rtl8127_mark_to_asic(tp, desc, rx_buf_sz); +} + +#ifdef ENABLE_PAGE_REUSE + +static int +rtl8127_alloc_rx_page(struct rtl8127_private *tp, struct rtl8127_rx_ring *ring, + struct rtl8127_rx_buffer *rxb) +{ + struct page *page; + dma_addr_t dma; + unsigned int order = tp->rx_buf_page_order; + + //get free page + page = dev_alloc_pages(order); + + if (unlikely(!page)) + return -ENOMEM; + + dma = dma_map_page_attrs(&tp->pci_dev->dev, page, 0, + tp->rx_buf_page_size, + DMA_FROM_DEVICE, + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)); + + if (unlikely(dma_mapping_error(&tp->pci_dev->dev, dma))) { + __free_pages(page, order); + return -ENOMEM; + } + + rxb->page = page; + rxb->data = page_address(page); + rxb->page_offset = ring->rx_offset; + rxb->dma = dma; + + //after page alloc, page refcount already = 1 + + return 0; +} + +static void +rtl8127_free_rx_page(struct rtl8127_private *tp, struct rtl8127_rx_buffer *rxb) +{ + if (!rxb->page) + return; + + dma_unmap_page_attrs(&tp->pci_dev->dev, rxb->dma, + tp->rx_buf_page_size, + DMA_FROM_DEVICE, + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)); + __free_pages(rxb->page, tp->rx_buf_page_order); + rxb->page = NULL; +} + +static void +_rtl8127_rx_clear(struct rtl8127_private *tp, struct rtl8127_rx_ring *ring) +{ + int i; + struct rtl8127_rx_buffer *rxb; + + for (i = 0; i < ring->num_rx_desc; i++) { + rxb = &ring->rx_buffer[i]; + if (rxb->skb) { + dev_kfree_skb(rxb->skb); + rxb->skb = NULL; + } + rtl8127_free_rx_page(tp, rxb); + } +} + +static u32 +rtl8127_rx_fill(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct net_device *dev, + u32 start, + u32 end, + u8 in_intr) +{ + u32 cur; + struct rtl8127_rx_buffer *rxb; + + for (cur = start; end - cur > 0; cur++) { + int ret, i = cur % ring->num_rx_desc; + + rxb = &ring->rx_buffer[i]; + if (rxb->page) + continue; + + ret = rtl8127_alloc_rx_page(tp, ring, rxb); + if (ret) + break; + + dma_sync_single_range_for_device(tp_to_dev(tp), + rxb->dma, + rxb->page_offset, + tp->rx_buf_sz, + DMA_FROM_DEVICE); + + rtl8127_map_to_asic(tp, ring, + rtl8127_get_rxdesc(tp, ring->RxDescArray, i), + rxb->dma + rxb->page_offset, + tp->rx_buf_sz, i); + } + return cur - start; +} + +#else //ENABLE_PAGE_REUSE + +static void +rtl8127_free_rx_skb(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct sk_buff **sk_buff, + struct RxDesc *desc, + const u32 cur_rx) +{ + struct pci_dev *pdev = tp->pci_dev; + + dma_unmap_single(&pdev->dev, ring->RxDescPhyAddr[cur_rx], tp->rx_buf_sz, + DMA_FROM_DEVICE); + dev_kfree_skb(*sk_buff); + *sk_buff = NULL; + rtl8127_make_unusable_by_asic(tp, desc); +} + +static int +rtl8127_alloc_rx_skb(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct sk_buff **sk_buff, + struct RxDesc *desc, + int rx_buf_sz, + const u32 cur_rx, + u8 in_intr) +{ + struct sk_buff *skb; + dma_addr_t mapping; + int ret = 0; + + if (in_intr) + skb = RTL_ALLOC_SKB_INTR(&tp->r8127napi[ring->index].napi, rx_buf_sz + R8127_RX_ALIGN); + else + skb = dev_alloc_skb(rx_buf_sz + R8127_RX_ALIGN); + + if (unlikely(!skb)) + goto err_out; + + if (!in_intr || !R8127_USE_NAPI_ALLOC_SKB) + skb_reserve(skb, R8127_RX_ALIGN); + + mapping = dma_map_single(tp_to_dev(tp), skb->data, rx_buf_sz, + DMA_FROM_DEVICE); + if (unlikely(dma_mapping_error(tp_to_dev(tp), mapping))) { + if (unlikely(net_ratelimit())) + netif_err(tp, drv, tp->dev, "Failed to map RX DMA!\n"); + goto err_out; + } + + *sk_buff = skb; + rtl8127_map_to_asic(tp, ring, desc, mapping, rx_buf_sz, cur_rx); +out: + return ret; + +err_out: + if (skb) + dev_kfree_skb(skb); + ret = -ENOMEM; + rtl8127_make_unusable_by_asic(tp, desc); + goto out; +} + +static void +_rtl8127_rx_clear(struct rtl8127_private *tp, struct rtl8127_rx_ring *ring) +{ + int i; + + for (i = 0; i < ring->num_rx_desc; i++) { + if (ring->Rx_skbuff[i]) { + rtl8127_free_rx_skb(tp, + ring, + ring->Rx_skbuff + i, + rtl8127_get_rxdesc(tp, ring->RxDescArray, i), + i); + ring->Rx_skbuff[i] = NULL; + } + } +} + +static u32 +rtl8127_rx_fill(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct net_device *dev, + u32 start, + u32 end, + u8 in_intr) +{ + u32 cur; + + for (cur = start; end - cur > 0; cur++) { + int ret, i = cur % ring->num_rx_desc; + + if (ring->Rx_skbuff[i]) + continue; + + ret = rtl8127_alloc_rx_skb(tp, + ring, + ring->Rx_skbuff + i, + rtl8127_get_rxdesc(tp, ring->RxDescArray, i), + tp->rx_buf_sz, + i, + in_intr); + if (ret < 0) + break; + } + return cur - start; +} + +#endif //ENABLE_PAGE_REUSE + +void +rtl8127_rx_clear(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + + _rtl8127_rx_clear(tp, ring); + } +} + +static void +rtl8127_mark_as_last_descriptor_v1(struct RxDesc *desc) +{ + desc->opts1 |= cpu_to_le32(RingEnd); +} + +static void +rtl8127_mark_as_last_descriptor_v3(struct RxDescV3 *descv3) +{ + descv3->RxDescNormalDDWord4.opts1 |= cpu_to_le32(RingEnd); +} + +static void +rtl8127_mark_as_last_descriptor_v4(struct RxDescV4 *descv4) +{ + descv4->RxDescNormalDDWord2.opts1 |= cpu_to_le32(RingEnd); +} + +void +rtl8127_mark_as_last_descriptor(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + rtl8127_mark_as_last_descriptor_v3((struct RxDescV3 *)desc); + break; + case RX_DESC_RING_TYPE_4: + rtl8127_mark_as_last_descriptor_v4((struct RxDescV4 *)desc); + break; + default: + rtl8127_mark_as_last_descriptor_v1(desc); + break; + } +} + +static void +rtl8127_desc_addr_fill(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + RTL_W32(tp, ring->tdsar_reg, ((u64)ring->TxPhyAddr & DMA_BIT_MASK(32))); + RTL_W32(tp, ring->tdsar_reg + 4, ((u64)ring->TxPhyAddr >> 32)); + } + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + RTL_W32(tp, ring->rdsar_reg, ((u64)ring->RxPhyAddr & DMA_BIT_MASK(32))); + RTL_W32(tp, ring->rdsar_reg + 4, ((u64)ring->RxPhyAddr >> 32)); + } +} + +static void +rtl8127_tx_desc_init(struct rtl8127_private *tp) +{ + int i = 0; + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + memset(ring->TxDescArray, 0x0, ring->TxDescAllocSize); + + ring->TxDescArray[ring->num_tx_desc - 1].opts1 = cpu_to_le32(RingEnd); + } +} + +static void +rtl8127_rx_desc_init(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; + memset(ring->RxDescArray, 0x0, ring->RxDescAllocSize); + } +} + +int +rtl8127_init_ring(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + + rtl8127_init_ring_indexes(tp); + + rtl8127_tx_desc_init(tp); + rtl8127_rx_desc_init(tp); + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + memset(ring->tx_skb, 0x0, sizeof(ring->tx_skb)); + } + + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring = &tp->rx_ring[i]; +#ifdef ENABLE_PAGE_REUSE + ring->rx_offset = R8127_RX_ALIGN; +#else + memset(ring->Rx_skbuff, 0x0, sizeof(ring->Rx_skbuff)); +#endif //ENABLE_PAGE_REUSE + if (rtl8127_rx_fill(tp, ring, dev, 0, ring->num_rx_desc, 0) != ring->num_rx_desc) + goto err_out; + + rtl8127_mark_as_last_descriptor(tp, rtl8127_get_rxdesc(tp, ring->RxDescArray, ring->num_rx_desc - 1)); + } + + return 0; + +err_out: + rtl8127_rx_clear(tp); + return -ENOMEM; +} + +static void +rtl8127_unmap_tx_skb(struct pci_dev *pdev, + struct ring_info *tx_skb, + struct TxDesc *desc) +{ + unsigned int len = tx_skb->len; + + dma_unmap_single(&pdev->dev, le64_to_cpu(desc->addr), len, DMA_TO_DEVICE); + + desc->opts1 = cpu_to_le32(RTK_MAGIC_DEBUG_VALUE); + desc->opts2 = 0x00; + desc->addr = RTL8127_MAGIC_NUMBER; + tx_skb->len = 0; +} + +static void +rtl8127_tx_clear_range(struct rtl8127_private *tp, + struct rtl8127_tx_ring *ring, + u32 start, + unsigned int n) +{ + unsigned int i; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,22) + struct net_device *dev = tp->dev; +#endif + + for (i = 0; i < n; i++) { + unsigned int entry = (start + i) % ring->num_tx_desc; + struct ring_info *tx_skb = ring->tx_skb + entry; + unsigned int len = tx_skb->len; + + if (len) { + struct sk_buff *skb = tx_skb->skb; + + rtl8127_unmap_tx_skb(tp->pci_dev, tx_skb, + ring->TxDescArray + entry); + if (skb) { + RTLDEV->stats.tx_dropped++; + dev_kfree_skb_any(skb); + tx_skb->skb = NULL; + } + } + } +} + +void +rtl8127_tx_clear(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->num_tx_rings; i++) { + struct rtl8127_tx_ring *ring = &tp->tx_ring[i]; + rtl8127_tx_clear_range(tp, ring, ring->dirty_tx, ring->num_tx_desc); + ring->cur_tx = ring->dirty_tx = 0; + } +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +static void rtl8127_schedule_reset_work(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + set_bit(R8127_FLAG_TASK_RESET_PENDING, tp->task_flags); + schedule_delayed_work(&tp->reset_task, 4); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) +} + +static void rtl8127_schedule_esd_work(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + set_bit(R8127_FLAG_TASK_ESD_CHECK_PENDING, tp->task_flags); + schedule_delayed_work(&tp->esd_task, RTL8127_ESD_TIMEOUT); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) +} + +static void rtl8127_schedule_linkchg_work(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + set_bit(R8127_FLAG_TASK_LINKCHG_CHECK_PENDING, tp->task_flags); + schedule_delayed_work(&tp->linkchg_task, 4); +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) +} + +#define rtl8127_cancel_schedule_reset_work(a) +#define rtl8127_cancel_schedule_esd_work(a) +#define rtl8127_cancel_schedule_linkchg_work(a) + +#else +static void rtl8127_schedule_reset_work(struct rtl8127_private *tp) +{ + set_bit(R8127_FLAG_TASK_RESET_PENDING, tp->task_flags); + schedule_delayed_work(&tp->reset_task, 4); +} + +static void rtl8127_cancel_schedule_reset_work(struct rtl8127_private *tp) +{ + struct work_struct *work = &tp->reset_task.work; + + if (!work->func) + return; + + cancel_delayed_work_sync(&tp->reset_task); +} + +static void rtl8127_schedule_esd_work(struct rtl8127_private *tp) +{ + set_bit(R8127_FLAG_TASK_ESD_CHECK_PENDING, tp->task_flags); + schedule_delayed_work(&tp->esd_task, RTL8127_ESD_TIMEOUT); +} + +static void rtl8127_cancel_schedule_esd_work(struct rtl8127_private *tp) +{ + struct work_struct *work = &tp->esd_task.work; + + if (!work->func) + return; + + cancel_delayed_work_sync(&tp->esd_task); +} + +static void rtl8127_schedule_linkchg_work(struct rtl8127_private *tp) +{ + set_bit(R8127_FLAG_TASK_LINKCHG_CHECK_PENDING, tp->task_flags); + schedule_delayed_work(&tp->linkchg_task, 4); +} + +static void rtl8127_cancel_schedule_linkchg_work(struct rtl8127_private *tp) +{ + struct work_struct *work = &tp->linkchg_task.work; + + if (!work->func) + return; + + cancel_delayed_work_sync(&tp->linkchg_task); +} +#endif + +static void rtl8127_init_all_schedule_work(struct rtl8127_private *tp) +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) + INIT_WORK(&tp->reset_task, rtl8127_reset_task, dev); + INIT_WORK(&tp->esd_task, rtl8127_esd_task, dev); + INIT_WORK(&tp->linkchg_task, rtl8127_linkchg_task, dev); +#else + INIT_DELAYED_WORK(&tp->reset_task, rtl8127_reset_task); + INIT_DELAYED_WORK(&tp->esd_task, rtl8127_esd_task); + INIT_DELAYED_WORK(&tp->linkchg_task, rtl8127_linkchg_task); +#endif +} + +static void rtl8127_cancel_all_schedule_work(struct rtl8127_private *tp) +{ + rtl8127_cancel_schedule_reset_work(tp); + rtl8127_cancel_schedule_esd_work(tp); + rtl8127_cancel_schedule_linkchg_work(tp); +} + +static void +rtl8127_wait_for_irq_complete(struct rtl8127_private *tp) +{ + if (tp->features & RTL_FEATURE_MSIX) { + int i; + for (i = 0; i < tp->irq_nvecs; i++) + synchronize_irq(tp->irq_tbl[i].vector); + } else { + synchronize_irq(tp->dev->irq); + } +} + +void +_rtl8127_wait_for_quiescence(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + /* Wait for any pending NAPI task to complete */ +#ifdef CONFIG_R8127_NAPI + rtl8127_disable_napi(tp); +#endif//CONFIG_R8127_NAPI + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,5,67) + /* Give a racing hard_start_xmit a few cycles to complete. */ + synchronize_net(); +#endif + + rtl8127_irq_mask_and_ack(tp); + + rtl8127_wait_for_irq_complete(tp); +} + +static void +rtl8127_wait_for_quiescence(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + //suppress unused variable + (void)(tp); + + _rtl8127_wait_for_quiescence(dev); + +#ifdef CONFIG_R8127_NAPI + rtl8127_enable_napi(tp); +#endif//CONFIG_R8127_NAPI +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +static void rtl8127_reset_task(void *_data) +{ + struct net_device *dev = _data; + struct rtl8127_private *tp = netdev_priv(dev); +#else +static void rtl8127_reset_task(struct work_struct *work) +{ + struct rtl8127_private *tp = + container_of(work, struct rtl8127_private, reset_task.work); + struct net_device *dev = tp->dev; +#endif + int i; + + rtnl_lock(); + + if (!netif_running(dev) || + test_bit(R8127_FLAG_DOWN, tp->task_flags) || + !test_and_clear_bit(R8127_FLAG_TASK_RESET_PENDING, tp->task_flags)) + goto out_unlock; + + netdev_err(dev, "Device reseting!\n"); + + netif_carrier_off(dev); + netif_tx_disable(dev); + _rtl8127_wait_for_quiescence(dev); + rtl8127_hw_reset(dev); + + rtl8127_tx_clear(tp); + + rtl8127_init_ring_indexes(tp); + + rtl8127_tx_desc_init(tp); + for (i = 0; i < tp->num_rx_rings; i++) { + struct rtl8127_rx_ring *ring; + u32 entry; + + ring = &tp->rx_ring[i]; + for (entry = 0; entry < ring->num_rx_desc; entry++) { + struct RxDesc *desc; + + desc = rtl8127_get_rxdesc(tp, ring->RxDescArray, entry); + rtl8127_mark_to_asic(tp, desc, tp->rx_buf_sz); + } + } + +#ifdef ENABLE_PTP_SUPPORT + rtl8127_ptp_reset(tp); +#endif + +#ifdef CONFIG_R8127_NAPI + rtl8127_enable_napi(tp); +#endif //CONFIG_R8127_NAPI + + if (tp->resume_not_chg_speed) { + _rtl8127_check_link_status(dev, R8127_LINK_STATE_UNKNOWN); + + tp->resume_not_chg_speed = 0; + } else { + rtl8127_enable_hw_linkchg_interrupt(tp); + + rtl8127_set_speed(dev, tp->autoneg, tp->speed, tp->duplex, tp->advertising); + } + +out_unlock: + rtnl_unlock(); +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +static void rtl8127_esd_task(void *_data) +{ + struct net_device *dev = _data; + struct rtl8127_private *tp = netdev_priv(dev); +#else +static void rtl8127_esd_task(struct work_struct *work) +{ + struct rtl8127_private *tp = + container_of(work, struct rtl8127_private, esd_task.work); + struct net_device *dev = tp->dev; +#endif + rtnl_lock(); + + if (!netif_running(dev) || + test_bit(R8127_FLAG_DOWN, tp->task_flags) || + !test_and_clear_bit(R8127_FLAG_TASK_ESD_CHECK_PENDING, tp->task_flags)) + goto out_unlock; + + rtl8127_esd_checker(tp); + + rtl8127_schedule_esd_work(tp); + +out_unlock: + rtnl_unlock(); +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,20) +static void rtl8127_linkchg_task(void *_data) +{ + struct net_device *dev = _data; + //struct rtl8127_private *tp = netdev_priv(dev); +#else +static void rtl8127_linkchg_task(struct work_struct *work) +{ + struct rtl8127_private *tp = + container_of(work, struct rtl8127_private, linkchg_task.work); + struct net_device *dev = tp->dev; +#endif + rtnl_lock(); + + if (!netif_running(dev) || + test_bit(R8127_FLAG_DOWN, tp->task_flags) || + !test_and_clear_bit(R8127_FLAG_TASK_LINKCHG_CHECK_PENDING, tp->task_flags)) + goto out_unlock; + + rtl8127_check_link_status(dev); + +out_unlock: + rtnl_unlock(); +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,6,0) +static void +rtl8127_tx_timeout(struct net_device *dev, unsigned int txqueue) +#else +static void +rtl8127_tx_timeout(struct net_device *dev) +#endif +{ + struct rtl8127_private *tp = netdev_priv(dev); + + netdev_err(dev, "Transmit timeout reset Device!\n"); + + /* Let's wait a bit while any (async) irq lands on */ + rtl8127_schedule_reset_work(tp); +} + +static u32 +rtl8127_get_txd_opts1(struct rtl8127_tx_ring *ring, + u32 opts1, + u32 len, + unsigned int entry) +{ + u32 status = opts1 | len; + + if (entry == ring->num_tx_desc - 1) + status |= RingEnd; + + return status; +} + +static int +rtl8127_xmit_frags(struct rtl8127_private *tp, + struct rtl8127_tx_ring *ring, + struct sk_buff *skb, + const u32 *opts) +{ + struct skb_shared_info *info = skb_shinfo(skb); + unsigned int cur_frag, entry; + struct TxDesc *txd = NULL; + const unsigned char nr_frags = info->nr_frags; + unsigned long PktLenCnt = 0; + bool LsoPatchEnabled = FALSE; + + entry = ring->cur_tx; + for (cur_frag = 0; cur_frag < nr_frags; cur_frag++) { + skb_frag_t *frag = info->frags + cur_frag; + dma_addr_t mapping; + u32 status, len; + void *addr; + + entry = (entry + 1) % ring->num_tx_desc; + + txd = ring->TxDescArray + entry; +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,2,0) + len = frag->size; + addr = ((void *) page_address(frag->page)) + frag->page_offset; +#else + len = skb_frag_size(frag); + addr = skb_frag_address(frag); +#endif + mapping = dma_map_single(tp_to_dev(tp), addr, len, DMA_TO_DEVICE); + + if (unlikely(dma_mapping_error(tp_to_dev(tp), mapping))) { + if (unlikely(net_ratelimit())) + netif_err(tp, drv, tp->dev, + "Failed to map TX fragments DMA!\n"); + goto err_out; + } + + /* anti gcc 2.95.3 bugware (sic) */ + status = rtl8127_get_txd_opts1(ring, opts[0], len, entry); + if (cur_frag == (nr_frags - 1) || LsoPatchEnabled == TRUE) + status |= LastFrag; + + txd->addr = cpu_to_le64(mapping); + + ring->tx_skb[entry].len = len; + + txd->opts2 = cpu_to_le32(opts[1]); + wmb(); + txd->opts1 = cpu_to_le32(status); + + PktLenCnt += len; + } + + return cur_frag; + +err_out: + rtl8127_tx_clear_range(tp, ring, ring->cur_tx + 1, cur_frag); + return -EIO; +} + +static inline +__be16 get_protocol(struct sk_buff *skb) +{ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,37) + return vlan_get_protocol(skb); +#else + __be16 protocol; + + if (skb->protocol == htons(ETH_P_8021Q)) + protocol = vlan_eth_hdr(skb)->h_vlan_encapsulated_proto; + else + protocol = skb->protocol; + + return protocol; +#endif +} + +static inline +u8 rtl8127_get_l4_protocol(struct sk_buff *skb) +{ + int no = skb_network_offset(skb); + struct ipv6hdr *i6h, _i6h; + struct iphdr *ih, _ih; + u8 ip_protocol = IPPROTO_RAW; + + switch (get_protocol(skb)) { + case __constant_htons(ETH_P_IP): + ih = skb_header_pointer(skb, no, sizeof(_ih), &_ih); + if (ih) + ip_protocol = ih->protocol; + break; + case __constant_htons(ETH_P_IPV6): + i6h = skb_header_pointer(skb, no, sizeof(_i6h), &_i6h); + if (i6h) + ip_protocol = i6h->nexthdr; + break; + } + + return ip_protocol; +} + +static bool rtl8127_skb_pad_with_len(struct sk_buff *skb, unsigned int len) +{ + if (skb_padto(skb, len)) + return false; + skb_put(skb, len - skb->len); + return true; +} + +static bool rtl8127_skb_pad(struct sk_buff *skb) +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,19,0) + return rtl8127_skb_pad_with_len(skb, ETH_ZLEN); +#else + return !eth_skb_pad(skb); +#endif +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,14,0) +/* msdn_giant_send_check() + * According to the document of microsoft, the TCP Pseudo Header excludes the + * packet length for IPv6 TCP large packets. + */ +static int msdn_giant_send_check(struct sk_buff *skb) +{ + const struct ipv6hdr *ipv6h; + struct tcphdr *th; + int ret; + + ret = skb_cow_head(skb, 0); + if (ret) + return ret; + + ipv6h = ipv6_hdr(skb); + th = tcp_hdr(skb); + + th->check = 0; + th->check = ~tcp_v6_check(0, &ipv6h->saddr, &ipv6h->daddr, 0); + + return ret; +} +#endif + +static bool rtl8127_require_pad_ptp_pkt(struct rtl8127_private *tp) +{ + return false; +} + +#define MIN_PATCH_LEN (47) +static u32 +rtl8127_get_patch_pad_len(struct rtl8127_private *tp, + struct sk_buff *skb) +{ + u32 pad_len = 0; + int trans_data_len; + u32 hdr_len; + u32 pkt_len = skb->len; + u8 ip_protocol; + bool has_trans = skb_transport_header_was_set(skb); + + if (!rtl8127_require_pad_ptp_pkt(tp)) + goto no_padding; + + if (!(has_trans && (pkt_len < 175))) //128 + MIN_PATCH_LEN + goto no_padding; + + ip_protocol = rtl8127_get_l4_protocol(skb); + if (!(ip_protocol == IPPROTO_TCP || ip_protocol == IPPROTO_UDP)) + goto no_padding; + + trans_data_len = pkt_len - + (skb->transport_header - + skb_headroom(skb)); + if (ip_protocol == IPPROTO_UDP) { + if (trans_data_len > 3 && trans_data_len < MIN_PATCH_LEN) { + u16 dest_port = 0; + + skb_copy_bits(skb, skb->transport_header - skb_headroom(skb) + 2, &dest_port, 2); + dest_port = ntohs(dest_port); + + if (dest_port == 0x13f || + dest_port == 0x140) { + pad_len = MIN_PATCH_LEN - trans_data_len; + goto out; + } + } + } + + hdr_len = 0; + if (ip_protocol == IPPROTO_TCP) + hdr_len = 20; + else if (ip_protocol == IPPROTO_UDP) + hdr_len = 8; + if (trans_data_len < hdr_len) + pad_len = hdr_len - trans_data_len; + +out: + if ((pkt_len + pad_len) < ETH_ZLEN) + pad_len = ETH_ZLEN - pkt_len; + + return pad_len; + +no_padding: + + return 0; +} + +static bool +rtl8127_tso_csum(struct sk_buff *skb, + struct net_device *dev, + u32 *opts, + unsigned int *bytecount, + unsigned short *gso_segs) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned long large_send = 0; + u32 csum_cmd = 0; + u8 sw_calc_csum = false; + u8 check_patch_required = true; + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + if (dev->features & (NETIF_F_TSO | NETIF_F_TSO6)) { +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,18) + u32 mss = skb_shinfo(skb)->tso_size; +#else + u32 mss = skb_shinfo(skb)->gso_size; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,18) + + /* TCP Segmentation Offload (or TCP Large Send) */ + if (mss) { + union { + struct iphdr *v4; + struct ipv6hdr *v6; + unsigned char *hdr; + } ip; + union { + struct tcphdr *tcp; + struct udphdr *udp; + unsigned char *hdr; + } l4; + u32 l4_offset, hdr_len; + + ip.hdr = skb_network_header(skb); + l4.hdr = skb_checksum_start(skb); + + l4_offset = skb_transport_offset(skb); + assert((l4_offset%2) == 0); + switch (get_protocol(skb)) { + case __constant_htons(ETH_P_IP): + if (l4_offset <= GTTCPHO_MAX) { + opts[0] |= GiantSendv4; + opts[0] |= l4_offset << GTTCPHO_SHIFT; + opts[1] |= min(mss, MSS_MAX) << 18; + large_send = 1; + } + break; + case __constant_htons(ETH_P_IPV6): +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,14,0) + if (msdn_giant_send_check(skb)) + return false; +#endif + if (l4_offset <= GTTCPHO_MAX) { + opts[0] |= GiantSendv6; + opts[0] |= l4_offset << GTTCPHO_SHIFT; + opts[1] |= min(mss, MSS_MAX) << 18; + large_send = 1; + } + break; + default: + if (unlikely(net_ratelimit())) + dprintk("tso proto=%x!\n", skb->protocol); + break; + } + + if (large_send == 0) + return false; + + + /* compute length of segmentation header */ + hdr_len = (l4.tcp->doff * 4) + l4_offset; + /* update gso size and bytecount with header size */ + *gso_segs = skb_shinfo(skb)->gso_segs; + *bytecount += (*gso_segs - 1) * hdr_len; + + return true; + } + } +#endif //LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + + if (skb->ip_summed == CHECKSUM_PARTIAL) { +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,22) + const struct iphdr *ip = skb->nh.iph; + + if (dev->features & NETIF_F_IP_CSUM) { + if (ip->protocol == IPPROTO_TCP) + csum_cmd = tp->tx_ip_csum_cmd | tp->tx_tcp_csum_cmd; + else if (ip->protocol == IPPROTO_UDP) + csum_cmd = tp->tx_ip_csum_cmd | tp->tx_udp_csum_cmd; + else if (ip->protocol == IPPROTO_IP) + csum_cmd = tp->tx_ip_csum_cmd; + } +#else + u8 ip_protocol = IPPROTO_RAW; + + switch (get_protocol(skb)) { + case __constant_htons(ETH_P_IP): + if (dev->features & NETIF_F_IP_CSUM) { + ip_protocol = ip_hdr(skb)->protocol; + csum_cmd = tp->tx_ip_csum_cmd; + } + break; + case __constant_htons(ETH_P_IPV6): + if (dev->features & NETIF_F_IPV6_CSUM) { + if (skb_transport_offset(skb) > 0 && skb_transport_offset(skb) <= TCPHO_MAX) { + ip_protocol = ipv6_hdr(skb)->nexthdr; + csum_cmd = tp->tx_ipv6_csum_cmd; + csum_cmd |= skb_transport_offset(skb) << TCPHO_SHIFT; + } + } + break; + default: + if (unlikely(net_ratelimit())) + dprintk("checksum_partial proto=%x!\n", skb->protocol); + break; + } + + if (ip_protocol == IPPROTO_TCP) + csum_cmd |= tp->tx_tcp_csum_cmd; + else if (ip_protocol == IPPROTO_UDP) + csum_cmd |= tp->tx_udp_csum_cmd; +#endif + if (csum_cmd == 0) { + sw_calc_csum = true; +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + WARN_ON(1); /* we need a WARN() */ +#endif + } + + if (ip_protocol == IPPROTO_TCP) + check_patch_required = false; + } + + if (check_patch_required) { + u32 pad_len = rtl8127_get_patch_pad_len(tp, skb); + + if (pad_len > 0) { + if (!rtl8127_skb_pad_with_len(skb, skb->len + pad_len)) + return false; + + if (csum_cmd != 0) + sw_calc_csum = true; + } + } + + if (skb->len < ETH_ZLEN) { + if (tp->UseSwPaddingShortPkt || + (tp->ShortPacketSwChecksum && csum_cmd != 0)) { + if (!rtl8127_skb_pad(skb)) + return false; + + if (csum_cmd != 0) + sw_calc_csum = true; + } + } + + if (sw_calc_csum) { +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10) && LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,7) + skb_checksum_help(&skb, 0); +#elif LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) && LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,10) + skb_checksum_help(skb, 0); +#else + skb_checksum_help(skb); +#endif + } else + opts[1] |= csum_cmd; + + return true; +} + +static bool rtl8127_tx_slots_avail(struct rtl8127_private *tp, + struct rtl8127_tx_ring *ring) +{ + unsigned int slots_avail = READ_ONCE(ring->dirty_tx) + ring->num_tx_desc + - READ_ONCE(ring->cur_tx); + + /* A skbuff with nr_frags needs nr_frags+1 entries in the tx queue */ + return slots_avail > MAX_SKB_FRAGS; +} + +static inline u32 +rtl8127_fast_mod_mask(const u32 input, const u32 mask) +{ + return input > mask ? input & mask : input; +} + +static void rtl8127_doorbell(struct rtl8127_private *tp, + struct rtl8127_tx_ring *ring) +{ + if (tp->EnableTxNoClose) { + if (tp->HwSuppTxNoCloseVer > 3) + RTL_W32(tp, ring->sw_tail_ptr_reg, ring->cur_tx); + else + RTL_W16(tp, ring->sw_tail_ptr_reg, ring->cur_tx); + } else { + /* set polling bit */ + RTL_W32(tp, TPPOLL_8125, BIT(ring->index)); + } +} + +static netdev_tx_t +rtl8127_start_xmit(struct sk_buff *skb, + struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + unsigned int bytecount; + unsigned short gso_segs; + struct ring_info *last; + unsigned int last_entry; + unsigned int entry; + struct TxDesc *txd; + dma_addr_t mapping; + u32 len; + u32 opts[2]; + netdev_tx_t ret = NETDEV_TX_OK; + int frags; + u8 EnableTxNoClose = tp->EnableTxNoClose; + const u16 queue_mapping = skb_get_queue_mapping(skb); + struct rtl8127_tx_ring *ring; + bool stop_queue; + + assert(queue_mapping < tp->num_tx_rings); + + ring = &tp->tx_ring[queue_mapping]; + + if (unlikely(!rtl8127_tx_slots_avail(tp, ring))) { + if (netif_msg_drv(tp)) { + printk(KERN_ERR + "%s: BUG! Tx Ring[%d] full when queue awake!\n", + dev->name, + queue_mapping); + } + goto err_stop; + } + + entry = ring->cur_tx % ring->num_tx_desc; + txd = ring->TxDescArray + entry; + + if (!EnableTxNoClose) { + if (unlikely(le32_to_cpu(txd->opts1) & DescOwn)) { + if (netif_msg_drv(tp)) { + printk(KERN_ERR + "%s: BUG! Tx Desc is own by hardware!\n", + dev->name); + } + goto err_stop; + } + } + + bytecount = skb->len; + gso_segs = 1; + + opts[0] = DescOwn; + opts[1] = rtl8127_tx_vlan_tag(tp, skb); + + if (unlikely(!rtl8127_tso_csum(skb, dev, opts, &bytecount, &gso_segs))) + goto err_dma_0; + + frags = rtl8127_xmit_frags(tp, ring, skb, opts); + if (unlikely(frags < 0)) + goto err_dma_0; + if (frags) { + len = skb_headlen(skb); + opts[0] |= FirstFrag; + } else { + len = skb->len; + opts[0] |= FirstFrag | LastFrag; + } + + opts[0] = rtl8127_get_txd_opts1(ring, opts[0], len, entry); + mapping = dma_map_single(tp_to_dev(tp), skb->data, len, DMA_TO_DEVICE); + if (unlikely(dma_mapping_error(tp_to_dev(tp), mapping))) { + if (unlikely(net_ratelimit())) + netif_err(tp, drv, dev, "Failed to map TX DMA!\n"); + goto err_dma_1; + } + +#ifdef ENABLE_PTP_SUPPORT + if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) { + if (!test_and_set_bit_lock(__RTL8127_PTP_TX_IN_PROGRESS, &tp->state)) { + if (tp->hwtstamp_config.tx_type == HWTSTAMP_TX_ON && + !tp->ptp_tx_skb) { + skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; + + tp->ptp_tx_skb = skb_get(skb); + tp->ptp_tx_start = jiffies; + schedule_work(&tp->ptp_tx_work); + } else + tp->tx_hwtstamp_skipped++; + } + } +#endif + /* set first fragment's length */ + ring->tx_skb[entry].len = len; + + /* set skb to last fragment */ + last_entry = (entry + frags) % ring->num_tx_desc; + last = &ring->tx_skb[last_entry]; + last->skb = skb; + last->gso_segs = gso_segs; + last->bytecount = bytecount; + + txd->addr = cpu_to_le64(mapping); + txd->opts2 = cpu_to_le32(opts[1]); + wmb(); + txd->opts1 = cpu_to_le32(opts[0]); + + netdev_tx_sent_queue(txring_txq(ring), bytecount); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,5,0) + dev->trans_start = jiffies; +#else + skb_tx_timestamp(skb); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(3,5,0) + + /* rtl_tx needs to see descriptor changes before updated tp->cur_tx */ + smp_wmb(); + + WRITE_ONCE(ring->cur_tx, ring->cur_tx + frags + 1); + + stop_queue = !rtl8127_tx_slots_avail(tp, ring); + if (unlikely(stop_queue)) { + /* Avoid wrongly optimistic queue wake-up: rtl_tx thread must + * not miss a ring update when it notices a stopped queue. + */ + smp_wmb(); + netif_stop_subqueue(dev, queue_mapping); + } + + if (netif_xmit_stopped(txring_txq(ring)) || !netdev_xmit_more()) + rtl8127_doorbell(tp, ring); + + if (unlikely(stop_queue)) { + /* Sync with rtl_tx: + * - publish queue status and cur_tx ring index (write barrier) + * - refresh dirty_tx ring index (read barrier). + * May the current thread have a pessimistic view of the ring + * status and forget to wake up queue, a racing rtl_tx thread + * can't. + */ + smp_mb(); + if (rtl8127_tx_slots_avail(tp, ring)) + netif_start_subqueue(dev, queue_mapping); + } +out: + return ret; +err_dma_1: + rtl8127_tx_clear_range(tp, ring, ring->cur_tx + 1, frags); +err_dma_0: + RTLDEV->stats.tx_dropped++; + dev_kfree_skb_any(skb); + ret = NETDEV_TX_OK; + goto out; +err_stop: + netif_stop_subqueue(dev, queue_mapping); + ret = NETDEV_TX_BUSY; + RTLDEV->stats.tx_dropped++; + goto out; +} + +/* recycle tx no close desc*/ +static int +rtl8127_tx_interrupt_noclose(struct rtl8127_tx_ring *ring, int budget) +{ + unsigned int total_bytes = 0, total_packets = 0; + struct rtl8127_private *tp = ring->priv; + struct net_device *dev = tp->dev; + unsigned int dirty_tx, tx_left; + unsigned int tx_desc_closed; + unsigned int count = 0; + + dirty_tx = ring->dirty_tx; + ring->NextHwDesCloPtr = rtl8127_get_hw_clo_ptr(ring); + tx_desc_closed = rtl8127_fast_mod_mask(ring->NextHwDesCloPtr - + ring->BeginHwDesCloPtr, + tp->MaxTxDescPtrMask); + tx_left = min((READ_ONCE(ring->cur_tx) - dirty_tx), tx_desc_closed); + ring->BeginHwDesCloPtr += tx_left; + + while (tx_left > 0) { + unsigned int entry = dirty_tx % ring->num_tx_desc; + struct ring_info *tx_skb = ring->tx_skb + entry; + + rtl8127_unmap_tx_skb(tp->pci_dev, + tx_skb, + ring->TxDescArray + entry); + + if (tx_skb->skb != NULL) { + /* update the statistics for this packet */ + total_bytes += tx_skb->bytecount; + total_packets += tx_skb->gso_segs; + + RTL_NAPI_CONSUME_SKB_ANY(tx_skb->skb, budget); + tx_skb->skb = NULL; + } + dirty_tx++; + tx_left--; + } + + if (total_packets) { + netdev_tx_completed_queue(txring_txq(ring), + total_packets, total_bytes); + + RTLDEV->stats.tx_bytes += total_bytes; + RTLDEV->stats.tx_packets+= total_packets; + } + + if (ring->dirty_tx != dirty_tx) { + count = dirty_tx - ring->dirty_tx; + WRITE_ONCE(ring->dirty_tx, dirty_tx); + smp_wmb(); + if (__netif_subqueue_stopped(dev, ring->index) && + (rtl8127_tx_slots_avail(tp, ring))) { + netif_start_subqueue(dev, ring->index); + } + } + + return count; +} + +/* recycle tx close desc*/ +static int +rtl8127_tx_interrupt_close(struct rtl8127_tx_ring *ring, int budget) +{ + unsigned int total_bytes = 0, total_packets = 0; + struct rtl8127_private *tp = ring->priv; + struct net_device *dev = tp->dev; + unsigned int dirty_tx, tx_left; + unsigned int count = 0; + + dirty_tx = ring->dirty_tx; + tx_left = READ_ONCE(ring->cur_tx) - dirty_tx; + + while (tx_left > 0) { + unsigned int entry = dirty_tx % ring->num_tx_desc; + struct ring_info *tx_skb = ring->tx_skb + entry; + + if (le32_to_cpu(READ_ONCE(ring->TxDescArray[entry].opts1)) & DescOwn) + break; + + rtl8127_unmap_tx_skb(tp->pci_dev, + tx_skb, + ring->TxDescArray + entry); + + if (tx_skb->skb != NULL) { + /* update the statistics for this packet */ + total_bytes += tx_skb->bytecount; + total_packets += tx_skb->gso_segs; + + RTL_NAPI_CONSUME_SKB_ANY(tx_skb->skb, budget); + tx_skb->skb = NULL; + } + dirty_tx++; + tx_left--; + } + + if (total_packets) { + netdev_tx_completed_queue(txring_txq(ring), + total_packets, total_bytes); + + RTLDEV->stats.tx_bytes += total_bytes; + RTLDEV->stats.tx_packets+= total_packets; + } + + if (ring->dirty_tx != dirty_tx) { + count = dirty_tx - ring->dirty_tx; + WRITE_ONCE(ring->dirty_tx, dirty_tx); + smp_wmb(); + if (__netif_subqueue_stopped(dev, ring->index) && + (rtl8127_tx_slots_avail(tp, ring))) { + netif_start_subqueue(dev, ring->index); + } + + if (READ_ONCE(ring->cur_tx) != dirty_tx) + rtl8127_doorbell(tp, ring); + } + + return count; +} + +static int +rtl8127_tx_interrupt(struct rtl8127_tx_ring *ring, int budget) +{ + struct rtl8127_private *tp = ring->priv; + + if (tp->EnableTxNoClose) + return rtl8127_tx_interrupt_noclose(ring, budget); + else + return rtl8127_tx_interrupt_close(ring, budget); +} + +static int +rtl8127_tx_interrupt_with_vector(struct rtl8127_private *tp, + const int message_id, + int budget) +{ + int count = 0; + + switch (tp->HwCurrIsrVer) { + case 3: + case 4: + if (message_id < tp->num_tx_rings) + count += rtl8127_tx_interrupt(&tp->tx_ring[message_id], budget); + break; + case 5: + if (message_id == 16) + count += rtl8127_tx_interrupt(&tp->tx_ring[0], budget); +#ifdef ENABLE_MULTIPLE_TX_QUEUE + else if (message_id == 17 && tp->num_tx_rings > 1) + count += rtl8127_tx_interrupt(&tp->tx_ring[1], budget); +#endif + break; + case 6: + if (message_id == 8) + count += rtl8127_tx_interrupt(&tp->tx_ring[0], budget); +#ifdef ENABLE_MULTIPLE_TX_QUEUE + else if (message_id == 9 && tp->num_tx_rings > 1) + count += rtl8127_tx_interrupt(&tp->tx_ring[1], budget); +#endif + break; + default: + if (message_id == 16) + count += rtl8127_tx_interrupt(&tp->tx_ring[0], budget); +#ifdef ENABLE_MULTIPLE_TX_QUEUE + else if (message_id == 18 && tp->num_tx_rings > 1) + count += rtl8127_tx_interrupt(&tp->tx_ring[1], budget); +#endif + break; + } + + return count; +} + +static inline int +rtl8127_fragmented_frame(struct rtl8127_private *tp, u32 status) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + return (status & (FirstFrag_V3 | LastFrag_V3)) != (FirstFrag_V3 | LastFrag_V3); + case RX_DESC_RING_TYPE_4: + return (status & (FirstFrag_V4 | LastFrag_V4)) != (FirstFrag_V4 | LastFrag_V4); + default: + return (status & (FirstFrag | LastFrag)) != (FirstFrag | LastFrag); + } +} + +static inline int +rtl8127_is_non_eop(struct rtl8127_private *tp, u32 status) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + return !(status & LastFrag_V3); + case RX_DESC_RING_TYPE_4: + return !(status & LastFrag_V4); + default: + return !(status & LastFrag); + } +} + +static inline int +rtl8127_rx_desc_type(u32 status) +{ + return ((status >> 26) & 0x0F); +} + +static inline void +rtl8127_rx_v1_csum(struct rtl8127_private *tp, + struct sk_buff *skb, + struct RxDesc *desc) +{ + u32 opts1 = le32_to_cpu(desc->opts1); + + if (((opts1 & RxTCPT) && !(opts1 & RxTCPF)) || + ((opts1 & RxUDPT) && !(opts1 & RxUDPF))) + skb->ip_summed = CHECKSUM_UNNECESSARY; + else + skb_checksum_none_assert(skb); +} + +static inline void +rtl8127_rx_v3_csum(struct rtl8127_private *tp, + struct sk_buff *skb, + struct RxDescV3 *descv3) +{ + u32 opts2 = le32_to_cpu(descv3->RxDescNormalDDWord4.opts2); + + /* rx csum offload for RTL8125 */ + if (((opts2 & RxTCPT_v3) && !(opts2 & RxTCPF_v3)) || + ((opts2 & RxUDPT_v3) && !(opts2 & RxUDPF_v3))) + skb->ip_summed = CHECKSUM_UNNECESSARY; + else + skb_checksum_none_assert(skb); +} + +static inline void +rtl8127_rx_v4_csum(struct rtl8127_private *tp, + struct sk_buff *skb, + struct RxDescV4 *descv4) +{ + u32 opts1 = le32_to_cpu(descv4->RxDescNormalDDWord2.opts1); + + /* rx csum offload for RTL8125 */ + if (((opts1 & RxTCPT_v4) && !(opts1 & RxTCPF_v4)) || + ((opts1 & RxUDPT_v4) && !(opts1 & RxUDPF_v4))) + skb->ip_summed = CHECKSUM_UNNECESSARY; + else + skb_checksum_none_assert(skb); +} + +static inline void +rtl8127_rx_csum(struct rtl8127_private *tp, + struct sk_buff *skb, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + rtl8127_rx_v3_csum(tp, skb, (struct RxDescV3 *)desc); + break; + case RX_DESC_RING_TYPE_4: + rtl8127_rx_v4_csum(tp, skb, (struct RxDescV4 *)desc); + break; + default: + rtl8127_rx_v1_csum(tp, skb, desc); + break; + } +} + +/* +static inline int +rtl8127_try_rx_copy(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + struct sk_buff **sk_buff, + int pkt_size, + struct RxDesc *desc, + int rx_buf_sz) +{ + int ret = -1; + + struct sk_buff *skb; + + skb = RTL_ALLOC_SKB_INTR(&tp->r8127napi[ring->index].napi, pkt_size + R8127_RX_ALIGN); + if (skb) { + u8 *data; + + data = sk_buff[0]->data; + if (!R8127_USE_NAPI_ALLOC_SKB) + skb_reserve(skb, R8127_RX_ALIGN); +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,4,37) + prefetch(data - R8127_RX_ALIGN); +#endif + eth_copy_and_sum(skb, data, pkt_size, 0); + *sk_buff = skb; + rtl8127_mark_to_asic(tp, desc, rx_buf_sz); + ret = 0; + } + + return ret; +} +*/ + +static inline void +rtl8127_rx_skb(struct rtl8127_private *tp, + struct sk_buff *skb, + u32 ring_index) +{ +#ifdef CONFIG_R8127_NAPI +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) + netif_receive_skb(skb); +#else + napi_gro_receive(&tp->r8127napi[ring_index].napi, skb); +#endif +#else + netif_rx(skb); +#endif +} + +static int +rtl8127_check_rx_desc_error(struct net_device *dev, + struct rtl8127_private *tp, + u32 status) +{ + int ret = 0; + + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + if (unlikely(status & RxRES_V3)) { + if (status & (RxRWT_V3 | RxRUNT_V3)) + RTLDEV->stats.rx_length_errors++; + if (status & RxCRC_V3) + RTLDEV->stats.rx_crc_errors++; + + ret = -1; + } + break; + case RX_DESC_RING_TYPE_4: + if (unlikely(status & RxRES_V4)) { + if (status & RxRUNT_V4) + RTLDEV->stats.rx_length_errors++; + if (status & RxCRC_V4) + RTLDEV->stats.rx_crc_errors++; + + ret = -1; + } + break; + default: + if (unlikely(status & RxRES)) { + if (status & (RxRWT | RxRUNT)) + RTLDEV->stats.rx_length_errors++; + if (status & RxCRC) + RTLDEV->stats.rx_crc_errors++; + + ret = -1; + } + break; + } + + return ret; +} + +#ifdef ENABLE_PAGE_REUSE + +static inline bool +rtl8127_reuse_rx_ok(struct page *page) +{ + /* avoid re-using remote pages */ + if (!dev_page_is_reusable(page)) { + //printk(KERN_INFO "r8127 page pfmemalloc, can't reuse!\n"); + return false; + } + /* if we are only owner of page we can reuse it */ + if (unlikely(page_ref_count(page) != 1)) { + //printk(KERN_INFO "r8127 page refcnt %d, can't reuse!\n", page_ref_count(page)); + return false; + } + + return true; +} + +static void +rtl8127_reuse_rx_buffer(struct rtl8127_private *tp, struct rtl8127_rx_ring *ring, u32 cur_rx, struct rtl8127_rx_buffer *rxb) +{ + struct page *page = rxb->page; + + u32 dirty_rx = ring->dirty_rx; + u32 entry = dirty_rx % ring->num_rx_desc; + struct rtl8127_rx_buffer *nrxb = &ring->rx_buffer[entry]; + + u32 noffset; + + //the page gonna be shared by us and kernel, keep page ref = 2 + page_ref_inc(page); + + //flip the buffer in page to use next + noffset = rxb->page_offset ^ (tp->rx_buf_page_size / 2); //one page, two buffer, ping-pong + + nrxb->dma = rxb->dma; + nrxb->page_offset = noffset; + nrxb->data = rxb->data; + + if (cur_rx != dirty_rx) { + //move the buffer to other slot + nrxb->page = page; + rxb->page = NULL; + } +} + +static void rtl8127_put_rx_buffer(struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + u32 cur_rx, + struct rtl8127_rx_buffer *rxb) +{ + struct rtl8127_rx_buffer *nrxb; + struct page *page = rxb->page; + u32 entry; + + entry = ring->dirty_rx % ring->num_rx_desc; + nrxb = &ring->rx_buffer[entry]; + if (likely(rtl8127_reuse_rx_ok(page))) { + /* hand second half of page back to the ring */ + rtl8127_reuse_rx_buffer(tp, ring, cur_rx, rxb); + } else { + tp->page_reuse_fail_cnt++; + + dma_unmap_page_attrs(&tp->pci_dev->dev, rxb->dma, + tp->rx_buf_page_size, + DMA_FROM_DEVICE, + (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)); + //the page ref is kept 1, uniquely owned by kernel now + rxb->page = NULL; + + return; + } + + dma_sync_single_range_for_device(tp_to_dev(tp), + nrxb->dma, + nrxb->page_offset, + tp->rx_buf_sz, + DMA_FROM_DEVICE); + + rtl8127_map_to_asic(tp, ring, + rtl8127_get_rxdesc(tp, ring->RxDescArray, entry), + nrxb->dma + nrxb->page_offset, + tp->rx_buf_sz, entry); + + ring->dirty_rx++; +} + +#endif //ENABLE_PAGE_REUSE + +static int +rtl8127_rx_interrupt(struct net_device *dev, + struct rtl8127_private *tp, + struct rtl8127_rx_ring *ring, + napi_budget budget) +{ + unsigned int cur_rx, rx_left; + unsigned int delta, count = 0; + unsigned int entry; + struct RxDesc *desc; + struct sk_buff *skb; + u32 status; + u32 rx_quota; + u32 ring_index = ring->index; +#ifdef ENABLE_PAGE_REUSE + struct rtl8127_rx_buffer *rxb; +#else //ENABLE_PAGE_REUSE + u64 rx_buf_phy_addr; +#endif //ENABLE_PAGE_REUSE + unsigned int total_rx_multicast_packets = 0; + unsigned int total_rx_bytes = 0, total_rx_packets = 0; + + assert(dev != NULL); + assert(tp != NULL); + + if (ring->RxDescArray == NULL) + goto rx_out; + + rx_quota = RTL_RX_QUOTA(budget); + cur_rx = ring->cur_rx; + rx_left = ring->num_rx_desc + ring->dirty_rx - cur_rx; + rx_left = rtl8127_rx_quota(rx_left, (u32)rx_quota); + + for (; rx_left > 0; rx_left--, cur_rx++) { +#ifndef ENABLE_PAGE_REUSE + const void *rx_buf; +#endif //!ENABLE_PAGE_REUSE + u32 pkt_size; + + entry = cur_rx % ring->num_rx_desc; + desc = rtl8127_get_rxdesc(tp, ring->RxDescArray, entry); + status = le32_to_cpu(rtl8127_rx_desc_opts1(tp, desc)); + if (status & DescOwn) { + RTL_R8(tp, tp->imr_reg[0]); + status = le32_to_cpu(rtl8127_rx_desc_opts1(tp, desc)); + if (status & DescOwn) + break; + } + + rmb(); + + if (unlikely(rtl8127_check_rx_desc_error(dev, tp, status) < 0)) { + if (netif_msg_rx_err(tp)) { + printk(KERN_INFO + "%s: Rx ERROR. status = %08x\n", + dev->name, status); + } + + RTLDEV->stats.rx_errors++; + + if (!(dev->features & NETIF_F_RXALL)) + goto release_descriptor; + } + pkt_size = status & 0x00003fff; + if (likely(!(dev->features & NETIF_F_RXFCS))) { +#ifdef ENABLE_RX_PACKET_FRAGMENT + if (rtl8127_is_non_eop(tp, status) && + pkt_size == tp->rx_buf_sz) { + struct RxDesc *desc_next; + unsigned int entry_next; + int pkt_size_next; + u32 status_next; + + entry_next = (cur_rx + 1) % ring->num_rx_desc; + desc_next = rtl8127_get_rxdesc(tp, ring->RxDescArray, entry_next); + status_next = le32_to_cpu(rtl8127_rx_desc_opts1(tp, desc_next)); + if (!(status_next & DescOwn)) { + pkt_size_next = status_next & 0x00003fff; + if (pkt_size_next < ETH_FCS_LEN) + pkt_size -= (ETH_FCS_LEN - pkt_size_next); + } + } +#endif //ENABLE_RX_PACKET_FRAGMENT + if (!rtl8127_is_non_eop(tp, status)) { + if (pkt_size < ETH_FCS_LEN) { +#ifdef ENABLE_RX_PACKET_FRAGMENT + pkt_size = 0; +#else + goto drop_packet; +#endif //ENABLE_RX_PACKET_FRAGMENT + } else + pkt_size -= ETH_FCS_LEN; + } + } + + if (unlikely(pkt_size > tp->rx_buf_sz)) + goto drop_packet; + +#if !defined(ENABLE_RX_PACKET_FRAGMENT) || !defined(ENABLE_PAGE_REUSE) + /* + * The driver does not support incoming fragmented + * frames. They are seen as a symptom of over-mtu + * sized frames. + */ + if (unlikely(rtl8127_fragmented_frame(tp, status))) + goto drop_packet; +#endif //!ENABLE_RX_PACKET_FRAGMENT || !ENABLE_PAGE_REUSE + +#ifdef ENABLE_PAGE_REUSE + rxb = &ring->rx_buffer[entry]; + skb = rxb->skb; + rxb->skb = NULL; + if (!skb) { + skb = RTL_BUILD_SKB_INTR(rxb->data + rxb->page_offset - ring->rx_offset, tp->rx_buf_page_size / 2); + if (!skb) { + //netdev_err(tp->dev, "Failed to allocate RX skb!\n"); + goto drop_packet; + } + + skb->dev = dev; + if (!R8127_USE_NAPI_ALLOC_SKB) + skb_reserve(skb, R8127_RX_ALIGN); + skb_put(skb, pkt_size); +#ifdef ENABLE_RSS_SUPPORT + rtl8127_rx_hash(tp, desc, skb); +#endif + rtl8127_rx_csum(tp, skb, desc); + } else + skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rxb->page, + rxb->page_offset, pkt_size, tp->rx_buf_page_size / 2); + //recycle desc + rtl8127_put_rx_buffer(tp, ring, cur_rx, rxb); + + dma_sync_single_range_for_cpu(tp_to_dev(tp), + rxb->dma, + rxb->page_offset, + tp->rx_buf_sz, + DMA_FROM_DEVICE); +#else //ENABLE_PAGE_REUSE + skb = RTL_ALLOC_SKB_INTR(&tp->r8127napi[ring->index].napi, pkt_size + R8127_RX_ALIGN); + if (!skb) { + //netdev_err(tp->dev, "Failed to allocate RX skb!\n"); + goto drop_packet; + } + + skb->dev = dev; + if (!R8127_USE_NAPI_ALLOC_SKB) + skb_reserve(skb, R8127_RX_ALIGN); + skb_put(skb, pkt_size); + + rx_buf_phy_addr = ring->RxDescPhyAddr[entry]; + dma_sync_single_for_cpu(tp_to_dev(tp), + rx_buf_phy_addr, tp->rx_buf_sz, + DMA_FROM_DEVICE); + rx_buf = ring->Rx_skbuff[entry]->data; +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,4,37) + prefetch(rx_buf - R8127_RX_ALIGN); +#endif + eth_copy_and_sum(skb, rx_buf, pkt_size, 0); + + dma_sync_single_for_device(tp_to_dev(tp), rx_buf_phy_addr, + tp->rx_buf_sz, DMA_FROM_DEVICE); +#endif //ENABLE_PAGE_REUSE + +#ifdef ENABLE_PTP_SUPPORT + if (tp->flags & RTL_FLAG_RX_HWTSTAMP_ENABLED) + rtl8127_rx_ptp_timestamp(tp, skb); +#endif // ENABLE_PTP_SUPPORT + +#ifdef ENABLE_RX_PACKET_FRAGMENT + if (rtl8127_is_non_eop(tp, status)) { + unsigned int entry_next; + entry_next = (entry + 1) % ring->num_rx_desc; + rxb = &ring->rx_buffer[entry_next]; + rxb->skb = skb; + continue; + } +#endif //ENABLE_RX_PACKET_FRAGMENT + +#ifndef ENABLE_PAGE_REUSE +#ifdef ENABLE_RSS_SUPPORT + rtl8127_rx_hash(tp, desc, skb); +#endif + rtl8127_rx_csum(tp, skb, desc); +#endif /* !ENABLE_PAGE_REUSE */ + + skb->protocol = eth_type_trans(skb, dev); + + total_rx_bytes += skb->len; + + if (skb->pkt_type == PACKET_MULTICAST) + total_rx_multicast_packets++; + + if (rtl8127_rx_vlan_skb(tp, desc, skb) < 0) + rtl8127_rx_skb(tp, skb, ring_index); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(4,11,0) + dev->last_rx = jiffies; +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(4,11,0) + total_rx_packets++; + +#ifdef ENABLE_PAGE_REUSE + rxb->skb = NULL; + continue; +#endif + +release_descriptor: + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + case RX_DESC_RING_TYPE_4: + rtl8127_set_desc_dma_addr(tp, desc, + ring->RxDescPhyAddr[entry]); + wmb(); + break; + } + rtl8127_mark_to_asic(tp, desc, tp->rx_buf_sz); + continue; +drop_packet: + RTLDEV->stats.rx_dropped++; + RTLDEV->stats.rx_length_errors++; + goto release_descriptor; + } + + count = cur_rx - ring->cur_rx; + ring->cur_rx = cur_rx; + + delta = rtl8127_rx_fill(tp, ring, dev, ring->dirty_rx, ring->cur_rx, 1); + if (!delta && count && netif_msg_intr(tp)) + printk(KERN_INFO "%s: no Rx buffer allocated\n", dev->name); + ring->dirty_rx += delta; + + RTLDEV->stats.rx_bytes += total_rx_bytes; + RTLDEV->stats.rx_packets += total_rx_packets; + RTLDEV->stats.multicast += total_rx_multicast_packets; + + /* + * FIXME: until there is periodic timer to try and refill the ring, + * a temporary shortage may definitely kill the Rx process. + * - disable the asic to try and avoid an overflow and kick it again + * after refill ? + * - how do others driver handle this condition (Uh oh...). + */ + if ((ring->dirty_rx + ring->num_rx_desc == ring->cur_rx) && netif_msg_intr(tp)) + printk(KERN_EMERG "%s: Rx buffers exhausted\n", dev->name); + +rx_out: + return total_rx_packets; +} + +static bool +rtl8127_linkchg_interrupt(struct rtl8127_private *tp, u32 status) +{ + switch (tp->HwCurrIsrVer) { + case 2: + case 3: + return status & ISRIMR_V2_LINKCHG; + case 4: + return status & ISRIMR_V4_LINKCHG; + case 5: + return status & ISRIMR_V5_LINKCHG; + case 6: + return status & ISRIMR_V6_LINKCHG; + default: + return status & LinkChg; + } +} + +static u32 +rtl8127_get_linkchg_message_id(struct rtl8127_private *tp) +{ + switch (tp->HwCurrIsrVer) { + case 4: + case 6: + return 29; + case 5: + return 18; + default: + return 21; + } +} + +/* + *The interrupt handler does all of the Rx thread work and cleans up after + *the Tx thread. + */ +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) +static irqreturn_t rtl8127_interrupt(int irq, void *dev_instance, struct pt_regs *regs) +#else +static irqreturn_t rtl8127_interrupt(int irq, void *dev_instance) +#endif +{ + struct r8127_napi *r8127napi = dev_instance; + struct rtl8127_private *tp = r8127napi->priv; + struct net_device *dev = tp->dev; + u32 status; + int handled = 0; + + do { + status = RTL_R32(tp, tp->isr_reg[0]); + + if (!(tp->features & (RTL_FEATURE_MSI | RTL_FEATURE_MSIX))) { + /* hotplug/major error/no more work/shared irq */ + if (!status) + break; + + if (status == 0xFFFFFFFF) + break; + + if (!(status & (tp->intr_mask | tp->timer_intr_mask))) + break; + } + + handled = 1; + +#if defined(RTL_USE_NEW_INTR_API) + if (!tp->irq_tbl[0].requested) + break; +#endif + rtl8127_disable_hw_interrupt(tp); + + RTL_W32(tp, tp->isr_reg[0], status&~RxFIFOOver); + + if (rtl8127_linkchg_interrupt(tp, status)) + rtl8127_schedule_linkchg_work(tp); + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) { + if (HW_DASH_SUPPORT_TYPE_3(tp)) { + u8 DashIntType2Status; + + if (status & ISRIMR_DASH_INTR_CMAC_RESET) + tp->CmacResetIntr = TRUE; + + DashIntType2Status = RTL_CMAC_R8(tp, CMAC_IBISR0); + if (DashIntType2Status & ISRIMR_DASH_TYPE2_ROK) + tp->RcvFwDashOkEvt = TRUE; + if (DashIntType2Status & ISRIMR_DASH_TYPE2_TOK) + tp->SendFwHostOkEvt = TRUE; + if (DashIntType2Status & ISRIMR_DASH_TYPE2_RX_DISABLE_IDLE) + tp->DashFwDisableRx = TRUE; + + RTL_CMAC_W8(tp, CMAC_IBISR0, DashIntType2Status); + } + } +#endif + +#ifdef CONFIG_R8127_NAPI + if (status & tp->intr_mask || tp->keep_intr_cnt-- > 0) { + if (status & tp->intr_mask) + tp->keep_intr_cnt = RTK_KEEP_INTERRUPT_COUNT; + + if (likely(RTL_NETIF_RX_SCHEDULE_PREP(dev, &tp->r8127napi[0].napi))) + __RTL_NETIF_RX_SCHEDULE(dev, &tp->r8127napi[0].napi); + else if (netif_msg_intr(tp)) + printk(KERN_INFO "%s: interrupt %04x in poll\n", + dev->name, status); + } else { + tp->keep_intr_cnt = RTK_KEEP_INTERRUPT_COUNT; + rtl8127_switch_to_hw_interrupt(tp); + } +#else + if (status & tp->intr_mask || tp->keep_intr_cnt-- > 0) { + u32 budget = ~(u32)0; + int i; + + if (status & tp->intr_mask) + tp->keep_intr_cnt = RTK_KEEP_INTERRUPT_COUNT; + + for (i = 0; i < tp->num_tx_rings; i++) + rtl8127_tx_interrupt(&tp->tx_ring[i], ~(u32)0); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) + rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[0], &budget); +#else + rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[0], budget); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) + +#ifdef ENABLE_DASH_SUPPORT + if (tp->DASH) { + struct net_device *dev = tp->dev; + + HandleDashInterrupt(dev); + } +#endif + + rtl8127_switch_to_timer_interrupt(tp); + } else { + tp->keep_intr_cnt = RTK_KEEP_INTERRUPT_COUNT; + rtl8127_switch_to_hw_interrupt(tp); + } +#endif + } while (false); + + return IRQ_RETVAL(handled); +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,19) +static irqreturn_t rtl8127_interrupt_msix(int irq, void *dev_instance, struct pt_regs *regs) +#else +static irqreturn_t rtl8127_interrupt_msix(int irq, void *dev_instance) +#endif +{ + struct r8127_napi *r8127napi = dev_instance; + struct rtl8127_private *tp = r8127napi->priv; + struct net_device *dev = tp->dev; + int message_id = r8127napi->index; +#ifndef CONFIG_R8127_NAPI + u32 budget = ~(u32)0; +#endif + + do { +#if defined(RTL_USE_NEW_INTR_API) + if (!tp->irq_tbl[message_id].requested) + break; +#endif + //link change + if (message_id == rtl8127_get_linkchg_message_id(tp)) { + rtl8127_disable_hw_interrupt_v2(tp, message_id); + rtl8127_clear_hw_isr_v2(tp, message_id); + rtl8127_schedule_linkchg_work(tp); + break; + } + +#ifdef CONFIG_R8127_NAPI + if (likely(RTL_NETIF_RX_SCHEDULE_PREP(dev, &r8127napi->napi))) { + rtl8127_disable_hw_interrupt_v2(tp, message_id); + __RTL_NETIF_RX_SCHEDULE(dev, &r8127napi->napi); + } else if (netif_msg_intr(tp)) + printk(KERN_INFO "%s: interrupt message id %d in poll_msix\n", + dev->name, message_id); + rtl8127_clear_hw_isr_v2(tp, message_id); +#else + rtl8127_disable_hw_interrupt_v2(tp, message_id); + + rtl8127_clear_hw_isr_v2(tp, message_id); + + rtl8127_tx_interrupt_with_vector(tp, message_id, ~(u32)0); + + if (message_id < tp->num_rx_rings) { +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) + rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[message_id], &budget); +#else + rtl8127_rx_interrupt(dev, tp, &tp->rx_ring[message_id], budget); +#endif //LINUX_VERSION_CODE < KERNEL_VERSION(2,6,24) + } + + rtl8127_enable_hw_interrupt_v2(tp, message_id); +#endif + + } while (false); + + return IRQ_HANDLED; +} + +static void rtl8127_down(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + //rtl8127_delete_esd_timer(dev, &tp->esd_timer); + + //rtl8127_delete_link_timer(dev, &tp->link_timer); + + netif_carrier_off(dev); + + netif_tx_disable(dev); + + _rtl8127_wait_for_quiescence(dev); + + rtl8127_hw_reset(dev); + + rtl8127_tx_clear(tp); + + rtl8127_rx_clear(tp); +} + +static int rtl8127_resource_freed(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < tp->num_tx_rings; i++) + if (tp->tx_ring[i].TxDescArray) + return 0; + + for (i = 0; i < tp->num_rx_rings; i++) + if (tp->rx_ring[i].RxDescArray) + return 0; + + return 1; +} + +int rtl8127_close(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (!rtl8127_resource_freed(tp)) { + set_bit(R8127_FLAG_DOWN, tp->task_flags); + + rtl8127_down(dev); + + pci_clear_master(tp->pci_dev); + +#ifdef ENABLE_PTP_SUPPORT + rtl8127_ptp_stop(tp); +#endif + rtl8127_hw_d3_para(dev); + + rtl8127_powerdown_pll(dev, 0); + + rtl8127_free_irq(tp); + + rtl8127_free_alloc_resources(tp); + } else { + rtl8127_hw_d3_para(dev); + + rtl8127_powerdown_pll(dev, 0); + } + + return 0; +} + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,11) +static void rtl8127_shutdown(struct pci_dev *pdev) +{ + struct net_device *dev = pci_get_drvdata(pdev); + struct rtl8127_private *tp = netdev_priv(dev); + + rtnl_lock(); + + if (HW_DASH_SUPPORT_DASH(tp)) + rtl8127_driver_stop(tp); + + rtl8127_disable_pci_offset_180(tp); + + if (s5_keep_curr_mac == 0 && tp->random_mac == 0) + rtl8127_rar_set(tp, tp->org_mac_addr); + + if (s5wol == 0) + tp->wol_enabled = WOL_DISABLED; + + rtl8127_close(dev); + rtl8127_disable_msi(pdev, tp); + + rtnl_unlock(); + + if (system_state == SYSTEM_POWER_OFF) { + pci_clear_master(tp->pci_dev); + pci_wake_from_d3(pdev, tp->wol_enabled); + pci_set_power_state(pdev, PCI_D3hot); + } +} +#endif + +#ifdef CONFIG_PM + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,11) +static int +rtl8127_suspend(struct pci_dev *pdev, u32 state) +#elif LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29) +static int +rtl8127_suspend(struct device *device) +#else +static int +rtl8127_suspend(struct pci_dev *pdev, pm_message_t state) +#endif +{ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29) + struct pci_dev *pdev = to_pci_dev(device); + struct net_device *dev = pci_get_drvdata(pdev); +#else + struct net_device *dev = pci_get_drvdata(pdev); +#endif + struct rtl8127_private *tp = netdev_priv(dev); +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10) + u32 pci_pm_state = pci_choose_state(pdev, state); +#endif + rtnl_lock(); + + if (!netif_running(dev)) + goto out; + + set_bit(R8127_FLAG_DOWN, tp->task_flags); + + netif_carrier_off(dev); + + netif_tx_disable(dev); + + netif_device_detach(dev); + +#ifdef ENABLE_PTP_SUPPORT + rtl8127_ptp_suspend(tp); +#endif + rtl8127_hw_reset(dev); + + pci_clear_master(pdev); + + rtl8127_hw_d3_para(dev); + + rtl8127_powerdown_pll(dev, 1); +out: + if (HW_DASH_SUPPORT_DASH(tp)) + rtl8127_driver_stop(tp); + + rtnl_unlock(); + + pci_disable_device(pdev); + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10) + pci_save_state(pdev, &pci_pm_state); +#else + pci_save_state(pdev); +#endif +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) + pci_enable_wake(pdev, pci_choose_state(pdev, state), tp->wol_enabled); +#endif + + pci_prepare_to_sleep(pdev); + + return 0; +} + +static int +rtl8127_hw_d3_not_power_off(struct net_device *dev) +{ + return rtl8127_check_hw_phy_mcu_code_ver(dev); +} + +static int rtl8127_wait_phy_nway_complete_sleep(struct rtl8127_private *tp) +{ + int i, val; + + for (i = 0; i < 30; i++) { + val = rtl8127_mdio_read(tp, MII_BMSR) & BMSR_ANEGCOMPLETE; + if (val) + return 0; + + fsleep(100000); + } + + return -1; +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) +static int +rtl8127_resume(struct pci_dev *pdev) +#else +static int +rtl8127_resume(struct device *device) +#endif +{ +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29) + struct pci_dev *pdev = to_pci_dev(device); + struct net_device *dev = pci_get_drvdata(pdev); +#else + struct net_device *dev = pci_get_drvdata(pdev); +#endif + struct rtl8127_private *tp = netdev_priv(dev); +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10) + u32 pci_pm_state = PCI_D0; +#endif + u32 err; + + rtnl_lock(); + + err = pci_enable_device(pdev); + if (err) { + dev_err(&pdev->dev, "Cannot enable PCI device from suspend\n"); + goto out_unlock; + } +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,10) + pci_restore_state(pdev, &pci_pm_state); +#else + pci_restore_state(pdev); +#endif + pci_enable_wake(pdev, PCI_D0, 0); + + /* restore last modified mac address */ + rtl8127_rar_set(tp, dev->dev_addr); + + tp->resume_not_chg_speed = 0; + if (tp->check_keep_link_speed && + //tp->link_ok(dev) && + rtl8127_hw_d3_not_power_off(dev) && + rtl8127_wait_phy_nway_complete_sleep(tp) == 0) + tp->resume_not_chg_speed = 1; + + if (!netif_running(dev)) + goto out_unlock; + + pci_set_master(pdev); + + rtl8127_exit_oob(dev); + + rtl8127_up(dev); + + clear_bit(R8127_FLAG_DOWN, tp->task_flags); + + rtl8127_schedule_reset_work(tp); + + rtl8127_schedule_esd_work(tp); + + //mod_timer(&tp->esd_timer, jiffies + RTL8127_ESD_TIMEOUT); + //mod_timer(&tp->link_timer, jiffies + RTL8127_LINK_TIMEOUT); +out_unlock: + netif_device_attach(dev); + + rtnl_unlock(); + + return err; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,29) + +static struct dev_pm_ops rtl8127_pm_ops = { + .suspend = rtl8127_suspend, + .resume = rtl8127_resume, + .freeze = rtl8127_suspend, + .thaw = rtl8127_resume, + .poweroff = rtl8127_suspend, + .restore = rtl8127_resume, +}; + +#define RTL8127_PM_OPS (&rtl8127_pm_ops) + +#endif + +#else /* !CONFIG_PM */ + +#define RTL8127_PM_OPS NULL + +#endif /* CONFIG_PM */ + +static struct pci_driver rtl8127_pci_driver = { + .name = MODULENAME, + .id_table = rtl8127_pci_tbl, + .probe = rtl8127_init_one, + .remove = __devexit_p(rtl8127_remove_one), +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,11) + .shutdown = rtl8127_shutdown, +#endif +#ifdef CONFIG_PM +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,29) + .suspend = rtl8127_suspend, + .resume = rtl8127_resume, +#else + .driver.pm = RTL8127_PM_OPS, +#endif +#endif +}; + +static int __init +rtl8127_init_module(void) +{ + int ret = 0; +#ifdef ENABLE_R8127_PROCFS + rtl8127_proc_module_init(); +#endif + +#if LINUX_VERSION_CODE > KERNEL_VERSION(2,6,0) + + ret = pci_register_driver(&rtl8127_pci_driver); +#else + ret = pci_module_init(&rtl8127_pci_driver); +#endif + + return ret; +} + +static void __exit +rtl8127_cleanup_module(void) +{ + pci_unregister_driver(&rtl8127_pci_driver); + +#ifdef ENABLE_R8127_PROCFS + if (rtl8127_proc) { +#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + remove_proc_subtree(MODULENAME, init_net.proc_net); +#else +#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,32) + remove_proc_entry(MODULENAME, init_net.proc_net); +#else + remove_proc_entry(MODULENAME, proc_net); +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,32) +#endif //LINUX_VERSION_CODE >= KERNEL_VERSION(3,10,0) + rtl8127_proc = NULL; + } +#endif +} + +module_init(rtl8127_init_module); +module_exit(rtl8127_cleanup_module); diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c b/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c new file mode 100755 index 0000000000000..f3fd421625c0c --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c @@ -0,0 +1,944 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "r8127.h" +#include "r8127_ptp.h" + +static void rtl8127_wait_clkadj_ready(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) + if (!(rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CLK_CFG_8126) & CLKADJ_MODE_SET)) + break; +} + +static void rtl8127_set_clkadj_mode(struct rtl8127_private *tp, u16 cmd) +{ + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + PTP_CLK_CFG_8126, + BIT_3 | BIT_2 | BIT_1, + CLKADJ_MODE_SET | cmd); + + rtl8127_wait_clkadj_ready(tp); +} + +static int _rtl8127_phc_gettime(struct rtl8127_private *tp, struct timespec64 *ts64) +{ + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + + //Direct Read + rtl8127_set_clkadj_mode(tp, DIRECT_READ); + + /* nanoseconds */ + //Ns[29:16] E414[13:0] + ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_NS_HI_8126) & 0x3fff; + ts64->tv_nsec <<= 16; + //Ns[15:0] E412[15:0] + ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_NS_LO_8126); + + + /* seconds */ + //S[47:32] E41A[15:0] + ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_HI_8126); + ts64->tv_sec <<= 16; + //S[31:16] E418[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_MI_8126); + ts64->tv_sec <<= 16; + //S[15:0] E416[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_LO_8126); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return 0; +} + +static int _rtl8127_phc_settime(struct rtl8127_private *tp, const struct timespec64 *ts64) +{ + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + + /* nanoseconds */ + //Ns[15:0] E412[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, ts64->tv_nsec); + //Ns[29:16] E414[13:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (ts64->tv_nsec & 0x3fff0000) >> 16); + + /* seconds */ + //S[15:0] E416[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_LO_8126, ts64->tv_sec); + //S[31:16] E418[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_MI_8126, (ts64->tv_sec >> 16)); + //S[47:32] E41A[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_HI_8126, (ts64->tv_sec >> 32)); + + //Direct Write + rtl8127_set_clkadj_mode(tp, DIRECT_WRITE); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return 0; +} + +static int _rtl8127_phc_adjtime(struct rtl8127_private *tp, s64 delta) +{ + unsigned long flags; + struct timespec64 d; + bool negative; + u64 tohw; + u32 nsec; + u64 sec; + + if (delta < 0) { + negative = true; + tohw = -delta; + } else { + negative = false; + tohw = delta; + } + + d = ns_to_timespec64(tohw); + + nsec = d.tv_nsec; + sec = d.tv_sec; + + nsec &= 0x3fffffff; + sec &= 0x0000ffffffffffff; + + spin_lock_irqsave(&tp->phy_lock, flags); + + /* nanoseconds */ + //Ns[15:0] E412[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, nsec); + //Ns[29:16] E414[13:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (nsec >> 16)); + + /* seconds */ + //S[15:0] E416[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_LO_8126, sec); + //S[31:16] E418[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_MI_8126, (sec >> 16)); + //S[47:32] E41A[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_HI_8126, (sec >> 32)); + + if (negative) + rtl8127_set_clkadj_mode(tp, DECREMENT_STEP); + else + rtl8127_set_clkadj_mode(tp, INCREMENT_STEP); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return 0; +} + +static int rtl8127_phc_adjtime(struct ptp_clock_info *ptp, s64 delta) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + int ret; + + //netif_info(tp, drv, tp->dev, "phc adjust time\n"); + + ret = _rtl8127_phc_adjtime(tp, delta); + + return ret; +} + +/* + * delta = delta * 10^6 ppm = delta * 10^9 ppb (in this equation ppm and ppb are not variable) + * + * in adjfreq ppb is a variable + * ppb = delta * 10^9 + * delta = ppb / 10^9 + * rate_value = |delta| * 2^32 = |ppb| / 10^9 * 2^32 = (|ppb| << 32) / 10^9 + */ +static int _rtl8127_phc_adjfreq(struct ptp_clock_info *ptp, s32 ppb) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + unsigned long flags; + u32 rate_value; + + if (ppb < 0) { + rate_value = ((u64)-ppb << 32) / 1000000000; + rate_value = ~rate_value + 1; + } else + rate_value = ((u64)ppb << 32) / 1000000000; + + spin_lock_irqsave(&tp->phy_lock, flags); + + /* nanoseconds */ + //Ns[15:0] E412[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, rate_value); + //Ns[22:16] E414[13:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (rate_value & 0x003f0000) >> 16); + + rtl8127_set_clkadj_mode(tp, RATE_WRITE); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return 0; +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) +static int rtl8127_ptp_adjfine(struct ptp_clock_info *ptp, long scaled_ppm) +{ + s32 ppb = scaled_ppm_to_ppb(scaled_ppm); + + if (ppb > ptp->max_adj || ppb < -ptp->max_adj) + return -EINVAL; + + _rtl8127_phc_adjfreq(ptp, ppb); + + return 0; +} + +#else +static int rtl8127_phc_adjfreq(struct ptp_clock_info *ptp, s32 delta) +{ + //struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + + //netif_info(tp, drv, tp->dev, "phc adjust freq\n"); + + if (delta > ptp->max_adj || delta < -ptp->max_adj) + return -EINVAL; + + _rtl8127_phc_adjfreq(ptp, delta); + + return 0; +} +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) */ + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) +static int rtl8127_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts64, + struct ptp_system_timestamp *sts) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + int ret; + + //netif_info(tp, drv, tp->dev, "phc get ts\n"); + + ptp_read_system_prets(sts); + ret = _rtl8127_phc_gettime(tp, ts64); + ptp_read_system_postts(sts); + + return ret; +} +#else +static int rtl8127_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts64) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + int ret; + + //netif_info(tp, drv, tp->dev, "phc get ts\n"); + + ret = _rtl8127_phc_gettime(tp, ts64); + + return ret; +} +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) */ + +static int rtl8127_phc_settime(struct ptp_clock_info *ptp, + const struct timespec64 *ts64) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + int ret; + + //netif_info(tp, drv, tp->dev, "phc set ts\n"); + + ret = _rtl8127_phc_settime(tp, ts64); + + return ret; +} + +static void _rtl8127_phc_enable(struct ptp_clock_info *ptp, + struct ptp_clock_request *rq, int on) +{ + struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); + unsigned long flags; + u16 phy_ocp_data; + + if (on) { + tp->pps_enable = 1; + rtl8127_clear_mac_ocp_bit(tp, 0xDC00, BIT_6); + rtl8127_clear_mac_ocp_bit(tp, 0xDC20, BIT_1); + + spin_lock_irqsave(&tp->phy_lock, flags); + + /* Set periodic pulse 1pps */ + /* E432[8:0] = 0x017d */ + phy_ocp_data = rtl8127_mdio_direct_read_phy_ocp(tp, 0xE432); + phy_ocp_data &= 0xFE00; + phy_ocp_data |= 0x017d; + rtl8127_mdio_direct_write_phy_ocp(tp, 0xE432, phy_ocp_data); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xE434, 0x7840); + + /* E436[8:0] = 0xbe */ + phy_ocp_data = rtl8127_mdio_direct_read_phy_ocp(tp, 0xE436); + phy_ocp_data &= 0xFE00; + phy_ocp_data |= 0xbe; + rtl8127_mdio_direct_write_phy_ocp(tp, 0xE436, phy_ocp_data); + + rtl8127_mdio_direct_write_phy_ocp(tp, 0xE438, 0xbc20); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + /* start hrtimer */ + hrtimer_start(&tp->pps_timer, 1000000000, HRTIMER_MODE_REL); + } else + tp->pps_enable = 0; +} + +static int rtl8127_phc_enable(struct ptp_clock_info *ptp, + struct ptp_clock_request *rq, int on) +{ + switch (rq->type) { + case PTP_CLK_REQ_PPS: + _rtl8127_phc_enable(ptp, rq, on); + return 0; + default: + return -EOPNOTSUPP; + } +} + +static void rtl8127_ptp_enable_config(struct rtl8127_private *tp) +{ + if (tp->syncE_en) + rtl8127_set_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); + else + rtl8127_clear_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); + + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CTL, PTP_CTL_TYPE_3 | BIT_12); + + rtl8127_set_eth_phy_ocp_bit(tp, 0xA640, BIT_15); +} + +int rtl8127_get_ts_info(struct net_device *netdev, + struct ethtool_ts_info *info) +{ + struct rtl8127_private *tp = netdev_priv(netdev); + + /* we always support timestamping disabled */ + info->rx_filters = BIT(HWTSTAMP_FILTER_NONE); + + if (tp->HwSuppPtpVer == 0) + return ethtool_op_get_ts_info(netdev, info); + + info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE | + SOF_TIMESTAMPING_RX_SOFTWARE | + SOF_TIMESTAMPING_SOFTWARE | + SOF_TIMESTAMPING_TX_HARDWARE | + SOF_TIMESTAMPING_RX_HARDWARE | + SOF_TIMESTAMPING_RAW_HARDWARE; + + if (tp->ptp_clock) + info->phc_index = ptp_clock_index(tp->ptp_clock); + else + info->phc_index = -1; + + info->tx_types = BIT(HWTSTAMP_TX_OFF) | BIT(HWTSTAMP_TX_ON); + + info->rx_filters = BIT(HWTSTAMP_FILTER_NONE) | + BIT(HWTSTAMP_FILTER_PTP_V2_EVENT) | + BIT(HWTSTAMP_FILTER_PTP_V2_L4_EVENT) | + BIT(HWTSTAMP_FILTER_PTP_V2_SYNC) | + BIT(HWTSTAMP_FILTER_PTP_V2_L4_SYNC) | + BIT(HWTSTAMP_FILTER_PTP_V2_DELAY_REQ) | + BIT(HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ); + + return 0; +} + +static const struct ptp_clock_info rtl_ptp_clock_info = { + .owner = THIS_MODULE, + .n_alarm = 0, + .n_ext_ts = 0, + .n_per_out = 0, + .n_pins = 0, + .pps = 1, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) + .adjfine = rtl8127_ptp_adjfine, +#else + .adjfreq = rtl8127_phc_adjfreq, +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) */ + .adjtime = rtl8127_phc_adjtime, +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) + .gettimex64 = rtl8127_phc_gettime, +#else + .gettime64 = rtl8127_phc_gettime, +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) */ + + .settime64 = rtl8127_phc_settime, + .enable = rtl8127_phc_enable, +}; + +static u16 rtl8127_ptp_get_tx_msgtype(struct rtl8127_private *tp) +{ + u16 tx_ts_ready = 0; + int i; + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + tx_ts_ready = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & 0xF000; + if (tx_ts_ready) + break; + } + + switch (tx_ts_ready) { + case TX_TS_PDLYRSP_RDY: + return PTP_MSGTYPE_PDELAY_RESP; + case TX_TS_PDLYREQ_RDY: + return PTP_MSGTYPE_PDELAY_REQ; + case TX_TS_DLYREQ_RDY: + return PTP_MSGTYPE_DELAY_REQ; + case TX_TS_SYNC_RDY: + default: + return PTP_MSGTYPE_SYNC; + } +} + +/* +static u16 rtl8127_ptp_get_rx_msgtype(struct rtl8127_private *tp) +{ + u16 rx_ts_ready = 0; + int i; + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { + rx_ts_ready = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & 0x0F00; + if (rx_ts_ready) + break; + } + + switch (rx_ts_ready) { + case RX_TS_PDLYRSP_RDY: + return PTP_MSGTYPE_PDELAY_RESP; + case RX_TS_PDLYREQ_RDY: + return PTP_MSGTYPE_PDELAY_REQ; + case RX_TS_DLYREQ_RDY: + return PTP_MSGTYPE_DELAY_REQ; + case RX_TS_SYNC_RDY: + default: + return PTP_MSGTYPE_SYNC; + } +} +*/ + +static void rtl8127_wait_trx_ts_ready(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) + if (!(rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & TRX_TS_RD)) + break; +} + +static void rtl8127_set_trx_ts_cmd(struct rtl8127_private *tp, u16 cmd) +{ + rtl8127_clear_and_set_eth_phy_ocp_bit(tp, + PTP_TRX_TS_STA, + TRXTS_SEL | BIT_3 | BIT_2, + TRX_TS_RD | cmd); + + rtl8127_wait_trx_ts_ready(tp); +} + +static void rtl8127_ptp_egresstime(struct rtl8127_private *tp, struct timespec64 *ts64) +{ + u16 msgtype; + + msgtype = rtl8127_ptp_get_tx_msgtype(tp); + + msgtype <<= 2; + + rtl8127_set_trx_ts_cmd(tp, (msgtype | BIT_4)); + + /* nanoseconds */ + //Ns[29:16] E448[13:0] + ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_HI) & 0x3fff; + ts64->tv_nsec <<= 16; + //Ns[15:0] E446[15:0] + ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_LO); + + /* seconds */ + //S[47:32] E44E[15:0] + ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_HI); + ts64->tv_sec <<= 16; + //S[31:16] E44C[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_MI); + ts64->tv_sec <<= 16; + //S[15:0] E44A[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_LO); +} + +static void rtl8127_ptp_ingresstime(struct rtl8127_private *tp, struct timespec64 *ts64, u8 type) +{ + u16 msgtype; + + switch (type) { + case PTP_MSGTYPE_PDELAY_RESP: + case PTP_MSGTYPE_PDELAY_REQ: + case PTP_MSGTYPE_DELAY_REQ: + case PTP_MSGTYPE_SYNC: + msgtype = type << 2; + break; + default: + return; + } + + rtl8127_set_trx_ts_cmd(tp, (TRXTS_SEL | msgtype | BIT_4)); + + /* nanoseconds */ + //Ns[29:16] E448[13:0] + ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_HI) & 0x3fff; + ts64->tv_nsec <<= 16; + //Ns[15:0] E446[15:0] + ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_LO); + + /* seconds */ + //S[47:32] E44E[15:0] + ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_HI); + ts64->tv_sec <<= 16; + //S[31:16] E44C[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_MI); + ts64->tv_sec <<= 16; + //S[15:0] E44A[15:0] + ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_LO); +} + +static void rtl8127_ptp_tx_hwtstamp(struct rtl8127_private *tp) +{ + struct sk_buff *skb = tp->ptp_tx_skb; + struct skb_shared_hwtstamps shhwtstamps = { 0 }; + struct timespec64 ts64; + + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, TX_TX_INTR); + + rtl8127_ptp_egresstime(tp, &ts64); + + /* Upper 32 bits contain s, lower 32 bits contain ns. */ + shhwtstamps.hwtstamp = ktime_set(ts64.tv_sec, + ts64.tv_nsec); + + /* Clear the lock early before calling skb_tstamp_tx so that + * applications are not woken up before the lock bit is clear. We use + * a copy of the skb pointer to ensure other threads can't change it + * while we're notifying the stack. + */ + tp->ptp_tx_skb = NULL; + clear_bit_unlock(__RTL8127_PTP_TX_IN_PROGRESS, &tp->state); + + /* Notify the stack and free the skb after we've unlocked */ + skb_tstamp_tx(skb, &shhwtstamps); + dev_kfree_skb_any(skb); +} + +#define RTL8127_PTP_TX_TIMEOUT (HZ * 15) +static void rtl8127_ptp_tx_work(struct work_struct *work) +{ + struct rtl8127_private *tp = container_of(work, struct rtl8127_private, + ptp_tx_work); + unsigned long flags; + + if (!tp->ptp_tx_skb) + return; + + if (time_is_before_jiffies(tp->ptp_tx_start + + RTL8127_PTP_TX_TIMEOUT)) { + dev_kfree_skb_any(tp->ptp_tx_skb); + tp->ptp_tx_skb = NULL; + clear_bit_unlock(__RTL8127_PTP_TX_IN_PROGRESS, &tp->state); + tp->tx_hwtstamp_timeouts++; + /* Clear the tx valid bit in TSYNCTXCTL register to enable + * interrupt + */ + spin_lock_irqsave(&tp->phy_lock, flags); + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, TX_TX_INTR); + spin_unlock_irqrestore(&tp->phy_lock, flags); + return; + } + + spin_lock_irqsave(&tp->phy_lock, flags); + if (rtl8127_mdio_direct_read_phy_ocp(tp, PTP_INSR) & TX_TX_INTR) { + rtl8127_ptp_tx_hwtstamp(tp); + spin_unlock_irqrestore(&tp->phy_lock, flags); + } else { + spin_unlock_irqrestore(&tp->phy_lock, flags); + /* reschedule to check later */ + schedule_work(&tp->ptp_tx_work); + } +} + +static int rtl8127_hwtstamp_enable(struct rtl8127_private *tp, bool enable) +{ + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + + if (enable) { + //trx timestamp interrupt enable + rtl8127_set_eth_phy_ocp_bit(tp, PTP_INER, BIT_2 | BIT_3); + + //set isr clear mode + rtl8127_set_eth_phy_ocp_bit(tp, PTP_GEN_CFG, BIT_0); + + //clear ptp isr + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, 0xFFFF); + + //enable ptp + rtl8127_ptp_enable_config(tp); + + //rtl8127_set_local_time(tp); + } else { + /* trx timestamp interrupt disable */ + rtl8127_clear_eth_phy_ocp_bit(tp, PTP_INER, BIT_2 | BIT_3); + + /* disable ptp */ + rtl8127_clear_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); + rtl8127_clear_eth_phy_ocp_bit(tp, PTP_CTL, BIT_0); + rtl8127_set_eth_phy_ocp_bit(tp, 0xA640, BIT_15); + } + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + return 0; +} + +void rtl8127_set_local_time(struct rtl8127_private *tp) +{ + struct timespec64 ts64; + //set system time + ktime_get_real_ts64(&ts64); + _rtl8127_phc_settime(tp, &ts64); +} + +static long rtl8127_ptp_create_clock(struct rtl8127_private *tp) +{ + struct net_device *netdev = tp->dev; + long err; + + if (!IS_ERR_OR_NULL(tp->ptp_clock)) + return 0; + + if (tp->HwSuppPtpVer == 0) { + tp->ptp_clock = NULL; + return -EOPNOTSUPP; + } + + tp->ptp_clock_info = rtl_ptp_clock_info; + tp->ptp_clock_info.max_adj = 488281;//0x1FFFFF * 10^9 / 2^32 + + snprintf(tp->ptp_clock_info.name, sizeof(tp->ptp_clock_info.name), + "%pm", tp->dev->dev_addr); + tp->ptp_clock = ptp_clock_register(&tp->ptp_clock_info, &tp->pci_dev->dev); + if (IS_ERR(tp->ptp_clock)) { + err = PTR_ERR(tp->ptp_clock); + tp->ptp_clock = NULL; + netif_err(tp, drv, tp->dev, "ptp_clock_register failed\n"); + return err; + } else + netif_info(tp, drv, tp->dev, "registered PHC device on %s\n", netdev->name); + + return 0; +} + +static enum hrtimer_restart +rtl8127_hrtimer_for_pps(struct hrtimer *timer) { + struct rtl8127_private *tp = container_of(timer, struct rtl8127_private, pps_timer); + u16 tai_cfg = BIT_8 | BIT_3 | BIT_1 | BIT_0; + s64 pps_sec; + + if (tp->pps_enable) + { + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + + //Direct Read + rtl8127_set_clkadj_mode(tp, DIRECT_READ); + + pps_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_HI_8126); + pps_sec <<= 16; + pps_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_MI_8126); + pps_sec <<= 16; + pps_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_LO_8126); + pps_sec++; + + //E42A[15:0] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_TS_S_LO, pps_sec & 0xffff); + //E42C[31:16] + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_TS_S_HI, (pps_sec & 0xffff0000) >> 16); + //Periodic Tai start + rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_CFG, tai_cfg); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + hrtimer_forward_now(&tp->pps_timer, 1000000000); //rekick + return HRTIMER_RESTART; + } else + return HRTIMER_NORESTART; +} + +void rtl8127_ptp_reset(struct rtl8127_private *tp) +{ + if (!tp->ptp_clock) + return; + + netif_info(tp, drv, tp->dev, "reset PHC clock\n"); + + rtl8127_hwtstamp_enable(tp, false); +} + +void rtl8127_ptp_init(struct rtl8127_private *tp) +{ + /* obtain a PTP device, or re-use an existing device */ + if (rtl8127_ptp_create_clock(tp)) + return; + + /* we have a clock so we can initialize work now */ + INIT_WORK(&tp->ptp_tx_work, rtl8127_ptp_tx_work); + + /* init a hrtimer for pps */ + tp->pps_enable = 0; + hrtimer_init(&tp->pps_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); + tp->pps_timer.function = rtl8127_hrtimer_for_pps; + + /* reset the PTP related hardware bits */ + rtl8127_ptp_reset(tp); + + return; +} + +void rtl8127_ptp_suspend(struct rtl8127_private *tp) +{ + if (!tp->ptp_clock) + return; + + netif_info(tp, drv, tp->dev, "suspend PHC clock\n"); + + rtl8127_hwtstamp_enable(tp, false); + + /* ensure that we cancel any pending PTP Tx work item in progress */ + cancel_work_sync(&tp->ptp_tx_work); + + hrtimer_cancel(&tp->pps_timer); +} + +void rtl8127_ptp_stop(struct rtl8127_private *tp) +{ + struct net_device *netdev = tp->dev; + + netif_info(tp, drv, tp->dev, "stop PHC clock\n"); + + /* first, suspend PTP activity */ + rtl8127_ptp_suspend(tp); + + /* disable the PTP clock device */ + if (tp->ptp_clock) { + ptp_clock_unregister(tp->ptp_clock); + tp->ptp_clock = NULL; + netif_info(tp, drv, tp->dev, "removed PHC on %s\n", + netdev->name); + } +} + +static int rtl8127_set_tstamp(struct net_device *netdev, struct ifreq *ifr) +{ + struct rtl8127_private *tp = netdev_priv(netdev); + struct hwtstamp_config config; + bool hwtstamp = 0; + + //netif_info(tp, drv, tp->dev, "ptp set ts\n"); + + if (copy_from_user(&config, ifr->ifr_data, sizeof(config))) + return -EFAULT; + + if (config.flags) + return -EINVAL; + + switch (config.tx_type) { + case HWTSTAMP_TX_ON: + hwtstamp = 1; + break; + case HWTSTAMP_TX_OFF: + break; + case HWTSTAMP_TX_ONESTEP_SYNC: + default: + return -ERANGE; + } + + switch (config.rx_filter) { + case HWTSTAMP_FILTER_PTP_V2_EVENT: + case HWTSTAMP_FILTER_PTP_V2_L2_EVENT: + case HWTSTAMP_FILTER_PTP_V2_L4_EVENT: + case HWTSTAMP_FILTER_PTP_V2_SYNC: + case HWTSTAMP_FILTER_PTP_V2_L2_SYNC: + case HWTSTAMP_FILTER_PTP_V2_L4_SYNC: + case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ: + case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ: + case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: + config.rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT; + hwtstamp = 1; + tp->flags |= RTL_FLAG_RX_HWTSTAMP_ENABLED; + break; + case HWTSTAMP_FILTER_NONE: + tp->flags &= ~RTL_FLAG_RX_HWTSTAMP_ENABLED; + break; + default: + tp->flags &= ~RTL_FLAG_RX_HWTSTAMP_ENABLED; + return -ERANGE; + } + + if (tp->hwtstamp_config.tx_type != config.tx_type || + tp->hwtstamp_config.rx_filter != config.rx_filter) { + tp->hwtstamp_config = config; + + rtl8127_hwtstamp_enable(tp, hwtstamp); + } + + return copy_to_user(ifr->ifr_data, &config, + sizeof(config)) ? -EFAULT : 0; +} + +static int rtl8127_get_tstamp(struct net_device *netdev, struct ifreq *ifr) +{ + struct rtl8127_private *tp = netdev_priv(netdev); + + //netif_info(tp, drv, tp->dev, "ptp get ts\n"); + + return copy_to_user(ifr->ifr_data, &tp->hwtstamp_config, + sizeof(tp->hwtstamp_config)) ? -EFAULT : 0; +} + +int rtl8127_ptp_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd) +{ + int ret; + + //netif_info(tp, drv, tp->dev, "ptp ioctl\n"); + + switch (cmd) { +#ifdef ENABLE_PTP_SUPPORT + case SIOCSHWTSTAMP: + ret = rtl8127_set_tstamp(netdev, ifr); + break; + case SIOCGHWTSTAMP: + ret = rtl8127_get_tstamp(netdev, ifr); + break; +#endif + default: + ret = -EOPNOTSUPP; + break; + } + + return ret; +} + +static void rtl8127_rx_ptp_pktstamp(struct rtl8127_private *tp, struct sk_buff *skb, u8 type) +{ + struct timespec64 ts64; + unsigned long flags; + + spin_lock_irqsave(&tp->phy_lock, flags); + + rtl8127_ptp_ingresstime(tp, &ts64, type); + + spin_unlock_irqrestore(&tp->phy_lock, flags); + + skb_hwtstamps(skb)->hwtstamp = ktime_set(ts64.tv_sec, ts64.tv_nsec); + + return; +} + +void rtl8127_rx_ptp_timestamp(struct rtl8127_private *tp, struct sk_buff *skb) +{ + unsigned int ptp_class; + struct ptp_header *hdr; + u8 msgtype; + + ptp_class = ptp_classify_raw(skb); + if (ptp_class == PTP_CLASS_NONE) + return; + + skb_reset_mac_header(skb); + hdr = ptp_parse_header(skb, ptp_class); + if (unlikely(!hdr)) + return; + + msgtype = ptp_get_msgtype(hdr, ptp_class); + rtl8127_rx_ptp_pktstamp(tp, skb, msgtype); + + return; +} + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) +struct ptp_header *ptp_parse_header(struct sk_buff *skb, unsigned int type) +{ + u8 *ptr = skb_mac_header(skb); + + if (type & PTP_CLASS_VLAN) + //ptr += VLAN_HLEN; + ptr += 4; + + switch (type & PTP_CLASS_PMASK) { + case PTP_CLASS_IPV4: + ptr += IPV4_HLEN(ptr) + UDP_HLEN; + break; + case PTP_CLASS_IPV6: + ptr += IP6_HLEN + UDP_HLEN; + break; + case PTP_CLASS_L2: + break; + default: + return NULL; + } + + ptr += ETH_HLEN; + + /* Ensure that the entire header is present in this packet. */ + if (ptr + sizeof(struct ptp_header) > skb->data + skb->len) + return NULL; + + return (struct ptp_header *)ptr; +} +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h b/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h new file mode 100755 index 0000000000000..e96afafd70a85 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h @@ -0,0 +1,202 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_R8127_PTP_H +#define _LINUX_R8127_PTP_H + +#include +#include +#include +#include +#include + +#ifndef PTP_CLASS_NONE +#define PTP_CLASS_NONE 0x00 +#endif + +#ifndef PTP_MSGTYPE_SYNC +#define PTP_MSGTYPE_SYNC 0x0 +#endif +#ifndef PTP_MSGTYPE_DELAY_REQ +#define PTP_MSGTYPE_DELAY_REQ 0x1 +#endif +#ifndef PTP_MSGTYPE_PDELAY_REQ +#define PTP_MSGTYPE_PDELAY_REQ 0x2 +#endif +#ifndef PTP_MSGTYPE_PDELAY_RESP +#define PTP_MSGTYPE_PDELAY_RESP 0x3 +#endif + +struct rtl8127_ptp_info { + s64 time_sec; + u32 time_ns; + u16 ts_info; +}; + +#ifndef _STRUCT_TIMESPEC +#define _STRUCT_TIMESPEC +struct timespec { + __kernel_old_time_t tv_sec; /* seconds */ + long tv_nsec; /* nanoseconds */ +}; +#endif + +enum PTP_CMD_TYPE { + PTP_CMD_SET_LOCAL_TIME = 0, + PTP_CMD_DRIFT_LOCAL_TIME, + PTP_CMD_LATCHED_LOCAL_TIME, +}; + +enum PTP_CLKADJ_MOD_TYPE { + NO_FUNCTION = 0, + CLKADJ_MODE_SET = 1, + RESERVED = 2, + DIRECT_READ = 4, + DIRECT_WRITE = 6, + INCREMENT_STEP = 8, + DECREMENT_STEP = 10, + RATE_READ = 12, + RATE_WRITE = 14, +}; + +enum PTP_INSR_TYPE { + EVENT_CAP_INTR = (1 << 0), + TRIG_GEN_INTR = (1 << 1), + RX_TS_INTR = (1 << 2), + TX_TX_INTR = (1 << 3), +}; + +enum PTP_TRX_TS_STA_REG { + TRX_TS_RD = (1 << 0), + TRXTS_SEL = (1 << 1), + RX_TS_PDLYRSP_RDY = (1 << 8), + RX_TS_PDLYREQ_RDY = (1 << 9), + RX_TS_DLYREQ_RDY = (1 << 10), + RX_TS_SYNC_RDY = (1 << 11), + TX_TS_PDLYRSP_RDY = (1 << 12), + TX_TS_PDLYREQ_RDY = (1 << 13), + TX_TS_DLYREQ_RDY = (1 << 14), + TX_TS_SYNC_RDY = (1 << 15), +}; + +#define PTP_CTL_TYPE_0 (0xF3F) +#define PTP_CTL_TYPE_1 (0x2FF) +#define PTP_CTL_TYPE_2 (0x0FF) +#define PTP_CTL_TYPE_3 (0x03F) + +#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) +struct clock_identity { + u8 id[8]; +} __packed; + +struct port_identity { + struct clock_identity clock_identity; + __be16 port_number; +} __packed; + +struct ptp_header { + u8 tsmt; /* transportSpecific | messageType */ + u8 ver; /* reserved | versionPTP */ + __be16 message_length; + u8 domain_number; + u8 reserved1; + u8 flag_field[2]; + __be64 correction; + __be32 reserved2; + struct port_identity source_port_identity; + __be16 sequence_id; + u8 control; + u8 log_message_interval; +} __packed; + +/** + * ptp_parse_header - Get pointer to the PTP v2 header + * @skb: packet buffer + * @type: type of the packet (see ptp_classify_raw()) + * + * This function takes care of the VLAN, UDP, IPv4 and IPv6 headers. The length + * is checked. + * + * Note, internally skb_mac_header() is used. Make sure that the @skb is + * initialized accordingly. + * + * Return: Pointer to the ptp v2 header or NULL if not found + */ +struct ptp_header *ptp_parse_header(struct sk_buff *skb, unsigned int type); + +/** + * ptp_get_msgtype - Extract ptp message type from given header + * @hdr: ptp header + * @type: type of the packet (see ptp_classify_raw()) + * + * This function returns the message type for a given ptp header. It takes care + * of the different ptp header versions (v1 or v2). + * + * Return: The message type + */ +static inline u8 ptp_get_msgtype(const struct ptp_header *hdr, + unsigned int type) +{ + u8 msgtype; + + if (unlikely(type & PTP_CLASS_V1)) { + /* msg type is located at the control field for ptp v1 */ + msgtype = hdr->control; + } else { + msgtype = hdr->tsmt & 0x0f; + } + + return msgtype; +} + +#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) */ + +struct rtl8127_private; +struct RxDescV3; + +int rtl8127_get_ts_info(struct net_device *netdev, + struct ethtool_ts_info *info); + +void rtl8127_ptp_reset(struct rtl8127_private *tp); +void rtl8127_ptp_init(struct rtl8127_private *tp); +void rtl8127_ptp_suspend(struct rtl8127_private *tp); +void rtl8127_ptp_stop(struct rtl8127_private *tp); + +int rtl8127_ptp_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd); + +void rtl8127_rx_ptp_timestamp(struct rtl8127_private *tp, struct sk_buff *skb); + +void rtl8127_set_local_time(struct rtl8127_private *tp); + +#endif /* _LINUX_R8127_PTP_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h b/drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h new file mode 100755 index 0000000000000..a869b6c532b33 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h @@ -0,0 +1,118 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_R8127_REALWOW_H +#define _LINUX_R8127_REALWOW_H + +#define SIOCDEVPRIVATE_RTLREALWOW SIOCDEVPRIVATE+3 + +#define MAX_RealWoW_KCP_SIZE (100) +#define MAX_RealWoW_Payload (64) + +#define KA_TX_PACKET_SIZE (100) +#define KA_WAKEUP_PATTERN_SIZE (120) + +//HwSuppKeepAliveOffloadVer +#define HW_SUPPORT_KCP_OFFLOAD(_M) ((_M)->HwSuppKCPOffloadVer > 0) + +enum rtl_realwow_cmd { + + RTL_REALWOW_SET_KCP_DISABLE=0, + RTL_REALWOW_SET_KCP_INFO, + RTL_REALWOW_SET_KCP_CONTENT, + + RTL_REALWOW_SET_KCP_ACKPKTINFO, + RTL_REALWOW_SET_KCP_WPINFO, + RTL_REALWOW_SET_KCPDHCP_TIMEOUT, + + RTLT_REALWOW_COMMAND_INVALID +}; + +struct rtl_realwow_ioctl_struct { + __u32 cmd; + __u32 offset; + __u32 len; + union { + __u32 data; + void *data_buffer; + }; +}; + +typedef struct _MP_KCPInfo { + u8 DIPv4[4]; + u8 MacID[6]; + u16 UdpPort[2]; + u8 PKTLEN[2]; + + u16 ackLostCnt; + u8 KCP_WakePattern[MAX_RealWoW_Payload]; + u8 KCP_AckPacket[MAX_RealWoW_Payload]; + u32 KCP_interval; + u8 KCP_WakePattern_Len; + u8 KCP_AckPacket_Len; + u8 KCP_TxPacket[2][KA_TX_PACKET_SIZE]; +} MP_KCP_INFO, *PMP_KCP_INFO; + +typedef struct _KCPInfo { + u32 nId; // = id + u8 DIPv4[4]; + u8 MacID[6]; + u16 UdpPort; + u16 PKTLEN; +} KCPInfo, *PKCPInfo; + +typedef struct _KCPContent { + u32 id; // = id + u32 mSec; // = msec + u32 size; // =size + u8 bPacket[MAX_RealWoW_KCP_SIZE]; // put packet here +} KCPContent, *PKCPContent; + +typedef struct _RealWoWAckPktInfo { + u16 ackLostCnt; + u16 patterntSize; + u8 pattern[MAX_RealWoW_Payload]; +} RealWoWAckPktInfo,*PRealWoWAckPktInfo; + +typedef struct _RealWoWWPInfo { + u16 patterntSize; + u8 pattern[MAX_RealWoW_Payload]; +} RealWoWWPInfo,*PRealWoWWPInfo; + +int rtl8127_realwow_ioctl(struct net_device *dev, struct ifreq *ifr); +void rtl8127_realwow_hw_init(struct net_device *dev); +void rtl8127_get_realwow_hw_version(struct net_device *dev); +void rtl8127_set_realwow_d3_para(struct net_device *dev); + +#endif /* _LINUX_R8127_REALWOW_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_rss.c b/drivers/net/ethernet/realtek/r8127/src/r8127_rss.c new file mode 100755 index 0000000000000..e364621910052 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_rss.c @@ -0,0 +1,583 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#include +#include "r8127.h" + +enum rtl8127_rss_register_content { + /* RSS */ + RSS_CTRL_TCP_IPV4_SUPP = (1 << 0), + RSS_CTRL_IPV4_SUPP = (1 << 1), + RSS_CTRL_TCP_IPV6_SUPP = (1 << 2), + RSS_CTRL_IPV6_SUPP = (1 << 3), + RSS_CTRL_IPV6_EXT_SUPP = (1 << 4), + RSS_CTRL_TCP_IPV6_EXT_SUPP = (1 << 5), + RSS_HALF_SUPP = (1 << 7), + RSS_CTRL_UDP_IPV4_SUPP = (1 << 11), + RSS_CTRL_UDP_IPV6_SUPP = (1 << 12), + RSS_CTRL_UDP_IPV6_EXT_SUPP = (1 << 13), + RSS_QUAD_CPU_EN = (1 << 16), + RSS_HQ_Q_SUP_R = (1 << 31), +}; + +static int rtl8127_get_rss_hash_opts(struct rtl8127_private *tp, + struct ethtool_rxnfc *cmd) +{ + cmd->data = 0; + + /* Report default options for RSS */ + switch (cmd->flow_type) { + case TCP_V4_FLOW: + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + fallthrough; + case UDP_V4_FLOW: + if (tp->rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + fallthrough; + case IPV4_FLOW: + cmd->data |= RXH_IP_SRC | RXH_IP_DST; + break; + case TCP_V6_FLOW: + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + fallthrough; + case UDP_V6_FLOW: + if (tp->rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) + cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; + fallthrough; + case IPV6_FLOW: + cmd->data |= RXH_IP_SRC | RXH_IP_DST; + break; + default: + return -EINVAL; + } + + return 0; +} + +int rtl8127_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd, + u32 *rule_locs) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret = -EOPNOTSUPP; + + if (!(dev->features & NETIF_F_RXHASH)) + return ret; + + switch (cmd->cmd) { + case ETHTOOL_GRXRINGS: + cmd->data = rtl8127_tot_rx_rings(tp); + ret = 0; + break; + case ETHTOOL_GRXFH: + ret = rtl8127_get_rss_hash_opts(tp, cmd); + break; + default: + break; + } + + return ret; +} + +u32 rtl8127_rss_indir_tbl_entries(struct rtl8127_private *tp) +{ + return tp->HwSuppIndirTblEntries; +} + +#define RSS_MASK_BITS_OFFSET (8) +#define RSS_CPU_NUM_OFFSET (16) +#define RTL8127_UDP_RSS_FLAGS (RTL_8125_RSS_FLAG_HASH_UDP_IPV4 | \ + RTL_8125_RSS_FLAG_HASH_UDP_IPV6) +static int _rtl8127_set_rss_hash_opt(struct rtl8127_private *tp) +{ + u32 rss_flags = tp->rss_flags; + u32 hash_mask_len; + u32 rss_ctrl; + + rss_ctrl = ilog2(rtl8127_tot_rx_rings(tp)); + rss_ctrl &= (BIT_0 | BIT_1 | BIT_2); + rss_ctrl <<= RSS_CPU_NUM_OFFSET; + + /* Perform hash on these packet types */ + rss_ctrl |= RSS_CTRL_TCP_IPV4_SUPP + | RSS_CTRL_IPV4_SUPP + | RSS_CTRL_IPV6_SUPP + | RSS_CTRL_IPV6_EXT_SUPP + | RSS_CTRL_TCP_IPV6_SUPP + | RSS_CTRL_TCP_IPV6_EXT_SUPP; + + if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) + rss_ctrl |= RSS_CTRL_UDP_IPV4_SUPP; + + if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) + rss_ctrl |= RSS_CTRL_UDP_IPV6_SUPP | + RSS_CTRL_UDP_IPV6_EXT_SUPP; + + hash_mask_len = ilog2(rtl8127_rss_indir_tbl_entries(tp)); + hash_mask_len &= (BIT_0 | BIT_1 | BIT_2); + rss_ctrl |= hash_mask_len << RSS_MASK_BITS_OFFSET; + + RTL_W32(tp, RSS_CTRL_8125, rss_ctrl); + + return 0; +} + +static int rtl8127_set_rss_hash_opt(struct rtl8127_private *tp, + struct ethtool_rxnfc *nfc) +{ + u32 rss_flags = tp->rss_flags; + + /* + * RSS does not support anything other than hashing + * to queues on src and dst IPs and ports + */ + if (nfc->data & ~(RXH_IP_SRC | RXH_IP_DST | + RXH_L4_B_0_1 | RXH_L4_B_2_3)) + return -EINVAL; + + switch (nfc->flow_type) { + case TCP_V4_FLOW: + case TCP_V6_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST) || + !(nfc->data & RXH_L4_B_0_1) || + !(nfc->data & RXH_L4_B_2_3)) + return -EINVAL; + break; + case UDP_V4_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST)) + return -EINVAL; + switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { + case 0: + rss_flags &= ~RTL_8125_RSS_FLAG_HASH_UDP_IPV4; + break; + case (RXH_L4_B_0_1 | RXH_L4_B_2_3): + rss_flags |= RTL_8125_RSS_FLAG_HASH_UDP_IPV4; + break; + default: + return -EINVAL; + } + break; + case UDP_V6_FLOW: + if (!(nfc->data & RXH_IP_SRC) || + !(nfc->data & RXH_IP_DST)) + return -EINVAL; + switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { + case 0: + rss_flags &= ~RTL_8125_RSS_FLAG_HASH_UDP_IPV6; + break; + case (RXH_L4_B_0_1 | RXH_L4_B_2_3): + rss_flags |= RTL_8125_RSS_FLAG_HASH_UDP_IPV6; + break; + default: + return -EINVAL; + } + break; + case SCTP_V4_FLOW: + case AH_ESP_V4_FLOW: + case AH_V4_FLOW: + case ESP_V4_FLOW: + case SCTP_V6_FLOW: + case AH_ESP_V6_FLOW: + case AH_V6_FLOW: + case ESP_V6_FLOW: + case IP_USER_FLOW: + case ETHER_FLOW: + /* RSS is not supported for these protocols */ + if (nfc->data) { + netif_err(tp, drv, tp->dev, "Command parameters not supported\n"); + return -EINVAL; + } + return 0; + break; + default: + return -EINVAL; + } + + /* if we changed something we need to update flags */ + if (rss_flags != tp->rss_flags) { + u32 rss_ctrl = RTL_R32(tp, RSS_CTRL_8125); + + if ((rss_flags & RTL8127_UDP_RSS_FLAGS) && + !(tp->rss_flags & RTL8127_UDP_RSS_FLAGS)) + netdev_warn(tp->dev, + "enabling UDP RSS: fragmented packets may " + "arrive out of order to the stack above\n"); + + tp->rss_flags = rss_flags; + + /* Perform hash on these packet types */ + rss_ctrl |= RSS_CTRL_TCP_IPV4_SUPP + | RSS_CTRL_IPV4_SUPP + | RSS_CTRL_IPV6_SUPP + | RSS_CTRL_IPV6_EXT_SUPP + | RSS_CTRL_TCP_IPV6_SUPP + | RSS_CTRL_TCP_IPV6_EXT_SUPP; + + rss_ctrl &= ~(RSS_CTRL_UDP_IPV4_SUPP | + RSS_CTRL_UDP_IPV6_SUPP | + RSS_CTRL_UDP_IPV6_EXT_SUPP); + + if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) + rss_ctrl |= RSS_CTRL_UDP_IPV4_SUPP; + + if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) + rss_ctrl |= RSS_CTRL_UDP_IPV6_SUPP | + RSS_CTRL_UDP_IPV6_EXT_SUPP; + + RTL_W32(tp, RSS_CTRL_8125, rss_ctrl); + } + + return 0; +} + +int rtl8127_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int ret = -EOPNOTSUPP; + + if (!(dev->features & NETIF_F_RXHASH)) + return ret; + + switch (cmd->cmd) { + case ETHTOOL_SRXFH: + ret = rtl8127_set_rss_hash_opt(tp, cmd); + break; + default: + break; + } + + return ret; +} + +static u32 _rtl8127_get_rxfh_key_size(struct rtl8127_private *tp) +{ + return sizeof(tp->rss_key); +} + +u32 rtl8127_get_rxfh_key_size(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (!(dev->features & NETIF_F_RXHASH)) + return 0; + + return _rtl8127_get_rxfh_key_size(tp); +} + +u32 rtl8127_rss_indir_size(struct net_device *dev) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (!(dev->features & NETIF_F_RXHASH)) + return 0; + + return rtl8127_rss_indir_tbl_entries(tp); +} + +static void rtl8127_get_reta(struct rtl8127_private *tp, u32 *indir) +{ + int i, reta_size = rtl8127_rss_indir_tbl_entries(tp); + + for (i = 0; i < reta_size; i++) + indir[i] = tp->rss_indir_tbl[i]; +} + +static u32 rtl8127_rss_key_reg(struct rtl8127_private *tp) +{ + return RSS_KEY_8125; +} + +static u32 rtl8127_rss_indir_tbl_reg(struct rtl8127_private *tp) +{ + return RSS_INDIRECTION_TBL_8125_V2; +} + +static void rtl8127_store_reta(struct rtl8127_private *tp) +{ + u16 indir_tbl_reg = rtl8127_rss_indir_tbl_reg(tp); + u32 i, reta_entries = rtl8127_rss_indir_tbl_entries(tp); + u32 reta = 0; + u8 *indir_tbl = tp->rss_indir_tbl; + + /* Write redirection table to HW */ + for (i = 0; i < reta_entries; i++) { + reta |= indir_tbl[i] << (i & 0x3) * 8; + if ((i & 3) == 3) { + RTL_W32(tp, indir_tbl_reg, reta); + + indir_tbl_reg += 4; + reta = 0; + } + } +} + +static void rtl8127_store_rss_key(struct rtl8127_private *tp) +{ + const u16 rss_key_reg = rtl8127_rss_key_reg(tp); + u32 i, rss_key_size = _rtl8127_get_rxfh_key_size(tp); + u32 *rss_key = (u32*)tp->rss_key; + + /* Write redirection table to HW */ + for (i = 0; i < rss_key_size; i+=4) + RTL_W32(tp, rss_key_reg + i, *rss_key++); +} + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) +int rtl8127_get_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (!(dev->features & NETIF_F_RXHASH)) + return -EOPNOTSUPP; + + rxfh->hfunc = ETH_RSS_HASH_TOP; + + if (rxfh->indir) + rtl8127_get_reta(tp, rxfh->indir); + + if (rxfh->key) + memcpy(rxfh->key, tp->rss_key, RTL8127_RSS_KEY_SIZE); + + return 0; +} + +int rtl8127_set_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh, + struct netlink_ext_ack *extack) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + u32 reta_entries = rtl8127_rss_indir_tbl_entries(tp); + + /* We require at least one supported parameter to be changed and no + * change in any of the unsupported parameters + */ + if (rxfh->hfunc != ETH_RSS_HASH_NO_CHANGE && rxfh->hfunc != ETH_RSS_HASH_TOP) + return -EOPNOTSUPP; + + /* Fill out the redirection table */ + if (rxfh->indir) { + int max_queues = tp->num_rx_rings; + + /* Verify user input. */ + for (i = 0; i < reta_entries; i++) + if (rxfh->indir[i] >= max_queues) + return -EINVAL; + + for (i = 0; i < reta_entries; i++) + tp->rss_indir_tbl[i] = rxfh->indir[i]; + } + + /* Fill out the rss hash key */ + if (rxfh->key) + memcpy(tp->rss_key, rxfh->key, RTL8127_RSS_KEY_SIZE); + + rtl8127_store_reta(tp); + + rtl8127_store_rss_key(tp); + + return 0; +} +#else +int rtl8127_get_rxfh(struct net_device *dev, u32 *indir, u8 *key, + u8 *hfunc) +{ + struct rtl8127_private *tp = netdev_priv(dev); + + if (!(dev->features & NETIF_F_RXHASH)) + return -EOPNOTSUPP; + + if (hfunc) + *hfunc = ETH_RSS_HASH_TOP; + + if (indir) + rtl8127_get_reta(tp, indir); + + if (key) + memcpy(key, tp->rss_key, RTL8127_RSS_KEY_SIZE); + + return 0; +} + +int rtl8127_set_rxfh(struct net_device *dev, const u32 *indir, + const u8 *key, const u8 hfunc) +{ + struct rtl8127_private *tp = netdev_priv(dev); + int i; + u32 reta_entries = rtl8127_rss_indir_tbl_entries(tp); + + /* We require at least one supported parameter to be changed and no + * change in any of the unsupported parameters + */ + if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP) + return -EOPNOTSUPP; + + /* Fill out the redirection table */ + if (indir) { + int max_queues = tp->num_rx_rings; + + /* Verify user input. */ + for (i = 0; i < reta_entries; i++) + if (indir[i] >= max_queues) + return -EINVAL; + + for (i = 0; i < reta_entries; i++) + tp->rss_indir_tbl[i] = indir[i]; + } + + /* Fill out the rss hash key */ + if (key) + memcpy(tp->rss_key, key, RTL8127_RSS_KEY_SIZE); + + rtl8127_store_reta(tp); + + rtl8127_store_rss_key(tp); + + return 0; +} +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) */ + +static u32 rtl8127_get_rx_desc_hash(struct rtl8127_private *tp, + struct RxDesc *desc) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + return le32_to_cpu(((struct RxDescV3 *)desc)->RxDescNormalDDWord2.RSSResult); + case RX_DESC_RING_TYPE_4: + return le32_to_cpu(((struct RxDescV4 *)desc)->RxDescNormalDDWord1.RSSResult); + default: + return 0; + } +} + +#define RXS_8125B_RSS_UDP BIT(9) +#define RXS_8125_RSS_IPV4 BIT(10) +#define RXS_8125_RSS_IPV6 BIT(12) +#define RXS_8125_RSS_TCP BIT(13) +#define RTL8127_RXS_RSS_L3_TYPE_MASK (RXS_8125_RSS_IPV4 | RXS_8125_RSS_IPV6) +#define RTL8127_RXS_RSS_L4_TYPE_MASK (RXS_8125_RSS_TCP | RXS_8125B_RSS_UDP) + +#define RXS_8125B_RSS_UDP_V4 BIT(27) +#define RXS_8125_RSS_IPV4_V4 BIT(28) +#define RXS_8125_RSS_IPV6_V4 BIT(29) +#define RXS_8125_RSS_TCP_V4 BIT(30) +#define RTL8127_RXS_RSS_L3_TYPE_MASK_V4 (RXS_8125_RSS_IPV4_V4 | RXS_8125_RSS_IPV6_V4) +#define RTL8127_RXS_RSS_L4_TYPE_MASK_V4 (RXS_8125_RSS_TCP_V4 | RXS_8125B_RSS_UDP_V4) +static void rtl8127_rx_hash_v3(struct rtl8127_private *tp, + struct RxDescV3 *descv3, + struct sk_buff *skb) +{ + u16 rss_header_info; + + if (!(tp->dev->features & NETIF_F_RXHASH)) + return; + + rss_header_info = le16_to_cpu(descv3->RxDescNormalDDWord2.HeaderInfo); + + if (!(rss_header_info & RTL8127_RXS_RSS_L3_TYPE_MASK)) + return; + + skb_set_hash(skb, rtl8127_get_rx_desc_hash(tp, (struct RxDesc *)descv3), + (RTL8127_RXS_RSS_L4_TYPE_MASK & rss_header_info) ? + PKT_HASH_TYPE_L4 : PKT_HASH_TYPE_L3); +} + +static void rtl8127_rx_hash_v4(struct rtl8127_private *tp, + struct RxDescV4 *descv4, + struct sk_buff *skb) +{ + u32 rss_header_info; + + if (!(tp->dev->features & NETIF_F_RXHASH)) + return; + + rss_header_info = le32_to_cpu(descv4->RxDescNormalDDWord1.RSSInfo); + + if (!(rss_header_info & RTL8127_RXS_RSS_L3_TYPE_MASK_V4)) + return; + + skb_set_hash(skb, rtl8127_get_rx_desc_hash(tp, (struct RxDesc *)descv4), + (RTL8127_RXS_RSS_L4_TYPE_MASK_V4 & rss_header_info) ? + PKT_HASH_TYPE_L4 : PKT_HASH_TYPE_L3); +} + +void rtl8127_rx_hash(struct rtl8127_private *tp, + struct RxDesc *desc, + struct sk_buff *skb) +{ + switch (tp->InitRxDescType) { + case RX_DESC_RING_TYPE_3: + rtl8127_rx_hash_v3(tp, (struct RxDescV3 *)desc, skb); + break; + case RX_DESC_RING_TYPE_4: + rtl8127_rx_hash_v4(tp, (struct RxDescV4 *)desc, skb); + break; + default: + return; + } +} + +void rtl8127_disable_rss(struct rtl8127_private *tp) +{ + RTL_W32(tp, RSS_CTRL_8125, 0x00); +} + +void _rtl8127_config_rss(struct rtl8127_private *tp) +{ + _rtl8127_set_rss_hash_opt(tp); + + rtl8127_store_reta(tp); + + rtl8127_store_rss_key(tp); +} + +void rtl8127_config_rss(struct rtl8127_private *tp) +{ + if (!tp->EnableRss) { + rtl8127_disable_rss(tp); + return; + } + + _rtl8127_config_rss(tp); +} + +void rtl8127_init_rss(struct rtl8127_private *tp) +{ + int i; + + for (i = 0; i < rtl8127_rss_indir_tbl_entries(tp); i++) + tp->rss_indir_tbl[i] = ethtool_rxfh_indir_default(i, tp->num_rx_rings); + + netdev_rss_key_fill(tp->rss_key, RTL8127_RSS_KEY_SIZE); +} diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_rss.h b/drivers/net/ethernet/realtek/r8127/src/r8127_rss.h new file mode 100755 index 0000000000000..8e92bb830c969 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/r8127_rss.h @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_R8127_RSS_H +#define _LINUX_R8127_RSS_H + +#include +#include + +#define RTL8127_RSS_KEY_SIZE 40 /* size of RSS Hash Key in bytes */ +#define RTL8127_MAX_INDIRECTION_TABLE_ENTRIES 128 + +enum rtl8127_rss_flag { + RTL_8125_RSS_FLAG_HASH_UDP_IPV4 = (1 << 0), + RTL_8125_RSS_FLAG_HASH_UDP_IPV6 = (1 << 1), +}; + +struct rtl8127_private; +struct RxDesc; + +int rtl8127_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd, + u32 *rule_locs); +int rtl8127_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd); +u32 rtl8127_get_rxfh_key_size(struct net_device *netdev); +u32 rtl8127_rss_indir_size(struct net_device *netdev); +#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) +int rtl8127_get_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh); +int rtl8127_set_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh, + struct netlink_ext_ack *extack); +#else +int rtl8127_get_rxfh(struct net_device *netdev, u32 *indir, u8 *key, + u8 *hfunc); +int rtl8127_set_rxfh(struct net_device *netdev, const u32 *indir, + const u8 *key, const u8 hfunc); +#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) */ +void rtl8127_rx_hash(struct rtl8127_private *tp, + struct RxDesc *desc, + struct sk_buff *skb); +void _rtl8127_config_rss(struct rtl8127_private *tp); +void rtl8127_config_rss(struct rtl8127_private *tp); +void rtl8127_init_rss(struct rtl8127_private *tp); +u32 rtl8127_rss_indir_tbl_entries(struct rtl8127_private *tp); +void rtl8127_disable_rss(struct rtl8127_private *tp); + +#endif /* _LINUX_R8127_RSS_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c b/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c new file mode 100755 index 0000000000000..c95bbf0d19537 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c @@ -0,0 +1,285 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#include +#include +#include +#include +#include +#include + +#include + +#include "r8127.h" +#include "rtl_eeprom.h" + +//------------------------------------------------------------------- +//rtl8127_eeprom_type(): +// tell the eeprom type +//return value: +// 0: the eeprom type is 93C46 +// 1: the eeprom type is 93C56 or 93C66 +//------------------------------------------------------------------- +void rtl8127_eeprom_type(struct rtl8127_private *tp) +{ + u16 magic = 0; + + if (tp->mcfg == CFG_METHOD_DEFAULT) + goto out_no_eeprom; + + if(RTL_R8(tp, 0xD2)&0x04) { + //not support + //tp->eeprom_type = EEPROM_TWSI; + //tp->eeprom_len = 256; + goto out_no_eeprom; + } else if(RTL_R32(tp, RxConfig) & RxCfg_9356SEL) { + tp->eeprom_type = EEPROM_TYPE_93C56; + tp->eeprom_len = 256; + } else { + tp->eeprom_type = EEPROM_TYPE_93C46; + tp->eeprom_len = 128; + } + + magic = rtl8127_eeprom_read_sc(tp, 0); + +out_no_eeprom: + if ((magic != 0x8129) && (magic != 0x8128)) { + tp->eeprom_type = EEPROM_TYPE_NONE; + tp->eeprom_len = 0; + } +} + +void rtl8127_eeprom_cleanup(struct rtl8127_private *tp) +{ + u8 x; + + x = RTL_R8(tp, Cfg9346); + x &= ~(Cfg9346_EEDI | Cfg9346_EECS); + + RTL_W8(tp, Cfg9346, x); + + rtl8127_raise_clock(tp, &x); + rtl8127_lower_clock(tp, &x); +} + +static int rtl8127_eeprom_cmd_done(struct rtl8127_private *tp) +{ + u8 x; + int i; + + rtl8127_stand_by(tp); + + for (i = 0; i < 50000; i++) { + x = RTL_R8(tp, Cfg9346); + + if (x & Cfg9346_EEDO) { + fsleep(RTL_CLOCK_RATE * 2 * 3); + return 0; + } + fsleep(1); + } + + return -1; +} + +//------------------------------------------------------------------- +//rtl8127_eeprom_read_sc(): +// read one word from eeprom +//------------------------------------------------------------------- +u16 rtl8127_eeprom_read_sc(struct rtl8127_private *tp, u16 reg) +{ + int addr_sz = 6; + u8 x; + u16 data; + + if(tp->eeprom_type == EEPROM_TYPE_NONE) + return -1; + + if (tp->eeprom_type==EEPROM_TYPE_93C46) + addr_sz = 6; + else if (tp->eeprom_type==EEPROM_TYPE_93C56) + addr_sz = 8; + + x = Cfg9346_EEM1 | Cfg9346_EECS; + RTL_W8(tp, Cfg9346, x); + + rtl8127_shift_out_bits(tp, RTL_EEPROM_READ_OPCODE, 3); + rtl8127_shift_out_bits(tp, reg, addr_sz); + + data = rtl8127_shift_in_bits(tp); + + rtl8127_eeprom_cleanup(tp); + + RTL_W8(tp, Cfg9346, 0); + + return data; +} + +//------------------------------------------------------------------- +//rtl8127_eeprom_write_sc(): +// write one word to a specific address in the eeprom +//------------------------------------------------------------------- +void rtl8127_eeprom_write_sc(struct rtl8127_private *tp, u16 reg, u16 data) +{ + u8 x; + int addr_sz = 6; + int w_dummy_addr = 4; + + if(tp->eeprom_type == EEPROM_TYPE_NONE) + return; + + if (tp->eeprom_type==EEPROM_TYPE_93C46) { + addr_sz = 6; + w_dummy_addr = 4; + } else if (tp->eeprom_type==EEPROM_TYPE_93C56) { + addr_sz = 8; + w_dummy_addr = 6; + } + + x = Cfg9346_EEM1 | Cfg9346_EECS; + RTL_W8(tp, Cfg9346, x); + + rtl8127_shift_out_bits(tp, RTL_EEPROM_EWEN_OPCODE, 5); + rtl8127_shift_out_bits(tp, reg, w_dummy_addr); + rtl8127_stand_by(tp); + + rtl8127_shift_out_bits(tp, RTL_EEPROM_ERASE_OPCODE, 3); + rtl8127_shift_out_bits(tp, reg, addr_sz); + if (rtl8127_eeprom_cmd_done(tp) < 0) + return; + rtl8127_stand_by(tp); + + rtl8127_shift_out_bits(tp, RTL_EEPROM_WRITE_OPCODE, 3); + rtl8127_shift_out_bits(tp, reg, addr_sz); + rtl8127_shift_out_bits(tp, data, 16); + if (rtl8127_eeprom_cmd_done(tp) < 0) + return; + rtl8127_stand_by(tp); + + rtl8127_shift_out_bits(tp, RTL_EEPROM_EWDS_OPCODE, 5); + rtl8127_shift_out_bits(tp, reg, w_dummy_addr); + + rtl8127_eeprom_cleanup(tp); + RTL_W8(tp, Cfg9346, 0); +} + +void rtl8127_raise_clock(struct rtl8127_private *tp, u8 *x) +{ + *x = *x | Cfg9346_EESK; + RTL_W8(tp, Cfg9346, *x); + fsleep(RTL_CLOCK_RATE); +} + +void rtl8127_lower_clock(struct rtl8127_private *tp, u8 *x) +{ + + *x = *x & ~Cfg9346_EESK; + RTL_W8(tp, Cfg9346, *x); + fsleep(RTL_CLOCK_RATE); +} + +void rtl8127_shift_out_bits(struct rtl8127_private *tp, int data, int count) +{ + u8 x; + int mask; + + mask = 0x01 << (count - 1); + x = RTL_R8(tp, Cfg9346); + x &= ~(Cfg9346_EEDI | Cfg9346_EEDO); + + do { + if (data & mask) + x |= Cfg9346_EEDI; + else + x &= ~Cfg9346_EEDI; + + RTL_W8(tp, Cfg9346, x); + fsleep(RTL_CLOCK_RATE); + rtl8127_raise_clock(tp, &x); + rtl8127_lower_clock(tp, &x); + mask = mask >> 1; + } while(mask); + + x &= ~Cfg9346_EEDI; + RTL_W8(tp, Cfg9346, x); +} + +u16 rtl8127_shift_in_bits(struct rtl8127_private *tp) +{ + u8 x; + u16 d, i; + + x = RTL_R8(tp, Cfg9346); + x &= ~(Cfg9346_EEDI | Cfg9346_EEDO); + + d = 0; + + for (i = 0; i < 16; i++) { + d = d << 1; + rtl8127_raise_clock(tp, &x); + + x = RTL_R8(tp, Cfg9346); + x &= ~Cfg9346_EEDI; + + if (x & Cfg9346_EEDO) + d |= 1; + + rtl8127_lower_clock(tp, &x); + } + + return d; +} + +void rtl8127_stand_by(struct rtl8127_private *tp) +{ + u8 x; + + x = RTL_R8(tp, Cfg9346); + x &= ~(Cfg9346_EECS | Cfg9346_EESK); + RTL_W8(tp, Cfg9346, x); + fsleep(RTL_CLOCK_RATE); + + x |= Cfg9346_EECS; + RTL_W8(tp, Cfg9346, x); +} + +void rtl8127_set_eeprom_sel_low(struct rtl8127_private *tp) +{ + RTL_W8(tp, Cfg9346, Cfg9346_EEM1); + RTL_W8(tp, Cfg9346, Cfg9346_EEM1 | Cfg9346_EESK); + + fsleep(20); + + RTL_W8(tp, Cfg9346, Cfg9346_EEM1); +} diff --git a/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h b/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h new file mode 100755 index 0000000000000..e4d8c6c3f6765 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_RTLEEPROM_H +#define _LINUX_RTLEEPROM_H + +//EEPROM opcodes +#define RTL_EEPROM_READ_OPCODE 06 +#define RTL_EEPROM_WRITE_OPCODE 05 +#define RTL_EEPROM_ERASE_OPCODE 07 +#define RTL_EEPROM_EWEN_OPCODE 19 +#define RTL_EEPROM_EWDS_OPCODE 16 + +#define RTL_CLOCK_RATE 3 + +void rtl8127_eeprom_type(struct rtl8127_private *tp); +void rtl8127_eeprom_cleanup(struct rtl8127_private *tp); +u16 rtl8127_eeprom_read_sc(struct rtl8127_private *tp, u16 reg); +void rtl8127_eeprom_write_sc(struct rtl8127_private *tp, u16 reg, u16 data); +void rtl8127_shift_out_bits(struct rtl8127_private *tp, int data, int count); +u16 rtl8127_shift_in_bits(struct rtl8127_private *tp); +void rtl8127_raise_clock(struct rtl8127_private *tp, u8 *x); +void rtl8127_lower_clock(struct rtl8127_private *tp, u8 *x); +void rtl8127_stand_by(struct rtl8127_private *tp); +void rtl8127_set_eeprom_sel_low(struct rtl8127_private *tp); + +#endif /* _LINUX_RTLEEPROM_H */ diff --git a/drivers/net/ethernet/realtek/r8127/src/rtltool.c b/drivers/net/ethernet/realtek/r8127/src/rtltool.c new file mode 100755 index 0000000000000..ba3a0cfd32420 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/rtltool.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "r8127.h" +#include "rtl_eeprom.h" +#include "rtltool.h" + +int rtl8127_tool_ioctl(struct rtl8127_private *tp, struct ifreq *ifr) +{ + struct rtltool_cmd my_cmd; + int ret; + + if (copy_from_user(&my_cmd, ifr->ifr_data, sizeof(my_cmd))) + return -EFAULT; + + ret = 0; + switch (my_cmd.cmd) { + case RTLTOOL_READ_MAC: + if ((my_cmd.offset + my_cmd.len) > pci_resource_len(tp->pci_dev, 2)) { + ret = -EINVAL; + break; + } + + if (my_cmd.len==1) + my_cmd.data = readb(tp->mmio_addr+my_cmd.offset); + else if (my_cmd.len==2) + my_cmd.data = readw(tp->mmio_addr+(my_cmd.offset&~1)); + else if (my_cmd.len==4) + my_cmd.data = readl(tp->mmio_addr+(my_cmd.offset&~3)); + else { + ret = -EOPNOTSUPP; + break; + } + + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + break; + + case RTLTOOL_WRITE_MAC: + if ((my_cmd.offset + my_cmd.len) > pci_resource_len(tp->pci_dev, 2)) { + ret = -EINVAL; + break; + } + + if (my_cmd.len==1) + writeb(my_cmd.data, tp->mmio_addr+my_cmd.offset); + else if (my_cmd.len==2) + writew(my_cmd.data, tp->mmio_addr+(my_cmd.offset&~1)); + else if (my_cmd.len==4) + writel(my_cmd.data, tp->mmio_addr+(my_cmd.offset&~3)); + else { + ret = -EOPNOTSUPP; + break; + } + + break; + + case RTLTOOL_READ_PHY: + my_cmd.data = rtl8127_mdio_prot_read(tp, my_cmd.offset); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + + break; + + case RTLTOOL_WRITE_PHY: + rtl8127_mdio_prot_write(tp, my_cmd.offset, my_cmd.data); + break; + + case RTLTOOL_READ_EPHY: + my_cmd.data = rtl8127_ephy_read(tp, my_cmd.offset); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + + break; + + case RTLTOOL_WRITE_EPHY: + rtl8127_ephy_write(tp, my_cmd.offset, my_cmd.data); + break; + + case RTLTOOL_READ_ERI: + my_cmd.data = 0; + if (my_cmd.len==1 || my_cmd.len==2 || my_cmd.len==4) { + my_cmd.data = rtl8127_eri_read(tp, my_cmd.offset, my_cmd.len, ERIAR_ExGMAC); + } else { + ret = -EOPNOTSUPP; + break; + } + + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + + break; + + case RTLTOOL_WRITE_ERI: + if (my_cmd.len==1 || my_cmd.len==2 || my_cmd.len==4) { + rtl8127_eri_write(tp, my_cmd.offset, my_cmd.len, my_cmd.data, ERIAR_ExGMAC); + } else { + ret = -EOPNOTSUPP; + break; + } + break; + + case RTLTOOL_READ_PCI: + my_cmd.data = 0; + if (my_cmd.len==1) + pci_read_config_byte(tp->pci_dev, my_cmd.offset, + (u8 *)&my_cmd.data); + else if (my_cmd.len==2) + pci_read_config_word(tp->pci_dev, my_cmd.offset, + (u16 *)&my_cmd.data); + else if (my_cmd.len==4) + pci_read_config_dword(tp->pci_dev, my_cmd.offset, + &my_cmd.data); + else { + ret = -EOPNOTSUPP; + break; + } + + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + break; + + case RTLTOOL_WRITE_PCI: + if (my_cmd.len==1) + pci_write_config_byte(tp->pci_dev, my_cmd.offset, + my_cmd.data); + else if (my_cmd.len==2) + pci_write_config_word(tp->pci_dev, my_cmd.offset, + my_cmd.data); + else if (my_cmd.len==4) + pci_write_config_dword(tp->pci_dev, my_cmd.offset, + my_cmd.data); + else { + ret = -EOPNOTSUPP; + break; + } + + break; + + case RTLTOOL_READ_EEPROM: + my_cmd.data = rtl8127_eeprom_read_sc(tp, my_cmd.offset); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + + break; + + case RTLTOOL_WRITE_EEPROM: + rtl8127_eeprom_write_sc(tp, my_cmd.offset, my_cmd.data); + break; + + case RTL_READ_OOB_MAC: + rtl8127_oob_mutex_lock(tp); + my_cmd.data = rtl8127_ocp_read(tp, my_cmd.offset, 4); + rtl8127_oob_mutex_unlock(tp); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + break; + + case RTL_WRITE_OOB_MAC: + if (my_cmd.len == 0 || my_cmd.len > 4) + return -EOPNOTSUPP; + + rtl8127_oob_mutex_lock(tp); + rtl8127_ocp_write(tp, my_cmd.offset, my_cmd.len, my_cmd.data); + rtl8127_oob_mutex_unlock(tp); + break; + + case RTL_ENABLE_PCI_DIAG: + tp->rtk_enable_diag = 1; + + dprintk("enable rtk diag\n"); + break; + + case RTL_DISABLE_PCI_DIAG: + tp->rtk_enable_diag = 0; + + dprintk("disable rtk diag\n"); + break; + + case RTL_READ_MAC_OCP: + if (my_cmd.offset % 2) + return -EOPNOTSUPP; + + my_cmd.data = rtl8127_mac_ocp_read(tp, my_cmd.offset); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + break; + + case RTL_WRITE_MAC_OCP: + if ((my_cmd.offset % 2) || (my_cmd.len != 2)) + return -EOPNOTSUPP; + + rtl8127_mac_ocp_write(tp, my_cmd.offset, (u16)my_cmd.data); + break; + + case RTL_DIRECT_READ_PHY_OCP: + my_cmd.data = rtl8127_mdio_prot_direct_read_phy_ocp(tp, my_cmd.offset); + if (copy_to_user(ifr->ifr_data, &my_cmd, sizeof(my_cmd))) { + ret = -EFAULT; + break; + } + + break; + + case RTL_DIRECT_WRITE_PHY_OCP: + rtl8127_mdio_prot_direct_write_phy_ocp(tp, my_cmd.offset, my_cmd.data); + break; + + default: + ret = -EOPNOTSUPP; + break; + } + + return ret; +} diff --git a/drivers/net/ethernet/realtek/r8127/src/rtltool.h b/drivers/net/ethernet/realtek/r8127/src/rtltool.h new file mode 100755 index 0000000000000..7b0ee22707be0 --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/src/rtltool.h @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ +*/ + +/************************************************************************************ + * This product is covered by one or more of the following patents: + * US6,570,884, US6,115,776, and US6,327,625. + ***********************************************************************************/ + +#ifndef _LINUX_RTLTOOL_H +#define _LINUX_RTLTOOL_H + +#define SIOCRTLTOOL SIOCDEVPRIVATE+1 + +enum rtl_cmd { + RTLTOOL_READ_MAC=0, + RTLTOOL_WRITE_MAC, + RTLTOOL_READ_PHY, + RTLTOOL_WRITE_PHY, + RTLTOOL_READ_EPHY, + RTLTOOL_WRITE_EPHY, + RTLTOOL_READ_ERI, + RTLTOOL_WRITE_ERI, + RTLTOOL_READ_PCI, + RTLTOOL_WRITE_PCI, + RTLTOOL_READ_EEPROM, + RTLTOOL_WRITE_EEPROM, + + RTL_READ_OOB_MAC, + RTL_WRITE_OOB_MAC, + + RTL_ENABLE_PCI_DIAG, + RTL_DISABLE_PCI_DIAG, + + RTL_READ_MAC_OCP, + RTL_WRITE_MAC_OCP, + + RTL_DIRECT_READ_PHY_OCP, + RTL_DIRECT_WRITE_PHY_OCP, + + RTLTOOL_INVALID +}; + +struct rtltool_cmd { + __u32 cmd; + __u32 offset; + __u32 len; + __u32 data; +}; + +enum mode_access { + MODE_NONE=0, + MODE_READ, + MODE_WRITE +}; + +#ifdef __KERNEL__ +int rtl8127_tool_ioctl(struct rtl8127_private *tp, struct ifreq *ifr); +#endif + +#endif /* _LINUX_RTLTOOL_H */ From 4645d83d1c72c5bedf92a9db0c8f491db8d23e0d Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Mon, 28 Apr 2025 15:45:04 +0000 Subject: [PATCH 040/464] NVIDIA: SAUCE: r8127: Remove Realtek r8127 non required files BugLink: https://bugs.launchpad.net/bugs/2109730 These files are not needed to build r8127 as part of kernel source code build, so removed these non required files. Signed-off-by: Abhishek Sahu Acked-by: Matt Ochs Acked-by: Carol L Soto Acked-by: Ian May Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Ian May (cherry picked from commit 063d338317508dbb456771ea6a7ec0b1f0b9ea6e noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit 712fc60b2f26 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit ee5f3b0a99180394cdd9afb196f2cd9453ffeaca noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/Makefile | 59 ------- drivers/net/ethernet/realtek/r8127/README | 147 ------------------ drivers/net/ethernet/realtek/r8127/autorun.sh | 101 ------------ .../realtek/r8127/src/Makefile_linux24x | 75 --------- 4 files changed, 382 deletions(-) delete mode 100755 drivers/net/ethernet/realtek/r8127/Makefile delete mode 100755 drivers/net/ethernet/realtek/r8127/README delete mode 100755 drivers/net/ethernet/realtek/r8127/autorun.sh delete mode 100755 drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x diff --git a/drivers/net/ethernet/realtek/r8127/Makefile b/drivers/net/ethernet/realtek/r8127/Makefile deleted file mode 100755 index 39e846ad3fc9f..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/Makefile +++ /dev/null @@ -1,59 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ - -################################################################################ -# This product is covered by one or more of the following patents: -# US6,570,884, US6,115,776, and US6,327,625. -################################################################################ - -KFLAG := 2$(shell uname -r | sed -ne 's/^2\.[4]\..*/4/p')x - -all: clean modules install - -modules: -ifeq ($(KFLAG),24x) - $(MAKE) -C src/ -f Makefile_linux24x modules -else - $(MAKE) -C src/ modules -endif - -clean: -ifeq ($(KFLAG),24x) - $(MAKE) -C src/ -f Makefile_linux24x clean -else - $(MAKE) -C src/ clean -endif - -install: -ifeq ($(KFLAG),24x) - $(MAKE) -C src/ -f Makefile_linux24x install -else - $(MAKE) -C src/ install -endif - - - diff --git a/drivers/net/ethernet/realtek/r8127/README b/drivers/net/ethernet/realtek/r8127/README deleted file mode 100755 index a2d451d938cab..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/README +++ /dev/null @@ -1,147 +0,0 @@ - - - This is the Linux device driver released for Realtek 5 Gigabit Ethernet controllers with PCI-Express interface. - - - - - Kernel source tree (supported Linux kernel 2.6.x and 2.4.x) - - For linux kernel 2.4.x, this driver supports 2.4.20 and latter. - - Compiler/binutils for kernel compilation - - - Unpack the tarball : - # tar vjxf r8127-11.aaa.bb.tar.bz2 - - Change to the directory: - # cd r8127-11.aaa.bb - - If you are running the target kernel, then you should be able to do : - - # ./autorun.sh (as root or with sudo) - - You can check whether the driver is loaded by using following commands. - - # lsmod | grep r8127 - # ifconfig -a - - If there is a device name, ethX, shown on the monitor, the linux - driver is loaded. Then, you can use the following command to activate - the ethX. - - # ifconfig ethX up - - ,where X=0,1,2,... - - - 1. Set manually - a. Set the IP address of your machine. - - # ifconfig ethX "the IP address of your machine" - - b. Set the IP address of DNS. - - Insert the following configuration in /etc/resolv.conf. - - nameserver "the IP address of DNS" - - c. Set the IP address of gateway. - - # route add default gw "the IP address of gateway" - - 2. Set by doing configurations in /etc/sysconfig/network-scripts - /ifcfg-ethX for Redhat and Fedora, or /etc/sysconfig/network - /ifcfg-ethX for SuSE. There are two examples to set network - configurations. - - a. Fixed IP address: - DEVICE=eth0 - BOOTPROTO=static - ONBOOT=yes - TYPE=ethernet - NETMASK=255.255.255.0 - IPADDR=192.168.1.1 - GATEWAY=192.168.1.254 - BROADCAST=192.168.1.255 - - b. DHCP: - DEVICE=eth0 - BOOTPROTO=dhcp - ONBOOT=yes - - - There are two ways to modify the MAC address of the NIC. - 1. Use ifconfig: - - # ifconfig ethX hw ether YY:YY:YY:YY:YY:YY - - ,where X is the device number assigned by Linux kernel, and - YY:YY:YY:YY:YY:YY is the MAC address assigned by the user. - - 2. Use ip: - - # ip link set ethX address YY:YY:YY:YY:YY:YY - - ,where X is the device number assigned by Linux kernel, and - YY:YY:YY:YY:YY:YY is the MAC address assigned by the user. - - - - 1. Force the link status when insert the driver. - - If the user is in the path ~/r8127, the link status can be forced - to one of the 5 modes as following command. - - # insmod ./src/r8127.ko speed=SPEED_MODE duplex=DUPLEX_MODE autoneg=NWAY_OPTION - - ,where - SPEED_MODE = 1000 for 1000Mbps - = 100 for 100Mbps - = 10 for 10Mbps - DUPLEX_MODE = 0 for half-duplex - = 1 for full-duplex - NWAY_OPTION = 0 for auto-negotiation off (true force) - = 1 for auto-negotiation on (nway force) - For example: - - # insmod ./src/r8127.ko speed=100 duplex=0 autoneg=1 - - will force PHY to operate in 100Mpbs Half-duplex(nway force). - - 2. Force the link status by using ethtool. - a. Insert the driver first. - b. Make sure that ethtool exists in /sbin. - c. Force the link status as the following command. - - 2.5G before kernel v4.10 - # ethtool -s eth0 autoneg on advertise 0x802f - - 2.5G for kernel v4.10 and later - # ethtool -s eth0 autoneg on advertise 0x80000000002f - - 5G for kernel v4.10 and later (Couldn't be supported before kernel v4.10) - # ethtool -s eth0 autoneg on advertise 0x180000000002f - - # ethtool -s eth0 autoneg on advertise 0x1000 (10G) - # ethtool -s eth0 autoneg on advertise 0x002f (1G) - # ethtool -s eth0 autoneg on advertise 0x000f (100M full) - # ethtool -s eth0 autoneg on advertise 0x0003 (10M full) - - - Transmitting Jumbo Frames, whose packet size is bigger than 1500 bytes, please change mtu by the following command. - - # ifconfig ethX mtu MTU - - , where X=0,1,2,..., and MTU is configured by user. - - RTL8127 supports Jumbo Frame size up to 9 kBytes. - - - Get/Set device EEE status - - Get EEE device status - # ethtool --show-eee enp1s0 - - Set EEE device status - # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x0008 (100M full) - # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x0020 (1G) - # ethtool --set-eee enp1s0 eee on tx-lpi on tx-timer 1546 advertise 0x8000 (2.5G) diff --git a/drivers/net/ethernet/realtek/r8127/autorun.sh b/drivers/net/ethernet/realtek/r8127/autorun.sh deleted file mode 100755 index fd87bced11583..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/autorun.sh +++ /dev/null @@ -1,101 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0-only - -# invoke insmod with all arguments we got -# and use a pathname, as insmod doesn't look in . by default - -TARGET_PATH=$(find /lib/modules/$(uname -r)/kernel/drivers/net/ethernet -name realtek -type d) -if [ "$TARGET_PATH" = "" ]; then - TARGET_PATH=$(find /lib/modules/$(uname -r)/kernel/drivers/net -name realtek -type d) -fi -if [ "$TARGET_PATH" = "" ]; then - TARGET_PATH=/lib/modules/$(uname -r)/kernel/drivers/net -fi -echo -echo "Check old driver and unload it." -check=`lsmod | grep r8169` -if [ "$check" != "" ]; then - echo "rmmod r8169" - /sbin/rmmod r8169 -fi - -check=`lsmod | grep r8127` -if [ "$check" != "" ]; then - echo "rmmod r8127" - /sbin/rmmod r8127 -fi - -echo "Build the module and install" -echo "-------------------------------" >> log.txt -date 1>>log.txt -make $@ all 1>>log.txt || exit 1 -module=`ls src/*.ko` -module=${module#src/} -module=${module%.ko} - -if [ "$module" = "" ]; then - echo "No driver exists!!!" - exit 1 -elif [ "$module" != "r8169" ]; then - if test -e $TARGET_PATH/r8169.ko ; then - echo "Backup r8169.ko" - if test -e $TARGET_PATH/r8169.bak ; then - i=0 - while test -e $TARGET_PATH/r8169.bak$i - do - i=$(($i+1)) - done - echo "rename r8169.ko to r8169.bak$i" - mv $TARGET_PATH/r8169.ko $TARGET_PATH/r8169.bak$i - else - echo "rename r8169.ko to r8169.bak" - mv $TARGET_PATH/r8169.ko $TARGET_PATH/r8169.bak - fi - fi - if test -e $TARGET_PATH/r8169.ko.zst ; then - echo "Backup r8169.ko.zst" - if test -e $TARGET_PATH/r8169.zst.bak ; then - i=0 - while test -e $TARGET_PATH/r8169.zst.bak$i - do - i=$(($i+1)) - done - echo "rename r8169.ko.zst to r8169.zst.bak$i" - mv $TARGET_PATH/r8169.ko.zst $TARGET_PATH/r8169.zst.bak$i - else - echo "rename r8169.ko.zst to r8169.zst.bak" - mv $TARGET_PATH/r8169.ko.zst $TARGET_PATH/r8169.zst.bak - fi - fi -fi - -echo "DEPMOD $(uname -r)" -depmod `uname -r` -echo "load module $module" -modprobe $module - -is_update_initramfs=n -distrib_list="ubuntu debian" - -if [ -r /etc/debian_version ]; then - is_update_initramfs=y -elif [ -r /etc/lsb-release ]; then - for distrib in $distrib_list - do - /bin/grep -i "$distrib" /etc/lsb-release 2>&1 /dev/null && \ - is_update_initramfs=y && break - done -fi - -if [ "$is_update_initramfs" = "y" ]; then - if which update-initramfs >/dev/null ; then - echo "Updating initramfs. Please wait." - update-initramfs -u -k $(uname -r) - else - echo "update-initramfs: command not found" - exit 1 - fi -fi - -echo "Completed." -exit 0 diff --git a/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x b/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x deleted file mode 100755 index 7cb3d91a85a64..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/src/Makefile_linux24x +++ /dev/null @@ -1,75 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ - -################################################################################ -# This product is covered by one or more of the following patents: -# US6,570,884, US6,115,776, and US6,327,625. -################################################################################ - -CC := gcc -LD := ld -ARCH := $(shell uname -m | sed 's/i.86/i386/') -KSRC := /lib/modules/$(shell uname -r)/build -CONFIG_FILE := $(KSRC)/include/linux/autoconf.h -KMISC := /lib/modules/$(shell uname -r)/kernel/drivers/net/ - - -ifeq ($(ARCH),x86_64) - MODCFLAGS += -mcmodel=kernel -mno-red-zone -endif - -#standard flags for module builds -MODCFLAGS += -DLINUX -D__KERNEL__ -DMODULE -O2 -pipe -Wall -MODCFLAGS += -I$(KSRC)/include -I. -MODCFLAGS += -DMODVERSIONS -DEXPORT_SYMTAB -include $(KSRC)/include/linux/modversions.h -SOURCE := r8127_n.c rtl_eeprom.c rtltool.c -OBJS := $(SOURCE:.c=.o) - - -SMP := $(shell $(CC) $(MODCFLAGS) -E -dM $(CONFIG_FILE) | \ - grep CONFIG_SMP | awk '{print $$3}') - -ifneq ($(SMP),1) - SMP := 0 -endif - -ifeq ($(SMP),1) - MODCFLAGS += -D__SMP__ -endif - -modules: $(OBJS) - $(LD) -r $^ -o r8127.o - strip --strip-debug r8127.o - -%.o: %.c - $(CC) $(MODCFLAGS) -c $< -o $@ - -clean: - rm *.o -f - -install: - install -m 744 -c r8127.o $(KMISC) From b348d966e1941116881af9346b5735547fa98775 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Mon, 28 Apr 2025 15:45:44 +0000 Subject: [PATCH 041/464] NVIDIA: SAUCE: r8127: Moved files from r8127/src to r8127 folder BugLink: https://bugs.launchpad.net/bugs/2109730 This commit moved all files from src folder to parent folder itself. Signed-off-by: Abhishek Sahu Acked-by: Matt Ochs Acked-by: Carol L Soto Acked-by: Ian May Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Ian May (cherry picked from commit a5fe39b0572298d86a23047fa8d9e73ae012c3dc noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit 1802cd38abe2 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit f83397fd097642788c53fc208b4226f987312351 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/{src => }/Makefile | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_dash.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_firmware.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_firmware.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_n.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_ptp.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_ptp.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_realwow.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_rss.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/r8127_rss.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/rtl_eeprom.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/rtl_eeprom.h | 0 drivers/net/ethernet/realtek/r8127/{src => }/rtltool.c | 0 drivers/net/ethernet/realtek/r8127/{src => }/rtltool.h | 0 15 files changed, 0 insertions(+), 0 deletions(-) rename drivers/net/ethernet/realtek/r8127/{src => }/Makefile (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_dash.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_firmware.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_firmware.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_n.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_ptp.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_ptp.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_realwow.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_rss.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/r8127_rss.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/rtl_eeprom.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/rtl_eeprom.h (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/rtltool.c (100%) rename drivers/net/ethernet/realtek/r8127/{src => }/rtltool.h (100%) diff --git a/drivers/net/ethernet/realtek/r8127/src/Makefile b/drivers/net/ethernet/realtek/r8127/Makefile similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/Makefile rename to drivers/net/ethernet/realtek/r8127/Makefile diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127.h b/drivers/net/ethernet/realtek/r8127/r8127.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127.h rename to drivers/net/ethernet/realtek/r8127/r8127.h diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_dash.h b/drivers/net/ethernet/realtek/r8127/r8127_dash.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_dash.h rename to drivers/net/ethernet/realtek/r8127/r8127_dash.h diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c b/drivers/net/ethernet/realtek/r8127/r8127_firmware.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_firmware.c rename to drivers/net/ethernet/realtek/r8127/r8127_firmware.c diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h b/drivers/net/ethernet/realtek/r8127/r8127_firmware.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_firmware.h rename to drivers/net/ethernet/realtek/r8127/r8127_firmware.h diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_n.c rename to drivers/net/ethernet/realtek/r8127/r8127_n.c diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c b/drivers/net/ethernet/realtek/r8127/r8127_ptp.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_ptp.c rename to drivers/net/ethernet/realtek/r8127/r8127_ptp.c diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h b/drivers/net/ethernet/realtek/r8127/r8127_ptp.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_ptp.h rename to drivers/net/ethernet/realtek/r8127/r8127_ptp.h diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h b/drivers/net/ethernet/realtek/r8127/r8127_realwow.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_realwow.h rename to drivers/net/ethernet/realtek/r8127/r8127_realwow.h diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_rss.c b/drivers/net/ethernet/realtek/r8127/r8127_rss.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_rss.c rename to drivers/net/ethernet/realtek/r8127/r8127_rss.c diff --git a/drivers/net/ethernet/realtek/r8127/src/r8127_rss.h b/drivers/net/ethernet/realtek/r8127/r8127_rss.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/r8127_rss.h rename to drivers/net/ethernet/realtek/r8127/r8127_rss.h diff --git a/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c b/drivers/net/ethernet/realtek/r8127/rtl_eeprom.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.c rename to drivers/net/ethernet/realtek/r8127/rtl_eeprom.c diff --git a/drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h b/drivers/net/ethernet/realtek/r8127/rtl_eeprom.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/rtl_eeprom.h rename to drivers/net/ethernet/realtek/r8127/rtl_eeprom.h diff --git a/drivers/net/ethernet/realtek/r8127/src/rtltool.c b/drivers/net/ethernet/realtek/r8127/rtltool.c similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/rtltool.c rename to drivers/net/ethernet/realtek/r8127/rtltool.c diff --git a/drivers/net/ethernet/realtek/r8127/src/rtltool.h b/drivers/net/ethernet/realtek/r8127/rtltool.h similarity index 100% rename from drivers/net/ethernet/realtek/r8127/src/rtltool.h rename to drivers/net/ethernet/realtek/r8127/rtltool.h From acd7683eb387d2f191e7a46c63c6eb885fd1e6d7 Mon Sep 17 00:00:00 2001 From: tbergstrom Date: Tue, 22 Apr 2025 13:26:49 -0700 Subject: [PATCH 042/464] NVIDIA: SAUCE: Add r8127 in kernel build BugLink: https://bugs.launchpad.net/bugs/2109730 In the original code, r8127 driver was build as out of tree module. This commit adds Kconfig and updates Makefile for building it with kernel build. r8127 driver internally uses different config flags and these are set through EXTRA_CFLAGS. These config flags are now set in the Makefile with ccflags-y. All the flags, that were getting enabled by default in the original code, have been enabled in ccflags-y. This commit is not enabling any extra flags. Some of the files compilation are dependent upon a particular flag. Now, only default flags are set, so these files will become unused, This commit has removed these files. Signed-off-by: Terje Bergstrom Signed-off-by: Abhishek Sahu Acked-by: Matt Ochs Acked-by: Carol L Soto Acked-by: Ian May Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Ian May (backported from commit 04ea6d025e87e2d9b8f88541236ebdd116a15d95 noble:linux-nvidia-6.11) [jacobmartin: adjust context around RTASE definitions introduced in K6.14] Signed-off-by: Jacob Martin (cherry picked from commit 6217feae8e42 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit d423ea701ce8f15137682f6b33118bebf1290964 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/Kconfig | 2 + drivers/net/ethernet/realtek/Makefile | 1 + drivers/net/ethernet/realtek/r8127/Kconfig | 42 + drivers/net/ethernet/realtek/r8127/Makefile | 180 +--- .../ethernet/realtek/r8127/r8127_firmware.c | 264 ----- .../net/ethernet/realtek/r8127/r8127_ptp.c | 944 ------------------ .../net/ethernet/realtek/r8127/r8127_ptp.h | 202 ---- .../net/ethernet/realtek/r8127/r8127_rss.c | 583 ----------- 8 files changed, 48 insertions(+), 2170 deletions(-) create mode 100644 drivers/net/ethernet/realtek/r8127/Kconfig delete mode 100755 drivers/net/ethernet/realtek/r8127/r8127_firmware.c delete mode 100755 drivers/net/ethernet/realtek/r8127/r8127_ptp.c delete mode 100755 drivers/net/ethernet/realtek/r8127/r8127_ptp.h delete mode 100755 drivers/net/ethernet/realtek/r8127/r8127_rss.c diff --git a/drivers/net/ethernet/realtek/Kconfig b/drivers/net/ethernet/realtek/Kconfig index 9b0f4f9631dba..ca58b70f6f317 100644 --- a/drivers/net/ethernet/realtek/Kconfig +++ b/drivers/net/ethernet/realtek/Kconfig @@ -126,4 +126,6 @@ config RTASE To compile this driver as a module, choose M here: the module will be called rtase. This is recommended. +source "drivers/net/ethernet/realtek/r8127/Kconfig" + endif # NET_VENDOR_REALTEK diff --git a/drivers/net/ethernet/realtek/Makefile b/drivers/net/ethernet/realtek/Makefile index 12a9c399f40c6..4e1d21508e260 100644 --- a/drivers/net/ethernet/realtek/Makefile +++ b/drivers/net/ethernet/realtek/Makefile @@ -9,3 +9,4 @@ r8169-y += r8169_main.o r8169_firmware.o r8169_phy_config.o r8169-$(CONFIG_R8169_LEDS) += r8169_leds.o obj-$(CONFIG_R8169) += r8169.o obj-$(CONFIG_RTASE) += rtase/ +obj-$(CONFIG_R8127) += r8127/ diff --git a/drivers/net/ethernet/realtek/r8127/Kconfig b/drivers/net/ethernet/realtek/r8127/Kconfig new file mode 100644 index 0000000000000..e5a8e399390cd --- /dev/null +++ b/drivers/net/ethernet/realtek/r8127/Kconfig @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: GPL-2.0-only +################################################################################ +# +# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet +# controllers with PCI-Express interface. +# +# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 2 of the License, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, see . +# +# Author: +# Realtek NIC software team +# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan +# +################################################################################ + +################################################################################ +# This product is covered by one or more of the following patents: +# US6,570,884, US6,115,776, and US6,327,625. +################################################################################ + +config R8127 + tristate "RealTek RTL-8127 PCI 10 Gigabit Ethernet Adapter support" + depends on PCI + help + This is a driver for the 10 Gigabit Ethernet PCI network cards based on + the RTL-8127 chips. If you have one of those, say Y here. + + To compile this driver as a module, choose M here: the module + will be called r8127. This is recommended. + diff --git a/drivers/net/ethernet/realtek/r8127/Makefile b/drivers/net/ethernet/realtek/r8127/Makefile index d270904691bf3..d7cf5f65a9d8e 100755 --- a/drivers/net/ethernet/realtek/r8127/Makefile +++ b/drivers/net/ethernet/realtek/r8127/Makefile @@ -30,180 +30,6 @@ # US6,570,884, US6,115,776, and US6,327,625. ################################################################################ -CONFIG_SOC_LAN = y -ENABLE_REALWOW_SUPPORT = n -ENABLE_DASH_SUPPORT = n -ENABLE_DASH_PRINTER_SUPPORT = n -CONFIG_DOWN_SPEED_100 = n -CONFIG_ASPM = y -ENABLE_S5WOL = y -ENABLE_S5_KEEP_CURR_MAC = n -ENABLE_EEE = y -ENABLE_S0_MAGIC_PACKET = n -ENABLE_TX_NO_CLOSE = y -ENABLE_MULTIPLE_TX_QUEUE = n -ENABLE_PTP_SUPPORT = n -ENABLE_RSS_SUPPORT = n -ENABLE_LIB_SUPPORT = n -ENABLE_USE_FIRMWARE_FILE = n -DISABLE_WOL_SUPPORT = n -DISABLE_MULTI_MSIX_VECTOR = n -ENABLE_DOUBLE_VLAN = n -ENABLE_PAGE_REUSE = n -ENABLE_RX_PACKET_FRAGMENT = n -ENABLE_GIGA_LITE = y - -ifneq ($(KERNELRELEASE),) - obj-m := r8127.o - r8127-objs := r8127_n.o rtl_eeprom.o rtltool.o - ifeq ($(CONFIG_SOC_LAN), y) - EXTRA_CFLAGS += -DCONFIG_SOC_LAN - endif - ifeq ($(ENABLE_REALWOW_SUPPORT), y) - r8127-objs += r8127_realwow.o - EXTRA_CFLAGS += -DENABLE_REALWOW_SUPPORT - endif - ifeq ($(ENABLE_DASH_SUPPORT), y) - r8127-objs += r8127_dash.o - EXTRA_CFLAGS += -DENABLE_DASH_SUPPORT - endif - ifeq ($(ENABLE_DASH_PRINTER_SUPPORT), y) - r8127-objs += r8127_dash.o - EXTRA_CFLAGS += -DENABLE_DASH_SUPPORT -DENABLE_DASH_PRINTER_SUPPORT - endif - EXTRA_CFLAGS += -DCONFIG_R8127_NAPI - EXTRA_CFLAGS += -DCONFIG_R8127_VLAN - ifeq ($(CONFIG_DOWN_SPEED_100), y) - EXTRA_CFLAGS += -DCONFIG_DOWN_SPEED_100 - endif - ifeq ($(CONFIG_ASPM), y) - EXTRA_CFLAGS += -DCONFIG_ASPM - endif - ifeq ($(ENABLE_S5WOL), y) - EXTRA_CFLAGS += -DENABLE_S5WOL - endif - ifeq ($(ENABLE_S5_KEEP_CURR_MAC), y) - EXTRA_CFLAGS += -DENABLE_S5_KEEP_CURR_MAC - endif - ifeq ($(ENABLE_EEE), y) - EXTRA_CFLAGS += -DENABLE_EEE - endif - ifeq ($(ENABLE_S0_MAGIC_PACKET), y) - EXTRA_CFLAGS += -DENABLE_S0_MAGIC_PACKET - endif - ifeq ($(ENABLE_TX_NO_CLOSE), y) - EXTRA_CFLAGS += -DENABLE_TX_NO_CLOSE - endif - ifeq ($(ENABLE_MULTIPLE_TX_QUEUE), y) - EXTRA_CFLAGS += -DENABLE_MULTIPLE_TX_QUEUE - endif - ifeq ($(ENABLE_PTP_SUPPORT), y) - r8127-objs += r8127_ptp.o - EXTRA_CFLAGS += -DENABLE_PTP_SUPPORT - endif - ifeq ($(ENABLE_RSS_SUPPORT), y) - r8127-objs += r8127_rss.o - EXTRA_CFLAGS += -DENABLE_RSS_SUPPORT - endif - ifeq ($(ENABLE_LIB_SUPPORT), y) - r8127-objs += r8127_lib.o - EXTRA_CFLAGS += -DENABLE_LIB_SUPPORT - endif - ifeq ($(ENABLE_USE_FIRMWARE_FILE), y) - r8127-objs += r8127_firmware.o - EXTRA_CFLAGS += -DENABLE_USE_FIRMWARE_FILE - endif - ifeq ($(DISABLE_WOL_SUPPORT), y) - EXTRA_CFLAGS += -DDISABLE_WOL_SUPPORT - endif - ifeq ($(DISABLE_MULTI_MSIX_VECTOR), y) - EXTRA_CFLAGS += -DDISABLE_MULTI_MSIX_VECTOR - endif - ifeq ($(ENABLE_DOUBLE_VLAN), y) - EXTRA_CFLAGS += -DENABLE_DOUBLE_VLAN - endif - ifeq ($(ENABLE_PAGE_REUSE), y) - EXTRA_CFLAGS += -DENABLE_PAGE_REUSE - endif - ifeq ($(ENABLE_RX_PACKET_FRAGMENT), y) - EXTRA_CFLAGS += -DENABLE_RX_PACKET_FRAGMENT - endif - ifeq ($(ENABLE_GIGA_LITE), y) - EXTRA_CFLAGS += -DENABLE_GIGA_LITE - endif -else - BASEDIR := /lib/modules/$(shell uname -r) - KERNELDIR ?= $(BASEDIR)/build - PWD :=$(shell pwd) - DRIVERDIR := $(shell find $(BASEDIR)/kernel/drivers/net/ethernet -name realtek -type d) - ifeq ($(DRIVERDIR),) - DRIVERDIR := $(shell find $(BASEDIR)/kernel/drivers/net -name realtek -type d) - endif - ifeq ($(DRIVERDIR),) - DRIVERDIR := $(BASEDIR)/kernel/drivers/net - endif - RTKDIR := $(subst $(BASEDIR)/,,$(DRIVERDIR)) - - KERNEL_GCC_VERSION := $(shell cat /proc/version | sed -n 's/.*gcc version \([[:digit:]]\.[[:digit:]]\.[[:digit:]]\).*/\1/p') - CCVERSION = $(shell $(CC) -dumpversion) - - KVER = $(shell uname -r) - KMAJ = $(shell echo $(KVER) | \ - sed -e 's/^\([0-9][0-9]*\)\.[0-9][0-9]*\.[0-9][0-9]*.*/\1/') - KMIN = $(shell echo $(KVER) | \ - sed -e 's/^[0-9][0-9]*\.\([0-9][0-9]*\)\.[0-9][0-9]*.*/\1/') - KREV = $(shell echo $(KVER) | \ - sed -e 's/^[0-9][0-9]*\.[0-9][0-9]*\.\([0-9][0-9]*\).*/\1/') - - kver_ge = $(shell \ - echo test | awk '{if($(KMAJ) < $(1)) {print 0} else { \ - if($(KMAJ) > $(1)) {print 1} else { \ - if($(KMIN) < $(2)) {print 0} else { \ - if($(KMIN) > $(2)) {print 1} else { \ - if($(KREV) < $(3)) {print 0} else { print 1 } \ - }}}}}' \ - ) - -.PHONY: all -all: print_vars clean modules install - -print_vars: - @echo - @echo "CC: " $(CC) - @echo "CCVERSION: " $(CCVERSION) - @echo "KERNEL_GCC_VERSION: " $(KERNEL_GCC_VERSION) - @echo "KVER: " $(KVER) - @echo "KMAJ: " $(KMAJ) - @echo "KMIN: " $(KMIN) - @echo "KREV: " $(KREV) - @echo "BASEDIR: " $(BASEDIR) - @echo "DRIVERDIR: " $(DRIVERDIR) - @echo "PWD: " $(PWD) - @echo "RTKDIR: " $(RTKDIR) - @echo - -.PHONY:modules -modules: -#ifeq ($(call kver_ge,5,0,0),1) - $(MAKE) -C $(KERNELDIR) M=$(PWD) modules -#else -# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) modules -#endif - -.PHONY:clean -clean: -#ifeq ($(call kver_ge,5,0,0),1) - $(MAKE) -C $(KERNELDIR) M=$(PWD) clean -#else -# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) clean -#endif - -.PHONY:install -install: -#ifeq ($(call kver_ge,5,0,0),1) - $(MAKE) -C $(KERNELDIR) M=$(PWD) INSTALL_MOD_DIR=$(RTKDIR) modules_install -#else -# $(MAKE) -C $(KERNELDIR) SUBDIRS=$(PWD) INSTALL_MOD_DIR=$(RTKDIR) modules_install -#endif - -endif +ccflags-y += -DCONFIG_SOC_LAN -DCONFIG_R8127_NAPI -DCONFIG_R8127_VLAN -DCONFIG_ASPM -DENABLE_S5WOL -DENABLE_EEE -DENABLE_TX_NO_CLOSE -DENABLE_GIGA_LITE +obj-$(CONFIG_R8127) += r8127.o +r8127-y := r8127_n.o rtl_eeprom.o rtltool.o diff --git a/drivers/net/ethernet/realtek/r8127/r8127_firmware.c b/drivers/net/ethernet/realtek/r8127/r8127_firmware.c deleted file mode 100755 index 7ab59f641e77a..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/r8127_firmware.c +++ /dev/null @@ -1,264 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ -*/ - -/************************************************************************************ - * This product is covered by one or more of the following patents: - * US6,570,884, US6,115,776, and US6,327,625. - ***********************************************************************************/ - -#include -#include -#include - -#include "r8127_firmware.h" - -enum rtl_fw_opcode { - PHY_READ = 0x0, - PHY_DATA_OR = 0x1, - PHY_DATA_AND = 0x2, - PHY_BJMPN = 0x3, - PHY_MDIO_CHG = 0x4, - PHY_CLEAR_READCOUNT = 0x7, - PHY_WRITE = 0x8, - PHY_READCOUNT_EQ_SKIP = 0x9, - PHY_COMP_EQ_SKIPN = 0xa, - PHY_COMP_NEQ_SKIPN = 0xb, - PHY_WRITE_PREVIOUS = 0xc, - PHY_SKIPN = 0xd, - PHY_DELAY_MS = 0xe, -}; - -struct fw_info { - u32 magic; - char version[RTL8127_VER_SIZE]; - __le32 fw_start; - __le32 fw_len; - u8 chksum; -} __packed; - -#if LINUX_VERSION_CODE < KERNEL_VERSION(4,16,0) -#define sizeof_field(TYPE, MEMBER) sizeof((((TYPE *)0)->MEMBER)) -#endif -#define FW_OPCODE_SIZE sizeof_field(struct rtl8127_fw_phy_action, code[0]) - -static bool rtl8127_fw_format_ok(struct rtl8127_fw *rtl_fw) -{ - const struct firmware *fw = rtl_fw->fw; - struct fw_info *fw_info = (struct fw_info *)fw->data; - struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; - - if (fw->size < FW_OPCODE_SIZE) - return false; - - if (!fw_info->magic) { - size_t i, size, start; - u8 checksum = 0; - - if (fw->size < sizeof(*fw_info)) - return false; - - for (i = 0; i < fw->size; i++) - checksum += fw->data[i]; - if (checksum != 0) - return false; - - start = le32_to_cpu(fw_info->fw_start); - if (start > fw->size) - return false; - - size = le32_to_cpu(fw_info->fw_len); - if (size > (fw->size - start) / FW_OPCODE_SIZE) - return false; - - strscpy(rtl_fw->version, fw_info->version, RTL8127_VER_SIZE); - - pa->code = (__le32 *)(fw->data + start); - pa->size = size; - } else { - if (fw->size % FW_OPCODE_SIZE) - return false; - - strscpy(rtl_fw->version, rtl_fw->fw_name, RTL8127_VER_SIZE); - - pa->code = (__le32 *)fw->data; - pa->size = fw->size / FW_OPCODE_SIZE; - } - - return true; -} - -static bool rtl8127_fw_data_ok(struct rtl8127_fw *rtl_fw) -{ - struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; - size_t index; - - for (index = 0; index < pa->size; index++) { - u32 action = le32_to_cpu(pa->code[index]); - u32 val = action & 0x0000ffff; - u32 regno = (action & 0x0fff0000) >> 16; - - switch (action >> 28) { - case PHY_READ: - case PHY_DATA_OR: - case PHY_DATA_AND: - case PHY_CLEAR_READCOUNT: - case PHY_WRITE: - case PHY_WRITE_PREVIOUS: - case PHY_DELAY_MS: - break; - - case PHY_MDIO_CHG: - if (val > 1) - goto out; - break; - - case PHY_BJMPN: - if (regno > index) - goto out; - break; - case PHY_READCOUNT_EQ_SKIP: - if (index + 2 >= pa->size) - goto out; - break; - case PHY_COMP_EQ_SKIPN: - case PHY_COMP_NEQ_SKIPN: - case PHY_SKIPN: - if (index + 1 + regno >= pa->size) - goto out; - break; - - default: - dev_err(rtl_fw->dev, "Invalid action 0x%08x\n", action); - return false; - } - } - - return true; -out: - dev_err(rtl_fw->dev, "Out of range of firmware\n"); - return false; -} - -void rtl8127_fw_write_firmware(struct rtl8127_private *tp, struct rtl8127_fw *rtl_fw) -{ - struct rtl8127_fw_phy_action *pa = &rtl_fw->phy_action; - rtl8127_fw_write_t fw_write = rtl_fw->phy_write; - rtl8127_fw_read_t fw_read = rtl_fw->phy_read; - int predata = 0, count = 0; - size_t index; - - for (index = 0; index < pa->size; index++) { - u32 action = le32_to_cpu(pa->code[index]); - u32 data = action & 0x0000ffff; - u32 regno = (action & 0x0fff0000) >> 16; - enum rtl_fw_opcode opcode = action >> 28; - - if (!action) - break; - - switch (opcode) { - case PHY_READ: - predata = fw_read(tp, regno); - count++; - break; - case PHY_DATA_OR: - predata |= data; - break; - case PHY_DATA_AND: - predata &= data; - break; - case PHY_BJMPN: - index -= (regno + 1); - break; - case PHY_MDIO_CHG: - if (data) { - fw_write = rtl_fw->mac_mcu_write; - fw_read = rtl_fw->mac_mcu_read; - } else { - fw_write = rtl_fw->phy_write; - fw_read = rtl_fw->phy_read; - } - - break; - case PHY_CLEAR_READCOUNT: - count = 0; - break; - case PHY_WRITE: - fw_write(tp, regno, data); - break; - case PHY_READCOUNT_EQ_SKIP: - if (count == data) - index++; - break; - case PHY_COMP_EQ_SKIPN: - if (predata == data) - index += regno; - break; - case PHY_COMP_NEQ_SKIPN: - if (predata != data) - index += regno; - break; - case PHY_WRITE_PREVIOUS: - fw_write(tp, regno, predata); - break; - case PHY_SKIPN: - index += regno; - break; - case PHY_DELAY_MS: - mdelay(data); - break; - } - } -} - -void rtl8127_fw_release_firmware(struct rtl8127_fw *rtl_fw) -{ - release_firmware(rtl_fw->fw); -} - -int rtl8127_fw_request_firmware(struct rtl8127_fw *rtl_fw) -{ - int rc; - - rc = request_firmware(&rtl_fw->fw, rtl_fw->fw_name, rtl_fw->dev); - if (rc < 0) - goto out; - - if (!rtl8127_fw_format_ok(rtl_fw) || !rtl8127_fw_data_ok(rtl_fw)) { - release_firmware(rtl_fw->fw); - rc = -EINVAL; - goto out; - } - - return 0; -out: - dev_err(rtl_fw->dev, "Unable to load firmware %s (%d)\n", - rtl_fw->fw_name, rc); - return rc; -} diff --git a/drivers/net/ethernet/realtek/r8127/r8127_ptp.c b/drivers/net/ethernet/realtek/r8127/r8127_ptp.c deleted file mode 100755 index f3fd421625c0c..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/r8127_ptp.c +++ /dev/null @@ -1,944 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ -*/ - -/************************************************************************************ - * This product is covered by one or more of the following patents: - * US6,570,884, US6,115,776, and US6,327,625. - ***********************************************************************************/ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "r8127.h" -#include "r8127_ptp.h" - -static void rtl8127_wait_clkadj_ready(struct rtl8127_private *tp) -{ - int i; - - for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) - if (!(rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CLK_CFG_8126) & CLKADJ_MODE_SET)) - break; -} - -static void rtl8127_set_clkadj_mode(struct rtl8127_private *tp, u16 cmd) -{ - rtl8127_clear_and_set_eth_phy_ocp_bit(tp, - PTP_CLK_CFG_8126, - BIT_3 | BIT_2 | BIT_1, - CLKADJ_MODE_SET | cmd); - - rtl8127_wait_clkadj_ready(tp); -} - -static int _rtl8127_phc_gettime(struct rtl8127_private *tp, struct timespec64 *ts64) -{ - unsigned long flags; - - spin_lock_irqsave(&tp->phy_lock, flags); - - //Direct Read - rtl8127_set_clkadj_mode(tp, DIRECT_READ); - - /* nanoseconds */ - //Ns[29:16] E414[13:0] - ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_NS_HI_8126) & 0x3fff; - ts64->tv_nsec <<= 16; - //Ns[15:0] E412[15:0] - ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_NS_LO_8126); - - - /* seconds */ - //S[47:32] E41A[15:0] - ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_HI_8126); - ts64->tv_sec <<= 16; - //S[31:16] E418[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_MI_8126); - ts64->tv_sec <<= 16; - //S[15:0] E416[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_LO_8126); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - return 0; -} - -static int _rtl8127_phc_settime(struct rtl8127_private *tp, const struct timespec64 *ts64) -{ - unsigned long flags; - - spin_lock_irqsave(&tp->phy_lock, flags); - - /* nanoseconds */ - //Ns[15:0] E412[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, ts64->tv_nsec); - //Ns[29:16] E414[13:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (ts64->tv_nsec & 0x3fff0000) >> 16); - - /* seconds */ - //S[15:0] E416[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_LO_8126, ts64->tv_sec); - //S[31:16] E418[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_MI_8126, (ts64->tv_sec >> 16)); - //S[47:32] E41A[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_HI_8126, (ts64->tv_sec >> 32)); - - //Direct Write - rtl8127_set_clkadj_mode(tp, DIRECT_WRITE); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - return 0; -} - -static int _rtl8127_phc_adjtime(struct rtl8127_private *tp, s64 delta) -{ - unsigned long flags; - struct timespec64 d; - bool negative; - u64 tohw; - u32 nsec; - u64 sec; - - if (delta < 0) { - negative = true; - tohw = -delta; - } else { - negative = false; - tohw = delta; - } - - d = ns_to_timespec64(tohw); - - nsec = d.tv_nsec; - sec = d.tv_sec; - - nsec &= 0x3fffffff; - sec &= 0x0000ffffffffffff; - - spin_lock_irqsave(&tp->phy_lock, flags); - - /* nanoseconds */ - //Ns[15:0] E412[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, nsec); - //Ns[29:16] E414[13:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (nsec >> 16)); - - /* seconds */ - //S[15:0] E416[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_LO_8126, sec); - //S[31:16] E418[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_MI_8126, (sec >> 16)); - //S[47:32] E41A[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_S_HI_8126, (sec >> 32)); - - if (negative) - rtl8127_set_clkadj_mode(tp, DECREMENT_STEP); - else - rtl8127_set_clkadj_mode(tp, INCREMENT_STEP); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - return 0; -} - -static int rtl8127_phc_adjtime(struct ptp_clock_info *ptp, s64 delta) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - int ret; - - //netif_info(tp, drv, tp->dev, "phc adjust time\n"); - - ret = _rtl8127_phc_adjtime(tp, delta); - - return ret; -} - -/* - * delta = delta * 10^6 ppm = delta * 10^9 ppb (in this equation ppm and ppb are not variable) - * - * in adjfreq ppb is a variable - * ppb = delta * 10^9 - * delta = ppb / 10^9 - * rate_value = |delta| * 2^32 = |ppb| / 10^9 * 2^32 = (|ppb| << 32) / 10^9 - */ -static int _rtl8127_phc_adjfreq(struct ptp_clock_info *ptp, s32 ppb) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - unsigned long flags; - u32 rate_value; - - if (ppb < 0) { - rate_value = ((u64)-ppb << 32) / 1000000000; - rate_value = ~rate_value + 1; - } else - rate_value = ((u64)ppb << 32) / 1000000000; - - spin_lock_irqsave(&tp->phy_lock, flags); - - /* nanoseconds */ - //Ns[15:0] E412[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_LO_8126, rate_value); - //Ns[22:16] E414[13:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CFG_NS_HI_8126, (rate_value & 0x003f0000) >> 16); - - rtl8127_set_clkadj_mode(tp, RATE_WRITE); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - return 0; -} - -#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) -static int rtl8127_ptp_adjfine(struct ptp_clock_info *ptp, long scaled_ppm) -{ - s32 ppb = scaled_ppm_to_ppb(scaled_ppm); - - if (ppb > ptp->max_adj || ppb < -ptp->max_adj) - return -EINVAL; - - _rtl8127_phc_adjfreq(ptp, ppb); - - return 0; -} - -#else -static int rtl8127_phc_adjfreq(struct ptp_clock_info *ptp, s32 delta) -{ - //struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - - //netif_info(tp, drv, tp->dev, "phc adjust freq\n"); - - if (delta > ptp->max_adj || delta < -ptp->max_adj) - return -EINVAL; - - _rtl8127_phc_adjfreq(ptp, delta); - - return 0; -} -#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) */ - -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) -static int rtl8127_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts64, - struct ptp_system_timestamp *sts) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - int ret; - - //netif_info(tp, drv, tp->dev, "phc get ts\n"); - - ptp_read_system_prets(sts); - ret = _rtl8127_phc_gettime(tp, ts64); - ptp_read_system_postts(sts); - - return ret; -} -#else -static int rtl8127_phc_gettime(struct ptp_clock_info *ptp, struct timespec64 *ts64) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - int ret; - - //netif_info(tp, drv, tp->dev, "phc get ts\n"); - - ret = _rtl8127_phc_gettime(tp, ts64); - - return ret; -} -#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) */ - -static int rtl8127_phc_settime(struct ptp_clock_info *ptp, - const struct timespec64 *ts64) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - int ret; - - //netif_info(tp, drv, tp->dev, "phc set ts\n"); - - ret = _rtl8127_phc_settime(tp, ts64); - - return ret; -} - -static void _rtl8127_phc_enable(struct ptp_clock_info *ptp, - struct ptp_clock_request *rq, int on) -{ - struct rtl8127_private *tp = container_of(ptp, struct rtl8127_private, ptp_clock_info); - unsigned long flags; - u16 phy_ocp_data; - - if (on) { - tp->pps_enable = 1; - rtl8127_clear_mac_ocp_bit(tp, 0xDC00, BIT_6); - rtl8127_clear_mac_ocp_bit(tp, 0xDC20, BIT_1); - - spin_lock_irqsave(&tp->phy_lock, flags); - - /* Set periodic pulse 1pps */ - /* E432[8:0] = 0x017d */ - phy_ocp_data = rtl8127_mdio_direct_read_phy_ocp(tp, 0xE432); - phy_ocp_data &= 0xFE00; - phy_ocp_data |= 0x017d; - rtl8127_mdio_direct_write_phy_ocp(tp, 0xE432, phy_ocp_data); - - rtl8127_mdio_direct_write_phy_ocp(tp, 0xE434, 0x7840); - - /* E436[8:0] = 0xbe */ - phy_ocp_data = rtl8127_mdio_direct_read_phy_ocp(tp, 0xE436); - phy_ocp_data &= 0xFE00; - phy_ocp_data |= 0xbe; - rtl8127_mdio_direct_write_phy_ocp(tp, 0xE436, phy_ocp_data); - - rtl8127_mdio_direct_write_phy_ocp(tp, 0xE438, 0xbc20); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - /* start hrtimer */ - hrtimer_start(&tp->pps_timer, 1000000000, HRTIMER_MODE_REL); - } else - tp->pps_enable = 0; -} - -static int rtl8127_phc_enable(struct ptp_clock_info *ptp, - struct ptp_clock_request *rq, int on) -{ - switch (rq->type) { - case PTP_CLK_REQ_PPS: - _rtl8127_phc_enable(ptp, rq, on); - return 0; - default: - return -EOPNOTSUPP; - } -} - -static void rtl8127_ptp_enable_config(struct rtl8127_private *tp) -{ - if (tp->syncE_en) - rtl8127_set_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); - else - rtl8127_clear_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); - - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_CTL, PTP_CTL_TYPE_3 | BIT_12); - - rtl8127_set_eth_phy_ocp_bit(tp, 0xA640, BIT_15); -} - -int rtl8127_get_ts_info(struct net_device *netdev, - struct ethtool_ts_info *info) -{ - struct rtl8127_private *tp = netdev_priv(netdev); - - /* we always support timestamping disabled */ - info->rx_filters = BIT(HWTSTAMP_FILTER_NONE); - - if (tp->HwSuppPtpVer == 0) - return ethtool_op_get_ts_info(netdev, info); - - info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE | - SOF_TIMESTAMPING_RX_SOFTWARE | - SOF_TIMESTAMPING_SOFTWARE | - SOF_TIMESTAMPING_TX_HARDWARE | - SOF_TIMESTAMPING_RX_HARDWARE | - SOF_TIMESTAMPING_RAW_HARDWARE; - - if (tp->ptp_clock) - info->phc_index = ptp_clock_index(tp->ptp_clock); - else - info->phc_index = -1; - - info->tx_types = BIT(HWTSTAMP_TX_OFF) | BIT(HWTSTAMP_TX_ON); - - info->rx_filters = BIT(HWTSTAMP_FILTER_NONE) | - BIT(HWTSTAMP_FILTER_PTP_V2_EVENT) | - BIT(HWTSTAMP_FILTER_PTP_V2_L4_EVENT) | - BIT(HWTSTAMP_FILTER_PTP_V2_SYNC) | - BIT(HWTSTAMP_FILTER_PTP_V2_L4_SYNC) | - BIT(HWTSTAMP_FILTER_PTP_V2_DELAY_REQ) | - BIT(HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ); - - return 0; -} - -static const struct ptp_clock_info rtl_ptp_clock_info = { - .owner = THIS_MODULE, - .n_alarm = 0, - .n_ext_ts = 0, - .n_per_out = 0, - .n_pins = 0, - .pps = 1, -#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) - .adjfine = rtl8127_ptp_adjfine, -#else - .adjfreq = rtl8127_phc_adjfreq, -#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,2,0) */ - .adjtime = rtl8127_phc_adjtime, -#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) - .gettimex64 = rtl8127_phc_gettime, -#else - .gettime64 = rtl8127_phc_gettime, -#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(5,0,0) */ - - .settime64 = rtl8127_phc_settime, - .enable = rtl8127_phc_enable, -}; - -static u16 rtl8127_ptp_get_tx_msgtype(struct rtl8127_private *tp) -{ - u16 tx_ts_ready = 0; - int i; - - for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { - tx_ts_ready = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & 0xF000; - if (tx_ts_ready) - break; - } - - switch (tx_ts_ready) { - case TX_TS_PDLYRSP_RDY: - return PTP_MSGTYPE_PDELAY_RESP; - case TX_TS_PDLYREQ_RDY: - return PTP_MSGTYPE_PDELAY_REQ; - case TX_TS_DLYREQ_RDY: - return PTP_MSGTYPE_DELAY_REQ; - case TX_TS_SYNC_RDY: - default: - return PTP_MSGTYPE_SYNC; - } -} - -/* -static u16 rtl8127_ptp_get_rx_msgtype(struct rtl8127_private *tp) -{ - u16 rx_ts_ready = 0; - int i; - - for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) { - rx_ts_ready = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & 0x0F00; - if (rx_ts_ready) - break; - } - - switch (rx_ts_ready) { - case RX_TS_PDLYRSP_RDY: - return PTP_MSGTYPE_PDELAY_RESP; - case RX_TS_PDLYREQ_RDY: - return PTP_MSGTYPE_PDELAY_REQ; - case RX_TS_DLYREQ_RDY: - return PTP_MSGTYPE_DELAY_REQ; - case RX_TS_SYNC_RDY: - default: - return PTP_MSGTYPE_SYNC; - } -} -*/ - -static void rtl8127_wait_trx_ts_ready(struct rtl8127_private *tp) -{ - int i; - - for (i = 0; i < R8127_CHANNEL_WAIT_COUNT; i++) - if (!(rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_STA) & TRX_TS_RD)) - break; -} - -static void rtl8127_set_trx_ts_cmd(struct rtl8127_private *tp, u16 cmd) -{ - rtl8127_clear_and_set_eth_phy_ocp_bit(tp, - PTP_TRX_TS_STA, - TRXTS_SEL | BIT_3 | BIT_2, - TRX_TS_RD | cmd); - - rtl8127_wait_trx_ts_ready(tp); -} - -static void rtl8127_ptp_egresstime(struct rtl8127_private *tp, struct timespec64 *ts64) -{ - u16 msgtype; - - msgtype = rtl8127_ptp_get_tx_msgtype(tp); - - msgtype <<= 2; - - rtl8127_set_trx_ts_cmd(tp, (msgtype | BIT_4)); - - /* nanoseconds */ - //Ns[29:16] E448[13:0] - ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_HI) & 0x3fff; - ts64->tv_nsec <<= 16; - //Ns[15:0] E446[15:0] - ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_LO); - - /* seconds */ - //S[47:32] E44E[15:0] - ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_HI); - ts64->tv_sec <<= 16; - //S[31:16] E44C[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_MI); - ts64->tv_sec <<= 16; - //S[15:0] E44A[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_LO); -} - -static void rtl8127_ptp_ingresstime(struct rtl8127_private *tp, struct timespec64 *ts64, u8 type) -{ - u16 msgtype; - - switch (type) { - case PTP_MSGTYPE_PDELAY_RESP: - case PTP_MSGTYPE_PDELAY_REQ: - case PTP_MSGTYPE_DELAY_REQ: - case PTP_MSGTYPE_SYNC: - msgtype = type << 2; - break; - default: - return; - } - - rtl8127_set_trx_ts_cmd(tp, (TRXTS_SEL | msgtype | BIT_4)); - - /* nanoseconds */ - //Ns[29:16] E448[13:0] - ts64->tv_nsec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_HI) & 0x3fff; - ts64->tv_nsec <<= 16; - //Ns[15:0] E446[15:0] - ts64->tv_nsec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_NS_LO); - - /* seconds */ - //S[47:32] E44E[15:0] - ts64->tv_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_HI); - ts64->tv_sec <<= 16; - //S[31:16] E44C[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_MI); - ts64->tv_sec <<= 16; - //S[15:0] E44A[15:0] - ts64->tv_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_TRX_TS_S_LO); -} - -static void rtl8127_ptp_tx_hwtstamp(struct rtl8127_private *tp) -{ - struct sk_buff *skb = tp->ptp_tx_skb; - struct skb_shared_hwtstamps shhwtstamps = { 0 }; - struct timespec64 ts64; - - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, TX_TX_INTR); - - rtl8127_ptp_egresstime(tp, &ts64); - - /* Upper 32 bits contain s, lower 32 bits contain ns. */ - shhwtstamps.hwtstamp = ktime_set(ts64.tv_sec, - ts64.tv_nsec); - - /* Clear the lock early before calling skb_tstamp_tx so that - * applications are not woken up before the lock bit is clear. We use - * a copy of the skb pointer to ensure other threads can't change it - * while we're notifying the stack. - */ - tp->ptp_tx_skb = NULL; - clear_bit_unlock(__RTL8127_PTP_TX_IN_PROGRESS, &tp->state); - - /* Notify the stack and free the skb after we've unlocked */ - skb_tstamp_tx(skb, &shhwtstamps); - dev_kfree_skb_any(skb); -} - -#define RTL8127_PTP_TX_TIMEOUT (HZ * 15) -static void rtl8127_ptp_tx_work(struct work_struct *work) -{ - struct rtl8127_private *tp = container_of(work, struct rtl8127_private, - ptp_tx_work); - unsigned long flags; - - if (!tp->ptp_tx_skb) - return; - - if (time_is_before_jiffies(tp->ptp_tx_start + - RTL8127_PTP_TX_TIMEOUT)) { - dev_kfree_skb_any(tp->ptp_tx_skb); - tp->ptp_tx_skb = NULL; - clear_bit_unlock(__RTL8127_PTP_TX_IN_PROGRESS, &tp->state); - tp->tx_hwtstamp_timeouts++; - /* Clear the tx valid bit in TSYNCTXCTL register to enable - * interrupt - */ - spin_lock_irqsave(&tp->phy_lock, flags); - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, TX_TX_INTR); - spin_unlock_irqrestore(&tp->phy_lock, flags); - return; - } - - spin_lock_irqsave(&tp->phy_lock, flags); - if (rtl8127_mdio_direct_read_phy_ocp(tp, PTP_INSR) & TX_TX_INTR) { - rtl8127_ptp_tx_hwtstamp(tp); - spin_unlock_irqrestore(&tp->phy_lock, flags); - } else { - spin_unlock_irqrestore(&tp->phy_lock, flags); - /* reschedule to check later */ - schedule_work(&tp->ptp_tx_work); - } -} - -static int rtl8127_hwtstamp_enable(struct rtl8127_private *tp, bool enable) -{ - unsigned long flags; - - spin_lock_irqsave(&tp->phy_lock, flags); - - if (enable) { - //trx timestamp interrupt enable - rtl8127_set_eth_phy_ocp_bit(tp, PTP_INER, BIT_2 | BIT_3); - - //set isr clear mode - rtl8127_set_eth_phy_ocp_bit(tp, PTP_GEN_CFG, BIT_0); - - //clear ptp isr - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_INSR, 0xFFFF); - - //enable ptp - rtl8127_ptp_enable_config(tp); - - //rtl8127_set_local_time(tp); - } else { - /* trx timestamp interrupt disable */ - rtl8127_clear_eth_phy_ocp_bit(tp, PTP_INER, BIT_2 | BIT_3); - - /* disable ptp */ - rtl8127_clear_eth_phy_ocp_bit(tp, PTP_SYNCE_CTL, BIT_0); - rtl8127_clear_eth_phy_ocp_bit(tp, PTP_CTL, BIT_0); - rtl8127_set_eth_phy_ocp_bit(tp, 0xA640, BIT_15); - } - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - return 0; -} - -void rtl8127_set_local_time(struct rtl8127_private *tp) -{ - struct timespec64 ts64; - //set system time - ktime_get_real_ts64(&ts64); - _rtl8127_phc_settime(tp, &ts64); -} - -static long rtl8127_ptp_create_clock(struct rtl8127_private *tp) -{ - struct net_device *netdev = tp->dev; - long err; - - if (!IS_ERR_OR_NULL(tp->ptp_clock)) - return 0; - - if (tp->HwSuppPtpVer == 0) { - tp->ptp_clock = NULL; - return -EOPNOTSUPP; - } - - tp->ptp_clock_info = rtl_ptp_clock_info; - tp->ptp_clock_info.max_adj = 488281;//0x1FFFFF * 10^9 / 2^32 - - snprintf(tp->ptp_clock_info.name, sizeof(tp->ptp_clock_info.name), - "%pm", tp->dev->dev_addr); - tp->ptp_clock = ptp_clock_register(&tp->ptp_clock_info, &tp->pci_dev->dev); - if (IS_ERR(tp->ptp_clock)) { - err = PTR_ERR(tp->ptp_clock); - tp->ptp_clock = NULL; - netif_err(tp, drv, tp->dev, "ptp_clock_register failed\n"); - return err; - } else - netif_info(tp, drv, tp->dev, "registered PHC device on %s\n", netdev->name); - - return 0; -} - -static enum hrtimer_restart -rtl8127_hrtimer_for_pps(struct hrtimer *timer) { - struct rtl8127_private *tp = container_of(timer, struct rtl8127_private, pps_timer); - u16 tai_cfg = BIT_8 | BIT_3 | BIT_1 | BIT_0; - s64 pps_sec; - - if (tp->pps_enable) - { - unsigned long flags; - - spin_lock_irqsave(&tp->phy_lock, flags); - - //Direct Read - rtl8127_set_clkadj_mode(tp, DIRECT_READ); - - pps_sec = rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_HI_8126); - pps_sec <<= 16; - pps_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_MI_8126); - pps_sec <<= 16; - pps_sec |= rtl8127_mdio_direct_read_phy_ocp(tp, PTP_CFG_S_LO_8126); - pps_sec++; - - //E42A[15:0] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_TS_S_LO, pps_sec & 0xffff); - //E42C[31:16] - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_TS_S_HI, (pps_sec & 0xffff0000) >> 16); - //Periodic Tai start - rtl8127_mdio_direct_write_phy_ocp(tp, PTP_TAI_CFG, tai_cfg); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - hrtimer_forward_now(&tp->pps_timer, 1000000000); //rekick - return HRTIMER_RESTART; - } else - return HRTIMER_NORESTART; -} - -void rtl8127_ptp_reset(struct rtl8127_private *tp) -{ - if (!tp->ptp_clock) - return; - - netif_info(tp, drv, tp->dev, "reset PHC clock\n"); - - rtl8127_hwtstamp_enable(tp, false); -} - -void rtl8127_ptp_init(struct rtl8127_private *tp) -{ - /* obtain a PTP device, or re-use an existing device */ - if (rtl8127_ptp_create_clock(tp)) - return; - - /* we have a clock so we can initialize work now */ - INIT_WORK(&tp->ptp_tx_work, rtl8127_ptp_tx_work); - - /* init a hrtimer for pps */ - tp->pps_enable = 0; - hrtimer_init(&tp->pps_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL); - tp->pps_timer.function = rtl8127_hrtimer_for_pps; - - /* reset the PTP related hardware bits */ - rtl8127_ptp_reset(tp); - - return; -} - -void rtl8127_ptp_suspend(struct rtl8127_private *tp) -{ - if (!tp->ptp_clock) - return; - - netif_info(tp, drv, tp->dev, "suspend PHC clock\n"); - - rtl8127_hwtstamp_enable(tp, false); - - /* ensure that we cancel any pending PTP Tx work item in progress */ - cancel_work_sync(&tp->ptp_tx_work); - - hrtimer_cancel(&tp->pps_timer); -} - -void rtl8127_ptp_stop(struct rtl8127_private *tp) -{ - struct net_device *netdev = tp->dev; - - netif_info(tp, drv, tp->dev, "stop PHC clock\n"); - - /* first, suspend PTP activity */ - rtl8127_ptp_suspend(tp); - - /* disable the PTP clock device */ - if (tp->ptp_clock) { - ptp_clock_unregister(tp->ptp_clock); - tp->ptp_clock = NULL; - netif_info(tp, drv, tp->dev, "removed PHC on %s\n", - netdev->name); - } -} - -static int rtl8127_set_tstamp(struct net_device *netdev, struct ifreq *ifr) -{ - struct rtl8127_private *tp = netdev_priv(netdev); - struct hwtstamp_config config; - bool hwtstamp = 0; - - //netif_info(tp, drv, tp->dev, "ptp set ts\n"); - - if (copy_from_user(&config, ifr->ifr_data, sizeof(config))) - return -EFAULT; - - if (config.flags) - return -EINVAL; - - switch (config.tx_type) { - case HWTSTAMP_TX_ON: - hwtstamp = 1; - break; - case HWTSTAMP_TX_OFF: - break; - case HWTSTAMP_TX_ONESTEP_SYNC: - default: - return -ERANGE; - } - - switch (config.rx_filter) { - case HWTSTAMP_FILTER_PTP_V2_EVENT: - case HWTSTAMP_FILTER_PTP_V2_L2_EVENT: - case HWTSTAMP_FILTER_PTP_V2_L4_EVENT: - case HWTSTAMP_FILTER_PTP_V2_SYNC: - case HWTSTAMP_FILTER_PTP_V2_L2_SYNC: - case HWTSTAMP_FILTER_PTP_V2_L4_SYNC: - case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ: - case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ: - case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: - config.rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT; - hwtstamp = 1; - tp->flags |= RTL_FLAG_RX_HWTSTAMP_ENABLED; - break; - case HWTSTAMP_FILTER_NONE: - tp->flags &= ~RTL_FLAG_RX_HWTSTAMP_ENABLED; - break; - default: - tp->flags &= ~RTL_FLAG_RX_HWTSTAMP_ENABLED; - return -ERANGE; - } - - if (tp->hwtstamp_config.tx_type != config.tx_type || - tp->hwtstamp_config.rx_filter != config.rx_filter) { - tp->hwtstamp_config = config; - - rtl8127_hwtstamp_enable(tp, hwtstamp); - } - - return copy_to_user(ifr->ifr_data, &config, - sizeof(config)) ? -EFAULT : 0; -} - -static int rtl8127_get_tstamp(struct net_device *netdev, struct ifreq *ifr) -{ - struct rtl8127_private *tp = netdev_priv(netdev); - - //netif_info(tp, drv, tp->dev, "ptp get ts\n"); - - return copy_to_user(ifr->ifr_data, &tp->hwtstamp_config, - sizeof(tp->hwtstamp_config)) ? -EFAULT : 0; -} - -int rtl8127_ptp_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd) -{ - int ret; - - //netif_info(tp, drv, tp->dev, "ptp ioctl\n"); - - switch (cmd) { -#ifdef ENABLE_PTP_SUPPORT - case SIOCSHWTSTAMP: - ret = rtl8127_set_tstamp(netdev, ifr); - break; - case SIOCGHWTSTAMP: - ret = rtl8127_get_tstamp(netdev, ifr); - break; -#endif - default: - ret = -EOPNOTSUPP; - break; - } - - return ret; -} - -static void rtl8127_rx_ptp_pktstamp(struct rtl8127_private *tp, struct sk_buff *skb, u8 type) -{ - struct timespec64 ts64; - unsigned long flags; - - spin_lock_irqsave(&tp->phy_lock, flags); - - rtl8127_ptp_ingresstime(tp, &ts64, type); - - spin_unlock_irqrestore(&tp->phy_lock, flags); - - skb_hwtstamps(skb)->hwtstamp = ktime_set(ts64.tv_sec, ts64.tv_nsec); - - return; -} - -void rtl8127_rx_ptp_timestamp(struct rtl8127_private *tp, struct sk_buff *skb) -{ - unsigned int ptp_class; - struct ptp_header *hdr; - u8 msgtype; - - ptp_class = ptp_classify_raw(skb); - if (ptp_class == PTP_CLASS_NONE) - return; - - skb_reset_mac_header(skb); - hdr = ptp_parse_header(skb, ptp_class); - if (unlikely(!hdr)) - return; - - msgtype = ptp_get_msgtype(hdr, ptp_class); - rtl8127_rx_ptp_pktstamp(tp, skb, msgtype); - - return; -} - -#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) -struct ptp_header *ptp_parse_header(struct sk_buff *skb, unsigned int type) -{ - u8 *ptr = skb_mac_header(skb); - - if (type & PTP_CLASS_VLAN) - //ptr += VLAN_HLEN; - ptr += 4; - - switch (type & PTP_CLASS_PMASK) { - case PTP_CLASS_IPV4: - ptr += IPV4_HLEN(ptr) + UDP_HLEN; - break; - case PTP_CLASS_IPV6: - ptr += IP6_HLEN + UDP_HLEN; - break; - case PTP_CLASS_L2: - break; - default: - return NULL; - } - - ptr += ETH_HLEN; - - /* Ensure that the entire header is present in this packet. */ - if (ptr + sizeof(struct ptp_header) > skb->data + skb->len) - return NULL; - - return (struct ptp_header *)ptr; -} -#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) */ diff --git a/drivers/net/ethernet/realtek/r8127/r8127_ptp.h b/drivers/net/ethernet/realtek/r8127/r8127_ptp.h deleted file mode 100755 index e96afafd70a85..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/r8127_ptp.h +++ /dev/null @@ -1,202 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ -*/ - -/************************************************************************************ - * This product is covered by one or more of the following patents: - * US6,570,884, US6,115,776, and US6,327,625. - ***********************************************************************************/ - -#ifndef _LINUX_R8127_PTP_H -#define _LINUX_R8127_PTP_H - -#include -#include -#include -#include -#include - -#ifndef PTP_CLASS_NONE -#define PTP_CLASS_NONE 0x00 -#endif - -#ifndef PTP_MSGTYPE_SYNC -#define PTP_MSGTYPE_SYNC 0x0 -#endif -#ifndef PTP_MSGTYPE_DELAY_REQ -#define PTP_MSGTYPE_DELAY_REQ 0x1 -#endif -#ifndef PTP_MSGTYPE_PDELAY_REQ -#define PTP_MSGTYPE_PDELAY_REQ 0x2 -#endif -#ifndef PTP_MSGTYPE_PDELAY_RESP -#define PTP_MSGTYPE_PDELAY_RESP 0x3 -#endif - -struct rtl8127_ptp_info { - s64 time_sec; - u32 time_ns; - u16 ts_info; -}; - -#ifndef _STRUCT_TIMESPEC -#define _STRUCT_TIMESPEC -struct timespec { - __kernel_old_time_t tv_sec; /* seconds */ - long tv_nsec; /* nanoseconds */ -}; -#endif - -enum PTP_CMD_TYPE { - PTP_CMD_SET_LOCAL_TIME = 0, - PTP_CMD_DRIFT_LOCAL_TIME, - PTP_CMD_LATCHED_LOCAL_TIME, -}; - -enum PTP_CLKADJ_MOD_TYPE { - NO_FUNCTION = 0, - CLKADJ_MODE_SET = 1, - RESERVED = 2, - DIRECT_READ = 4, - DIRECT_WRITE = 6, - INCREMENT_STEP = 8, - DECREMENT_STEP = 10, - RATE_READ = 12, - RATE_WRITE = 14, -}; - -enum PTP_INSR_TYPE { - EVENT_CAP_INTR = (1 << 0), - TRIG_GEN_INTR = (1 << 1), - RX_TS_INTR = (1 << 2), - TX_TX_INTR = (1 << 3), -}; - -enum PTP_TRX_TS_STA_REG { - TRX_TS_RD = (1 << 0), - TRXTS_SEL = (1 << 1), - RX_TS_PDLYRSP_RDY = (1 << 8), - RX_TS_PDLYREQ_RDY = (1 << 9), - RX_TS_DLYREQ_RDY = (1 << 10), - RX_TS_SYNC_RDY = (1 << 11), - TX_TS_PDLYRSP_RDY = (1 << 12), - TX_TS_PDLYREQ_RDY = (1 << 13), - TX_TS_DLYREQ_RDY = (1 << 14), - TX_TS_SYNC_RDY = (1 << 15), -}; - -#define PTP_CTL_TYPE_0 (0xF3F) -#define PTP_CTL_TYPE_1 (0x2FF) -#define PTP_CTL_TYPE_2 (0x0FF) -#define PTP_CTL_TYPE_3 (0x03F) - -#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) -struct clock_identity { - u8 id[8]; -} __packed; - -struct port_identity { - struct clock_identity clock_identity; - __be16 port_number; -} __packed; - -struct ptp_header { - u8 tsmt; /* transportSpecific | messageType */ - u8 ver; /* reserved | versionPTP */ - __be16 message_length; - u8 domain_number; - u8 reserved1; - u8 flag_field[2]; - __be64 correction; - __be32 reserved2; - struct port_identity source_port_identity; - __be16 sequence_id; - u8 control; - u8 log_message_interval; -} __packed; - -/** - * ptp_parse_header - Get pointer to the PTP v2 header - * @skb: packet buffer - * @type: type of the packet (see ptp_classify_raw()) - * - * This function takes care of the VLAN, UDP, IPv4 and IPv6 headers. The length - * is checked. - * - * Note, internally skb_mac_header() is used. Make sure that the @skb is - * initialized accordingly. - * - * Return: Pointer to the ptp v2 header or NULL if not found - */ -struct ptp_header *ptp_parse_header(struct sk_buff *skb, unsigned int type); - -/** - * ptp_get_msgtype - Extract ptp message type from given header - * @hdr: ptp header - * @type: type of the packet (see ptp_classify_raw()) - * - * This function returns the message type for a given ptp header. It takes care - * of the different ptp header versions (v1 or v2). - * - * Return: The message type - */ -static inline u8 ptp_get_msgtype(const struct ptp_header *hdr, - unsigned int type) -{ - u8 msgtype; - - if (unlikely(type & PTP_CLASS_V1)) { - /* msg type is located at the control field for ptp v1 */ - msgtype = hdr->control; - } else { - msgtype = hdr->tsmt & 0x0f; - } - - return msgtype; -} - -#endif /* LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0) */ - -struct rtl8127_private; -struct RxDescV3; - -int rtl8127_get_ts_info(struct net_device *netdev, - struct ethtool_ts_info *info); - -void rtl8127_ptp_reset(struct rtl8127_private *tp); -void rtl8127_ptp_init(struct rtl8127_private *tp); -void rtl8127_ptp_suspend(struct rtl8127_private *tp); -void rtl8127_ptp_stop(struct rtl8127_private *tp); - -int rtl8127_ptp_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd); - -void rtl8127_rx_ptp_timestamp(struct rtl8127_private *tp, struct sk_buff *skb); - -void rtl8127_set_local_time(struct rtl8127_private *tp); - -#endif /* _LINUX_R8127_PTP_H */ diff --git a/drivers/net/ethernet/realtek/r8127/r8127_rss.c b/drivers/net/ethernet/realtek/r8127/r8127_rss.c deleted file mode 100755 index e364621910052..0000000000000 --- a/drivers/net/ethernet/realtek/r8127/r8127_rss.c +++ /dev/null @@ -1,583 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ -/* -################################################################################ -# -# r8127 is the Linux device driver released for Realtek 10 Gigabit Ethernet -# controllers with PCI-Express interface. -# -# Copyright(c) 2025 Realtek Semiconductor Corp. All rights reserved. -# -# This program is free software; you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 2 of the License, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or -# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for -# more details. -# -# You should have received a copy of the GNU General Public License along with -# this program; if not, see . -# -# Author: -# Realtek NIC software team -# No. 2, Innovation Road II, Hsinchu Science Park, Hsinchu 300, Taiwan -# -################################################################################ -*/ - -/************************************************************************************ - * This product is covered by one or more of the following patents: - * US6,570,884, US6,115,776, and US6,327,625. - ***********************************************************************************/ - -#include -#include "r8127.h" - -enum rtl8127_rss_register_content { - /* RSS */ - RSS_CTRL_TCP_IPV4_SUPP = (1 << 0), - RSS_CTRL_IPV4_SUPP = (1 << 1), - RSS_CTRL_TCP_IPV6_SUPP = (1 << 2), - RSS_CTRL_IPV6_SUPP = (1 << 3), - RSS_CTRL_IPV6_EXT_SUPP = (1 << 4), - RSS_CTRL_TCP_IPV6_EXT_SUPP = (1 << 5), - RSS_HALF_SUPP = (1 << 7), - RSS_CTRL_UDP_IPV4_SUPP = (1 << 11), - RSS_CTRL_UDP_IPV6_SUPP = (1 << 12), - RSS_CTRL_UDP_IPV6_EXT_SUPP = (1 << 13), - RSS_QUAD_CPU_EN = (1 << 16), - RSS_HQ_Q_SUP_R = (1 << 31), -}; - -static int rtl8127_get_rss_hash_opts(struct rtl8127_private *tp, - struct ethtool_rxnfc *cmd) -{ - cmd->data = 0; - - /* Report default options for RSS */ - switch (cmd->flow_type) { - case TCP_V4_FLOW: - cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; - fallthrough; - case UDP_V4_FLOW: - if (tp->rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) - cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; - fallthrough; - case IPV4_FLOW: - cmd->data |= RXH_IP_SRC | RXH_IP_DST; - break; - case TCP_V6_FLOW: - cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; - fallthrough; - case UDP_V6_FLOW: - if (tp->rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) - cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3; - fallthrough; - case IPV6_FLOW: - cmd->data |= RXH_IP_SRC | RXH_IP_DST; - break; - default: - return -EINVAL; - } - - return 0; -} - -int rtl8127_get_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd, - u32 *rule_locs) -{ - struct rtl8127_private *tp = netdev_priv(dev); - int ret = -EOPNOTSUPP; - - if (!(dev->features & NETIF_F_RXHASH)) - return ret; - - switch (cmd->cmd) { - case ETHTOOL_GRXRINGS: - cmd->data = rtl8127_tot_rx_rings(tp); - ret = 0; - break; - case ETHTOOL_GRXFH: - ret = rtl8127_get_rss_hash_opts(tp, cmd); - break; - default: - break; - } - - return ret; -} - -u32 rtl8127_rss_indir_tbl_entries(struct rtl8127_private *tp) -{ - return tp->HwSuppIndirTblEntries; -} - -#define RSS_MASK_BITS_OFFSET (8) -#define RSS_CPU_NUM_OFFSET (16) -#define RTL8127_UDP_RSS_FLAGS (RTL_8125_RSS_FLAG_HASH_UDP_IPV4 | \ - RTL_8125_RSS_FLAG_HASH_UDP_IPV6) -static int _rtl8127_set_rss_hash_opt(struct rtl8127_private *tp) -{ - u32 rss_flags = tp->rss_flags; - u32 hash_mask_len; - u32 rss_ctrl; - - rss_ctrl = ilog2(rtl8127_tot_rx_rings(tp)); - rss_ctrl &= (BIT_0 | BIT_1 | BIT_2); - rss_ctrl <<= RSS_CPU_NUM_OFFSET; - - /* Perform hash on these packet types */ - rss_ctrl |= RSS_CTRL_TCP_IPV4_SUPP - | RSS_CTRL_IPV4_SUPP - | RSS_CTRL_IPV6_SUPP - | RSS_CTRL_IPV6_EXT_SUPP - | RSS_CTRL_TCP_IPV6_SUPP - | RSS_CTRL_TCP_IPV6_EXT_SUPP; - - if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) - rss_ctrl |= RSS_CTRL_UDP_IPV4_SUPP; - - if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) - rss_ctrl |= RSS_CTRL_UDP_IPV6_SUPP | - RSS_CTRL_UDP_IPV6_EXT_SUPP; - - hash_mask_len = ilog2(rtl8127_rss_indir_tbl_entries(tp)); - hash_mask_len &= (BIT_0 | BIT_1 | BIT_2); - rss_ctrl |= hash_mask_len << RSS_MASK_BITS_OFFSET; - - RTL_W32(tp, RSS_CTRL_8125, rss_ctrl); - - return 0; -} - -static int rtl8127_set_rss_hash_opt(struct rtl8127_private *tp, - struct ethtool_rxnfc *nfc) -{ - u32 rss_flags = tp->rss_flags; - - /* - * RSS does not support anything other than hashing - * to queues on src and dst IPs and ports - */ - if (nfc->data & ~(RXH_IP_SRC | RXH_IP_DST | - RXH_L4_B_0_1 | RXH_L4_B_2_3)) - return -EINVAL; - - switch (nfc->flow_type) { - case TCP_V4_FLOW: - case TCP_V6_FLOW: - if (!(nfc->data & RXH_IP_SRC) || - !(nfc->data & RXH_IP_DST) || - !(nfc->data & RXH_L4_B_0_1) || - !(nfc->data & RXH_L4_B_2_3)) - return -EINVAL; - break; - case UDP_V4_FLOW: - if (!(nfc->data & RXH_IP_SRC) || - !(nfc->data & RXH_IP_DST)) - return -EINVAL; - switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { - case 0: - rss_flags &= ~RTL_8125_RSS_FLAG_HASH_UDP_IPV4; - break; - case (RXH_L4_B_0_1 | RXH_L4_B_2_3): - rss_flags |= RTL_8125_RSS_FLAG_HASH_UDP_IPV4; - break; - default: - return -EINVAL; - } - break; - case UDP_V6_FLOW: - if (!(nfc->data & RXH_IP_SRC) || - !(nfc->data & RXH_IP_DST)) - return -EINVAL; - switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) { - case 0: - rss_flags &= ~RTL_8125_RSS_FLAG_HASH_UDP_IPV6; - break; - case (RXH_L4_B_0_1 | RXH_L4_B_2_3): - rss_flags |= RTL_8125_RSS_FLAG_HASH_UDP_IPV6; - break; - default: - return -EINVAL; - } - break; - case SCTP_V4_FLOW: - case AH_ESP_V4_FLOW: - case AH_V4_FLOW: - case ESP_V4_FLOW: - case SCTP_V6_FLOW: - case AH_ESP_V6_FLOW: - case AH_V6_FLOW: - case ESP_V6_FLOW: - case IP_USER_FLOW: - case ETHER_FLOW: - /* RSS is not supported for these protocols */ - if (nfc->data) { - netif_err(tp, drv, tp->dev, "Command parameters not supported\n"); - return -EINVAL; - } - return 0; - break; - default: - return -EINVAL; - } - - /* if we changed something we need to update flags */ - if (rss_flags != tp->rss_flags) { - u32 rss_ctrl = RTL_R32(tp, RSS_CTRL_8125); - - if ((rss_flags & RTL8127_UDP_RSS_FLAGS) && - !(tp->rss_flags & RTL8127_UDP_RSS_FLAGS)) - netdev_warn(tp->dev, - "enabling UDP RSS: fragmented packets may " - "arrive out of order to the stack above\n"); - - tp->rss_flags = rss_flags; - - /* Perform hash on these packet types */ - rss_ctrl |= RSS_CTRL_TCP_IPV4_SUPP - | RSS_CTRL_IPV4_SUPP - | RSS_CTRL_IPV6_SUPP - | RSS_CTRL_IPV6_EXT_SUPP - | RSS_CTRL_TCP_IPV6_SUPP - | RSS_CTRL_TCP_IPV6_EXT_SUPP; - - rss_ctrl &= ~(RSS_CTRL_UDP_IPV4_SUPP | - RSS_CTRL_UDP_IPV6_SUPP | - RSS_CTRL_UDP_IPV6_EXT_SUPP); - - if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV4) - rss_ctrl |= RSS_CTRL_UDP_IPV4_SUPP; - - if (rss_flags & RTL_8125_RSS_FLAG_HASH_UDP_IPV6) - rss_ctrl |= RSS_CTRL_UDP_IPV6_SUPP | - RSS_CTRL_UDP_IPV6_EXT_SUPP; - - RTL_W32(tp, RSS_CTRL_8125, rss_ctrl); - } - - return 0; -} - -int rtl8127_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd) -{ - struct rtl8127_private *tp = netdev_priv(dev); - int ret = -EOPNOTSUPP; - - if (!(dev->features & NETIF_F_RXHASH)) - return ret; - - switch (cmd->cmd) { - case ETHTOOL_SRXFH: - ret = rtl8127_set_rss_hash_opt(tp, cmd); - break; - default: - break; - } - - return ret; -} - -static u32 _rtl8127_get_rxfh_key_size(struct rtl8127_private *tp) -{ - return sizeof(tp->rss_key); -} - -u32 rtl8127_get_rxfh_key_size(struct net_device *dev) -{ - struct rtl8127_private *tp = netdev_priv(dev); - - if (!(dev->features & NETIF_F_RXHASH)) - return 0; - - return _rtl8127_get_rxfh_key_size(tp); -} - -u32 rtl8127_rss_indir_size(struct net_device *dev) -{ - struct rtl8127_private *tp = netdev_priv(dev); - - if (!(dev->features & NETIF_F_RXHASH)) - return 0; - - return rtl8127_rss_indir_tbl_entries(tp); -} - -static void rtl8127_get_reta(struct rtl8127_private *tp, u32 *indir) -{ - int i, reta_size = rtl8127_rss_indir_tbl_entries(tp); - - for (i = 0; i < reta_size; i++) - indir[i] = tp->rss_indir_tbl[i]; -} - -static u32 rtl8127_rss_key_reg(struct rtl8127_private *tp) -{ - return RSS_KEY_8125; -} - -static u32 rtl8127_rss_indir_tbl_reg(struct rtl8127_private *tp) -{ - return RSS_INDIRECTION_TBL_8125_V2; -} - -static void rtl8127_store_reta(struct rtl8127_private *tp) -{ - u16 indir_tbl_reg = rtl8127_rss_indir_tbl_reg(tp); - u32 i, reta_entries = rtl8127_rss_indir_tbl_entries(tp); - u32 reta = 0; - u8 *indir_tbl = tp->rss_indir_tbl; - - /* Write redirection table to HW */ - for (i = 0; i < reta_entries; i++) { - reta |= indir_tbl[i] << (i & 0x3) * 8; - if ((i & 3) == 3) { - RTL_W32(tp, indir_tbl_reg, reta); - - indir_tbl_reg += 4; - reta = 0; - } - } -} - -static void rtl8127_store_rss_key(struct rtl8127_private *tp) -{ - const u16 rss_key_reg = rtl8127_rss_key_reg(tp); - u32 i, rss_key_size = _rtl8127_get_rxfh_key_size(tp); - u32 *rss_key = (u32*)tp->rss_key; - - /* Write redirection table to HW */ - for (i = 0; i < rss_key_size; i+=4) - RTL_W32(tp, rss_key_reg + i, *rss_key++); -} - -#if LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) -int rtl8127_get_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh) -{ - struct rtl8127_private *tp = netdev_priv(dev); - - if (!(dev->features & NETIF_F_RXHASH)) - return -EOPNOTSUPP; - - rxfh->hfunc = ETH_RSS_HASH_TOP; - - if (rxfh->indir) - rtl8127_get_reta(tp, rxfh->indir); - - if (rxfh->key) - memcpy(rxfh->key, tp->rss_key, RTL8127_RSS_KEY_SIZE); - - return 0; -} - -int rtl8127_set_rxfh(struct net_device *dev, struct ethtool_rxfh_param *rxfh, - struct netlink_ext_ack *extack) -{ - struct rtl8127_private *tp = netdev_priv(dev); - int i; - u32 reta_entries = rtl8127_rss_indir_tbl_entries(tp); - - /* We require at least one supported parameter to be changed and no - * change in any of the unsupported parameters - */ - if (rxfh->hfunc != ETH_RSS_HASH_NO_CHANGE && rxfh->hfunc != ETH_RSS_HASH_TOP) - return -EOPNOTSUPP; - - /* Fill out the redirection table */ - if (rxfh->indir) { - int max_queues = tp->num_rx_rings; - - /* Verify user input. */ - for (i = 0; i < reta_entries; i++) - if (rxfh->indir[i] >= max_queues) - return -EINVAL; - - for (i = 0; i < reta_entries; i++) - tp->rss_indir_tbl[i] = rxfh->indir[i]; - } - - /* Fill out the rss hash key */ - if (rxfh->key) - memcpy(tp->rss_key, rxfh->key, RTL8127_RSS_KEY_SIZE); - - rtl8127_store_reta(tp); - - rtl8127_store_rss_key(tp); - - return 0; -} -#else -int rtl8127_get_rxfh(struct net_device *dev, u32 *indir, u8 *key, - u8 *hfunc) -{ - struct rtl8127_private *tp = netdev_priv(dev); - - if (!(dev->features & NETIF_F_RXHASH)) - return -EOPNOTSUPP; - - if (hfunc) - *hfunc = ETH_RSS_HASH_TOP; - - if (indir) - rtl8127_get_reta(tp, indir); - - if (key) - memcpy(key, tp->rss_key, RTL8127_RSS_KEY_SIZE); - - return 0; -} - -int rtl8127_set_rxfh(struct net_device *dev, const u32 *indir, - const u8 *key, const u8 hfunc) -{ - struct rtl8127_private *tp = netdev_priv(dev); - int i; - u32 reta_entries = rtl8127_rss_indir_tbl_entries(tp); - - /* We require at least one supported parameter to be changed and no - * change in any of the unsupported parameters - */ - if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP) - return -EOPNOTSUPP; - - /* Fill out the redirection table */ - if (indir) { - int max_queues = tp->num_rx_rings; - - /* Verify user input. */ - for (i = 0; i < reta_entries; i++) - if (indir[i] >= max_queues) - return -EINVAL; - - for (i = 0; i < reta_entries; i++) - tp->rss_indir_tbl[i] = indir[i]; - } - - /* Fill out the rss hash key */ - if (key) - memcpy(tp->rss_key, key, RTL8127_RSS_KEY_SIZE); - - rtl8127_store_reta(tp); - - rtl8127_store_rss_key(tp); - - return 0; -} -#endif /* LINUX_VERSION_CODE >= KERNEL_VERSION(6,8,0) */ - -static u32 rtl8127_get_rx_desc_hash(struct rtl8127_private *tp, - struct RxDesc *desc) -{ - switch (tp->InitRxDescType) { - case RX_DESC_RING_TYPE_3: - return le32_to_cpu(((struct RxDescV3 *)desc)->RxDescNormalDDWord2.RSSResult); - case RX_DESC_RING_TYPE_4: - return le32_to_cpu(((struct RxDescV4 *)desc)->RxDescNormalDDWord1.RSSResult); - default: - return 0; - } -} - -#define RXS_8125B_RSS_UDP BIT(9) -#define RXS_8125_RSS_IPV4 BIT(10) -#define RXS_8125_RSS_IPV6 BIT(12) -#define RXS_8125_RSS_TCP BIT(13) -#define RTL8127_RXS_RSS_L3_TYPE_MASK (RXS_8125_RSS_IPV4 | RXS_8125_RSS_IPV6) -#define RTL8127_RXS_RSS_L4_TYPE_MASK (RXS_8125_RSS_TCP | RXS_8125B_RSS_UDP) - -#define RXS_8125B_RSS_UDP_V4 BIT(27) -#define RXS_8125_RSS_IPV4_V4 BIT(28) -#define RXS_8125_RSS_IPV6_V4 BIT(29) -#define RXS_8125_RSS_TCP_V4 BIT(30) -#define RTL8127_RXS_RSS_L3_TYPE_MASK_V4 (RXS_8125_RSS_IPV4_V4 | RXS_8125_RSS_IPV6_V4) -#define RTL8127_RXS_RSS_L4_TYPE_MASK_V4 (RXS_8125_RSS_TCP_V4 | RXS_8125B_RSS_UDP_V4) -static void rtl8127_rx_hash_v3(struct rtl8127_private *tp, - struct RxDescV3 *descv3, - struct sk_buff *skb) -{ - u16 rss_header_info; - - if (!(tp->dev->features & NETIF_F_RXHASH)) - return; - - rss_header_info = le16_to_cpu(descv3->RxDescNormalDDWord2.HeaderInfo); - - if (!(rss_header_info & RTL8127_RXS_RSS_L3_TYPE_MASK)) - return; - - skb_set_hash(skb, rtl8127_get_rx_desc_hash(tp, (struct RxDesc *)descv3), - (RTL8127_RXS_RSS_L4_TYPE_MASK & rss_header_info) ? - PKT_HASH_TYPE_L4 : PKT_HASH_TYPE_L3); -} - -static void rtl8127_rx_hash_v4(struct rtl8127_private *tp, - struct RxDescV4 *descv4, - struct sk_buff *skb) -{ - u32 rss_header_info; - - if (!(tp->dev->features & NETIF_F_RXHASH)) - return; - - rss_header_info = le32_to_cpu(descv4->RxDescNormalDDWord1.RSSInfo); - - if (!(rss_header_info & RTL8127_RXS_RSS_L3_TYPE_MASK_V4)) - return; - - skb_set_hash(skb, rtl8127_get_rx_desc_hash(tp, (struct RxDesc *)descv4), - (RTL8127_RXS_RSS_L4_TYPE_MASK_V4 & rss_header_info) ? - PKT_HASH_TYPE_L4 : PKT_HASH_TYPE_L3); -} - -void rtl8127_rx_hash(struct rtl8127_private *tp, - struct RxDesc *desc, - struct sk_buff *skb) -{ - switch (tp->InitRxDescType) { - case RX_DESC_RING_TYPE_3: - rtl8127_rx_hash_v3(tp, (struct RxDescV3 *)desc, skb); - break; - case RX_DESC_RING_TYPE_4: - rtl8127_rx_hash_v4(tp, (struct RxDescV4 *)desc, skb); - break; - default: - return; - } -} - -void rtl8127_disable_rss(struct rtl8127_private *tp) -{ - RTL_W32(tp, RSS_CTRL_8125, 0x00); -} - -void _rtl8127_config_rss(struct rtl8127_private *tp) -{ - _rtl8127_set_rss_hash_opt(tp); - - rtl8127_store_reta(tp); - - rtl8127_store_rss_key(tp); -} - -void rtl8127_config_rss(struct rtl8127_private *tp) -{ - if (!tp->EnableRss) { - rtl8127_disable_rss(tp); - return; - } - - _rtl8127_config_rss(tp); -} - -void rtl8127_init_rss(struct rtl8127_private *tp) -{ - int i; - - for (i = 0; i < rtl8127_rss_indir_tbl_entries(tp); i++) - tp->rss_indir_tbl[i] = ethtool_rxfh_indir_default(i, tp->num_rx_rings); - - netdev_rss_key_fill(tp->rss_key, RTL8127_RSS_KEY_SIZE); -} From e5062ed0f5c03bef24a18485bf36b5a326888bd7 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Thu, 24 Apr 2025 10:10:11 +0000 Subject: [PATCH 043/464] UBUNTU: [Config] nvidia-6.11: Update annotations to enable realtek R8127 module BugLink: https://bugs.launchpad.net/bugs/2109730 Signed-off-by: Abhishek Sahu Acked-by: Matt Ochs Acked-by: Carol L Soto Acked-by: Ian May Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Ian May (cherry picked from commit 59db3944a96c71fbe6c1659faae36a73e07b0d16 noble:linux-nvidia-6.11) Signed-off-by: Jacob Martin (cherry picked from commit aaa549042b742a74100ad1fcb28e003d965cd479) (cherry picked from commit aaa549042b74 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 1edd05ed496e76de824cb4600b8d0e59bb633874 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 7bfa5bcee00ba..c38faf08e2a3d 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -156,6 +156,9 @@ CONFIG_NR_CPUS note<'LP: #1864198'> CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> +CONFIG_R8127 policy<{'amd64': 'n', 'arm64': 'm'}> +CONFIG_R8127 note<'LP: #2109730'> + CONFIG_SAMPLE_CORESIGHT_SYSCFG policy<{'arm64': 'n'}> CONFIG_SAMPLE_CORESIGHT_SYSCFG note<'Required for Grace enablement'> From c3f24fb1a26f2dcc3df671337ff856e637a99a03 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Thu, 22 May 2025 04:53:00 +0000 Subject: [PATCH 044/464] UBUNTU: [Config] nvidia-6.14: Update annotations to enable TPM over FFA BugLink: https://bugs.launchpad.net/bugs/2111511 - crb_acpi_add() checks for start method - If start method is ACPI_TPM2_CRB_WITH_ARM_FFA, then it invokes tpm_crb_ffa_init(). - The tpm_crb_ffa_init() uses IS_REACHABLE() #if IS_REACHABLE(CONFIG_TCG_ARM_CRB_FFA) int tpm_crb_ffa_init(void); #else static inline int tpm_crb_ffa_init(void) { return 0; } #endif So, either tpm_crb (configured with CONFIG_TCG_CRB) should be module or we need to make tpm_crb_ffa (CONFIG_TCG_ARM_CRB_FFA) built-in. - CONFIG_TCG_CRB is selected by other configs so making it module won't be feasible. We can enable CONFIG_TCG_ARM_CRB_FFA to make tpm_crb_ffa built-in. - This also requires to select CONFIG_ARM_FFA_TRANSPORT=y Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 60809f8e7ee9efa255455b3393d96e4d6a2c7306) (cherry picked from commit 60809f8e7ee9 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit b054f0bb84d60776885e1333833c80512808346a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index c38faf08e2a3d..20ede527308a9 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -33,6 +33,9 @@ CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enable CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablement'> +CONFIG_ARM_FFA_TRANSPORT policy<{'arm64': 'y'}> +CONFIG_ARM_FFA_TRANSPORT note<'LP: #2111511'> + CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> @@ -168,6 +171,9 @@ CONFIG_SENSORS_AAEON note<'Disable all Ubuntu ODM dri CONFIG_SPI_TEGRA210_QUAD policy<{'arm64': 'y'}> CONFIG_SPI_TEGRA210_QUAD note<'Ensures the TPM is available before the IMA driver initializes'> +CONFIG_TCG_ARM_CRB_FFA policy<{'arm64': 'y'}> +CONFIG_TCG_ARM_CRB_FFA note<'LP: #2111511'> + CONFIG_TCG_TIS_SPI policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_TCG_TIS_SPI note<'Ensures the TPM is available before the IMA driver initializes'> From 0b3d3605e83c779008b3936940675519eb3313c9 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Wed, 7 May 2025 06:19:42 +0000 Subject: [PATCH 045/464] NVIDIA: SAUCE: Add support for custom ARM FFH offset handler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2114230 The FFH (Functional Fixed Hardware) operation region is maintained by ARM in https://developer.arm.com/documentation/den0048/latest/ OperationRegion (RegionName, RegionSpace, Offset, Length) For ARM FFH, Offset is used to identify the functionality offered by this FFH address space. It must be set to one of the following values: - 0x0 to indicate usage of 32-bit calling convention - 0x1 to indicate usage of 64-bit calling convention. - All other values are reserved. For GB10 and other similar SOC’s, to communicate with embedded controller, a new specification is being defined. It is currently in draft stage and maintained in https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/README.md https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md Offset 4 section: https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md#operation-region-definition This specification internally uses offset 0x4 which is not defined in published ARM specification. So, when ACPI request comes with offset 0x4, then it will fail due to missing support. This commit adds support for custom offset handler. A new EC interface driver will be added in subsequent patches which will registers it callback function. When FFH operation region will be executed with offsets other than 0x0 and 0x1, then it will be forwarded to custom handler. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 89b7d0384a9c noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit df76ec3fc884563f2d73013be08371e870a246d9 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/acpi/arm64/ffh.c | 32 ++++++++++++++++++++++++++++++++ include/linux/acpi.h | 16 ++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/drivers/acpi/arm64/ffh.c b/drivers/acpi/arm64/ffh.c index 04380bab193df..8bce1070d3716 100644 --- a/drivers/acpi/arm64/ffh.c +++ b/drivers/acpi/arm64/ffh.c @@ -19,6 +19,9 @@ struct acpi_ffh_data { struct arm_smccc_1_2_regs *res); }; +static int (*ffh_custom_handler)(struct acpi_ffh_info *info, + acpi_integer *value, void *region_context); + int acpi_ffh_address_space_arch_setup(void *handler_ctxt, void **region_ctxt) { enum arm_smccc_conduit conduit; @@ -99,9 +102,38 @@ int acpi_ffh_address_space_arch_handler(acpi_integer *value, void *region_contex ffh_ctxt->invoke_ffh64_fn(r, r); memcpy(value, r, ffh_ctxt->info.length); } + } else if (ffh_custom_handler) { + int err = ffh_custom_handler(&ffh_ctxt->info, value, + region_context); + if (err) { + pr_err("ARM FFH custom offset handler returned error=%d\n", + err); + ret = AE_ERROR; + } } else { ret = AE_ERROR; } return ret; } + +int acpi_arm64_ffh_update_custom_offset_handler( + int (*handler)(struct acpi_ffh_info *info, acpi_integer *value, + void *region_context)) +{ + if (!handler) { + pr_debug("ARM FFH custom offset handler unregistered\n"); + ffh_custom_handler = NULL; + return 0; + } + + if (ffh_custom_handler) + pr_debug("ARM FFH custom offset handler updated\n"); + else + pr_debug("ARM FFH custom offset handler registered\n"); + + ffh_custom_handler = handler; + + return 0; +} +EXPORT_SYMBOL_GPL(acpi_arm64_ffh_update_custom_offset_handler); diff --git a/include/linux/acpi.h b/include/linux/acpi.h index 4d2f0bed7a06d..ee3aa5604512a 100644 --- a/include/linux/acpi.h +++ b/include/linux/acpi.h @@ -1596,10 +1596,26 @@ extern int acpi_ffh_address_space_arch_setup(void *handler_ctxt, void **region_ctxt); extern int acpi_ffh_address_space_arch_handler(acpi_integer *value, void *region_context); +int acpi_ffh_address_space_arch_update_custom_offset_handler( + int (*handler)(struct acpi_ffh_info *info, acpi_integer *value, + void *region_context)); #else static inline void acpi_init_ffh(void) { } #endif +#if defined(CONFIG_ACPI_FFH) && defined(CONFIG_ARM64) +int acpi_arm64_ffh_update_custom_offset_handler( + int (*handler)(struct acpi_ffh_info *info, acpi_integer *value, + void *region_context)); +#else +static inline int acpi_arm64_ffh_update_custom_offset_handler( + int (*handler)(struct acpi_ffh_info *info, acpi_integer *value, + void *region_context)) +{ + return -EOPNOTSUPP; +} +#endif + #ifdef CONFIG_ACPI extern void acpi_device_notify(struct device *dev); extern void acpi_device_notify_remove(struct device *dev); From 5d49eed516943138910543e2c00ac7b884ae2286 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Wed, 7 May 2025 07:32:22 +0000 Subject: [PATCH 046/464] NVIDIA: SAUCE: Add nvidia ffa driver for EC communication BugLink: https://bugs.launchpad.net/bugs/2114230 Please refer https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md for details regarding FFA device details for secure EC services communication. The HID 'MSFT000C' is reserved for FFA devices. This HID is documented in https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md#hid-definition This commit adds a platform driver which binds with FFA device. In its probe routine, it executes the AVAL method to check if FFA can be used for secure EC services communication. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 555e41e166a4 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit bdd6ed09666656b94b7d75c56f3c7d9254e6c53f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/Kconfig | 13 ++++ drivers/platform/arm64/Makefile | 1 + drivers/platform/arm64/nvidia-ffa-ec.c | 92 ++++++++++++++++++++++++++ 3 files changed, 106 insertions(+) create mode 100644 drivers/platform/arm64/nvidia-ffa-ec.c diff --git a/drivers/platform/arm64/Kconfig b/drivers/platform/arm64/Kconfig index c1ca1d78eeb86..80cefd5772cec 100644 --- a/drivers/platform/arm64/Kconfig +++ b/drivers/platform/arm64/Kconfig @@ -102,4 +102,17 @@ config EC_LENOVO_YOGA_SLIM7X mute button, and reporting device suspend to the EC so it can take appropriate actions. +config NVIDIA_FFA_EC + tristate "NVIDIA FFA EC services driver" + depends on ARM_FFA_TRANSPORT || COMPILE_TEST + depends on ACPI + depends on ACPI_FFH + help + Enable NVIDIA FFA EC services. + For GB10 and other similar SOC’s, to communicate with embedded controller, a new + specification is being defined. It is currently in draft stage and maintained in + https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md + + Say M or Y here to include this support. + endif # ARM64_PLATFORM_DEVICES diff --git a/drivers/platform/arm64/Makefile b/drivers/platform/arm64/Makefile index c135a895a3ea5..c693a0501631b 100644 --- a/drivers/platform/arm64/Makefile +++ b/drivers/platform/arm64/Makefile @@ -10,3 +10,4 @@ obj-$(CONFIG_EC_HUAWEI_GAOKUN) += huawei-gaokun-ec.o obj-$(CONFIG_EC_LENOVO_YOGA_C630) += lenovo-yoga-c630.o obj-$(CONFIG_EC_LENOVO_THINKPAD_T14S) += lenovo-thinkpad-t14s.o obj-$(CONFIG_EC_LENOVO_YOGA_SLIM7X) += lenovo-yoga-slim7x.o +obj-$(CONFIG_NVIDIA_FFA_EC) += nvidia-ffa-ec.o diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c new file mode 100644 index 0000000000000..22dd49a1355cc --- /dev/null +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include +#include +#include + +#define DRV_NAME "nvidia-ffa-ec" + +/* platform device for FFA ACPI device (HID MSFT000C) */ +static struct platform_device *ffa_pdev; + +static const struct acpi_device_id nvidia_ffa_device_ids[] = { + /* + * Please refer + * https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md#hid-definition + * where MSFT000C is documented. + * + * The _HID 'MSFT000C' is reserved for FFA device which uses + * FFA interface for secure EC communication. + */ + {"MSFT000C", 0}, + {"", 0}, +}; + +MODULE_DEVICE_TABLE(acpi, nvidia_ffa_device_ids); + +static int nvidia_ffa_probe(struct platform_device *pdev) +{ + struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); + acpi_status status; + unsigned long long data = 0; + + if (ffa_pdev) { + dev_err(&pdev->dev, "FFA device already registered\n"); + return -EINVAL; + } + + if (!adev) { + dev_err(&pdev->dev, "No ACPI companion found\n"); + return -ENODEV; + } + + status = acpi_evaluate_integer(adev->handle, "AVAL", NULL, &data); + if (ACPI_FAILURE(status)) { + dev_err(&pdev->dev, "Failed to execute AVAL method\n"); + return -ENODEV; + } + + if (data != 1) { + dev_err(&pdev->dev, "FFA not available\n"); + return -ENODEV; + } + + ffa_pdev = pdev; + + return 0; +} + +static void nvidia_ffa_remove(struct platform_device *pdev) +{ + ffa_pdev = NULL; +} + +static struct platform_driver nvidia_ffa_driver = { + .probe = nvidia_ffa_probe, + .remove = nvidia_ffa_remove, + .driver = { + .name = "nvidia-ffa", + .acpi_match_table = nvidia_ffa_device_ids, + }, +}; + +static int __init nvidia_ffa_init(void) +{ + return platform_driver_register(&nvidia_ffa_driver); +} +module_init(nvidia_ffa_init); + +static void __exit nvidia_ffa_exit(void) +{ + platform_driver_unregister(&nvidia_ffa_driver); +} +module_exit(nvidia_ffa_exit); + +MODULE_SOFTDEP("pre: arm-ffa"); +MODULE_AUTHOR("NVIDIA CORPORATION"); +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("NVIDIA FFA EC services driver"); From 130018c206fd4e909895cec6a6d7a6108fbd3e45 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Wed, 7 May 2025 08:12:27 +0000 Subject: [PATCH 047/464] NVIDIA: SAUCE: Add ffa driver for each secure EC service MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2114230 Please refer https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md for details regarding FFA device details for secure EC services communication. Each secure EC service is identified by separate UUID. When generic FFA module loads (ffa_module), then it gets the list of partitions. Each EC service is a FFA partition and ffa_module creates a device for each partition. These devices will be added in arm_ffa bus type. The device will be named as arm-ffa-. For binding with these devices, a driver needs to be registered in arm_ffa bus type. This driver uses structure ‘struct ffa_driver’ where it uses UUID as ID table. The binding of the driver to device happens on basis of UUID. The secure EC services FFA driver is dependent upon main FFA device to be created (which uses ACPI ID MSFT000C), so ffa_driver_register()/ffa_driver_unregister() is invoked from nvidia_ffa_probe()/nvidia_ffa_remove(). Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 9613a5c07163 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 5ede0e8d753f77d7a792351102f800dda882032f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 109 +++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index 22dd49a1355cc..537e7a51bd133 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -7,12 +7,111 @@ #include #include #include +#include +#include #define DRV_NAME "nvidia-ffa-ec" /* platform device for FFA ACPI device (HID MSFT000C) */ static struct platform_device *ffa_pdev; +static const uuid_t nvidia_ec_managment_service_uuid = + UUID_INIT(0x330c1273, 0xfde5, 0x4757, 0x98, 0x19, 0x5b, 0x65, 0x39, 0x03, 0x75, 0x02); + +static const uuid_t nvidia_ec_power_service_uuid = + UUID_INIT(0x7157addf, 0x2fbe, 0x4c63, 0xae, 0x95, 0xef, 0xac, 0x16, 0xe3, 0xb0, 0x1c); + +static const uuid_t nvidia_ec_battery_service_uuid = + UUID_INIT(0x25cb5207, 0xac36, 0x427d, 0xaa, 0xef, 0x3a, 0xa7, 0x88, 0x77, 0xd2, 0x7e); + +static const uuid_t nvidia_ec_thermal_service_uuid = + UUID_INIT(0x31f56da7, 0x593c, 0x4d72, 0xa4, 0xb3, 0x8f, 0xc7, 0x17, 0x1a, 0xc0, 0x73); + +static const uuid_t nvidia_ec_fan_service_uuid = + UUID_INIT(0x7697530c, 0xd079, 0x4ec1, 0xa4, 0xc4, 0xcf, 0x0d, 0x2b, 0xdc, 0x93, 0xfa); + +static const uuid_t nvidia_ec_ucsi_service_uuid = + UUID_INIT(0x65467f50, 0x827f, 0x4e4f, 0x87, 0x70, 0xdb, 0xf4, 0xc3, 0xf7, 0x7f, 0x45); + +static const uuid_t nvidia_ec_input_service_uuid = + UUID_INIT(0xe3168a99, 0x4a57, 0x4a2b, 0x8c, 0x5e, 0x11, 0xbc, 0xfe, 0xc7, 0x34, 0x06); + +static const uuid_t nvidia_ec_time_alarm_service_uuid = + UUID_INIT(0x23ea63ed, 0xb593, 0x46ea, 0xb0, 0x27, 0x89, 0x24, 0xdf, 0x88, 0xe9, 0x2f); + +/* EC service FFA device structure */ +struct nvidia_ec_ffa_device { + struct ffa_device *ffa_dev; + struct list_head list; +}; + +/* List to contain all EC services FFA device */ +static LIST_HEAD(nvidia_ec_ffa_dev_head); + +/* Lock to serialize EC services FFA device list access */ +static DEFINE_MUTEX(nvidia_ffa_lock); + +static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) +{ + struct nvidia_ec_ffa_device *nvidia_ec_ffa_dev; + + if (!ffa_pdev) { + dev_err(&ffa_dev->dev, "nvidia ffa device not available\n"); + return -ENODEV; + } + + nvidia_ec_ffa_dev = devm_kmalloc(&ffa_dev->dev, + sizeof(*nvidia_ec_ffa_dev), + GFP_KERNEL); + if (!nvidia_ec_ffa_dev) { + dev_err(&ffa_dev->dev, "Failed to allocate memory\n"); + return -ENOMEM; + } + + nvidia_ec_ffa_dev->ffa_dev = ffa_dev; + INIT_LIST_HEAD(&nvidia_ec_ffa_dev->list); + + mutex_lock(&nvidia_ffa_lock); + list_add(&nvidia_ec_ffa_dev->list, &nvidia_ec_ffa_dev_head); + mutex_unlock(&nvidia_ffa_lock); + + return 0; +} + +static void nvidia_ffa_ec_service_remove(struct ffa_device *ffa_dev) +{ + struct nvidia_ec_ffa_device *cur, *tmp; + + mutex_lock(&nvidia_ffa_lock); + list_for_each_entry_safe(cur, tmp, &nvidia_ec_ffa_dev_head, list) { + if (cur->ffa_dev == ffa_dev) { + list_del(&cur->list); + devm_kfree(&ffa_dev->dev, cur); + break; + } + } + mutex_unlock(&nvidia_ffa_lock); +} + +static const struct ffa_device_id nvidia_ffa_ec_service_ids[] = { + { nvidia_ec_managment_service_uuid }, + { nvidia_ec_power_service_uuid }, + { nvidia_ec_battery_service_uuid }, + { nvidia_ec_thermal_service_uuid }, + { nvidia_ec_fan_service_uuid }, + { nvidia_ec_ucsi_service_uuid }, + { nvidia_ec_input_service_uuid }, + { nvidia_ec_time_alarm_service_uuid }, + {} +}; + +static struct ffa_driver nvidia_ffa_ec_service_driver = { + .name = DRV_NAME, + .probe = nvidia_ffa_ec_service_probe, + .remove = nvidia_ffa_ec_service_remove, + .id_table = nvidia_ffa_ec_service_ids, +}; + static const struct acpi_device_id nvidia_ffa_device_ids[] = { /* * Please refer @@ -33,6 +132,7 @@ static int nvidia_ffa_probe(struct platform_device *pdev) struct acpi_device *adev = ACPI_COMPANION(&pdev->dev); acpi_status status; unsigned long long data = 0; + int ret; if (ffa_pdev) { dev_err(&pdev->dev, "FFA device already registered\n"); @@ -57,11 +157,20 @@ static int nvidia_ffa_probe(struct platform_device *pdev) ffa_pdev = pdev; + ret = ffa_driver_register(&nvidia_ffa_ec_service_driver, THIS_MODULE, DRV_NAME); + if (ret) { + dev_err(&pdev->dev, + "Failed to register ec service driver error=%d\n", ret); + ffa_pdev = NULL; + return ret; + } + return 0; } static void nvidia_ffa_remove(struct platform_device *pdev) { + ffa_driver_unregister(&nvidia_ffa_ec_service_driver); ffa_pdev = NULL; } From 6d68957e5de718aed9c0ab510f76e6280a2c5d3c Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Wed, 7 May 2025 09:06:47 +0000 Subject: [PATCH 048/464] NVIDIA: SAUCE: Add support for EC secure service communication BugLink: https://bugs.launchpad.net/bugs/2114230 Please refer https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md for details regarding FFA device details for secure EC services communication. When ACPI interpreter runs code with FFH operation region offset 4, then this data is meant for EC secure services. The FFH buffer has data in FFA_REQ_PACKET format. In this packet, it has UUID for EC service and then the service specific raw data. This commit adds a custom FFH offset handler. When request comes with custom offset then it will be handled by nvdia FFA EC driver. Inside the custom ffh callback, it extracts the UUID and gets the ffa_device for it. Then it fills raw data in ffa_send_direct_data2 and invoke sync_send_receive2() routine for that ffa_device. Once it gets the response back, then it fill data in FFA_RESP_PACKET format and ACPI interpreter passes that data to upper layer. NOTE: In the above document, the FFA_REQ_PACKET and FFA_RESP_PACKET uses different format. But in latest firmware code, the ACPI implementation is done using same format for both request and response (follows the FFA_REQ_PACKET format). The status bit will be updated in the response (0 for success and 1 for failure). This mixed endian is documented in https://cdrdv2-public.intel.com/772722/asl-tutorial-v20190625.pdf In addition to Concatenate, there are several useful macros that generate buffers from strings. For example, the ToUUID macro takes a string of the form aabbccdd-eeff-gghh-iijj-kkllmmnnoopp where aa through pp represent one byte values encoded with hexadecimal characters. This string gets converted to a 16-byte buffer that looks like the following: Buffer() { dd, cc, bb, aa, ff, ee, hh, gg, ii, jj, kk, ll, mm, nn, oo, pp } This mixture of little endian and big-endian encoding UUID is called a mixed-endian format. The use of strings and the ToUUID macro is a convenient way to avoid having to manually encode the mixed-endian format. There are many other macros that provide similar conveniences, such as EISAID. In kernel, it is represented with guid_t. Inside nvidia_ffh_handler(), we need to covert buffer of 16 bytes from FFA UUID to AML UUID format. nvidia_get_uuid_from_aml_buf() converts the AML UUID buffer into FFA UUID format. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 40ca7bcc7774 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 613505b042bcdd9ac64a33690d9b175f596b0828 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 115 +++++++++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index 537e7a51bd133..4181addad8108 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -51,6 +51,113 @@ static LIST_HEAD(nvidia_ec_ffa_dev_head); /* Lock to serialize EC services FFA device list access */ static DEFINE_MUTEX(nvidia_ffa_lock); +/* EC secure services FFA packet structure sent via ACPI */ +struct nvidia_ec_ffa_packet { + u8 status; + u8 length; + u8 uuid[UUID_SIZE]; + u8 rawdata[]; +} __packed; + +/* + * ACPI ASL code uses ToUUID() macro which encodes it in mixed-endian format. + * Convert the AML UUID buffer into FFA UUID format. + */ +static uuid_t nvidia_get_uuid_from_aml_buf(const u8 *buf) +{ + return (uuid_t) {{ buf[3], buf[2], buf[1], buf[0], + buf[5], buf[4], buf[7], buf[6], + buf[8], buf[9], buf[10], buf[11], + buf[12], buf[13], buf[14], buf[15] }}; +} + +/* + * Handler function for FFH operation region offset 4. + * When ACPI interpreter runs code with FFH operation region offset 4, + * then this data is meant for EC secure services. The FFH buffer has + * data in 'struct nvidia_ec_ffa_packet' format. In this packet, it has UUID + * for EC secure service and then the service specific raw data. + * + * 1. Extract the UUID from this packet and get ffa_device for it. + * 2. Fill raw data in 'struct ffa_send_direct_data2' and + * invoke sync_send_receive2() routine for the ffa_device. + * 3. From response, fill the data in 'struct ffa_send_direct_data2' + * and return. + */ +static int nvidia_ffh_handler(struct acpi_ffh_info *info, acpi_integer *value, void *region_context) +{ + struct ffa_send_direct_data2 ffa_data = { 0 }; + struct nvidia_ec_ffa_packet *ffa_packet = (struct nvidia_ec_ffa_packet *)value; + struct nvidia_ec_ffa_device *cur, *ec_dev = NULL; + int ret; + uuid_t uuid; + + /* Only offset 4 is supported */ + if (info->offset != 4) + return -EOPNOTSUPP; + + /* Length should not be less than header length */ + if (info->length < offsetof(struct nvidia_ec_ffa_packet, rawdata)) + return -EINVAL; + + /* Length should not be less than actual packet length */ + if (info->length < + ffa_packet->length + offsetof(struct nvidia_ec_ffa_packet, rawdata)) { + ffa_packet->status = 1; + return -EINVAL; + } + + /* Packet length should not greater than FFA supported data length */ + if (ffa_packet->length > sizeof(ffa_data.data)) { + ffa_packet->status = 1; + return -EINVAL; + } + + /* Convert AML UUID to FFA UUID */ + uuid = nvidia_get_uuid_from_aml_buf((u8 *)ffa_packet->uuid); + + mutex_lock(&nvidia_ffa_lock); + /* Get nvidia_ec_ffa_device for the current UUID */ + list_for_each_entry(cur, &nvidia_ec_ffa_dev_head, list) { + if (uuid_equal(&uuid, &cur->ffa_dev->uuid)) { + ec_dev = cur; + break; + } + } + mutex_unlock(&nvidia_ffa_lock); + + if (!ec_dev) { + ffa_packet->status = 1; + return -EINVAL; + } + + /* Copy the ACPI FFH packet data into FFA data */ + memcpy(ffa_data.data, ffa_packet->rawdata, ffa_packet->length); + + if (!ec_dev->ffa_dev->ops || + !ec_dev->ffa_dev->ops->msg_ops || + !ec_dev->ffa_dev->ops->msg_ops->sync_send_receive2) { + return -EINVAL; + } + + ret = ec_dev->ffa_dev->ops->msg_ops->sync_send_receive2(ec_dev->ffa_dev, + &ffa_data); + if (ret) { + dev_err(&ec_dev->ffa_dev->dev, + "Failed to send FFA messages error=%d\n", ret); + ffa_packet->status = 1; + return ret; + } + + /* Set the status as success */ + ffa_packet->status = 0; + + /* Copy the ACPI FFA data back into ACPI FFH packet */ + memcpy(ffa_packet->rawdata, ffa_data.data, ffa_packet->length); + + return 0; +} + static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) { struct nvidia_ec_ffa_device *nvidia_ec_ffa_dev; @@ -155,6 +262,13 @@ static int nvidia_ffa_probe(struct platform_device *pdev) return -ENODEV; } + ret = acpi_arm64_ffh_update_custom_offset_handler(nvidia_ffh_handler); + if (ret) { + dev_err(&pdev->dev, + "Failed to register custom offset handler error=%d\n", ret); + return ret; + } + ffa_pdev = pdev; ret = ffa_driver_register(&nvidia_ffa_ec_service_driver, THIS_MODULE, DRV_NAME); @@ -172,6 +286,7 @@ static void nvidia_ffa_remove(struct platform_device *pdev) { ffa_driver_unregister(&nvidia_ffa_ec_service_driver); ffa_pdev = NULL; + acpi_arm64_ffh_update_custom_offset_handler(NULL); } static struct platform_driver nvidia_ffa_driver = { From 269b9edc47cd3223dd73f16eb8c4ed585dd1737f Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Wed, 7 May 2025 09:22:14 +0000 Subject: [PATCH 049/464] NVIDIA: SAUCE: Rescan acpi devices that uses secure EC communication BugLink: https://bugs.launchpad.net/bugs/2114230 - During boot time, ACPI probe happens first. It calls _STA method for each added device. - Inside _STA method for device managed by EC, it uses FFH offset 4. - The request will fail since there is no custom handler registered for offset 0x4 and device will be disabled. - If rescan happens on acpi bus, then device _STA method will be called again. This commit adds support to get acpi id from UUID and invokes acpi_bus_scan(). NOTE: nvidia_get_acpi_id_from_uuid() returns ACPI ID only for few services. We don't have a corresponding driver available for all the services in the current code. For few services only, its node uses generic ACPI ID and has driver available. For rest of the service, the driver is not yet available, or the published spec is not updated with full ACPI sample code. Once we have driver available for that, then we can add those ACPI IDs in this list. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 971a25e19691 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e4ec4146cc8afe52290d15148ecbff121d06a140 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 42 ++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index 4181addad8108..b47a99a5d45c3 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -71,6 +71,35 @@ static uuid_t nvidia_get_uuid_from_aml_buf(const u8 *buf) buf[12], buf[13], buf[14], buf[15] }}; } +static int nvidia_ffa_rescan_acpi_device(struct device *dev, void *data) +{ + struct acpi_device *adev = to_acpi_device(dev); + + if (acpi_dev_hid_uid_match(adev, data, NULL)) { + acpi_bus_scan(adev->handle); + return 1; + } + + return 0; +} + +static const char *nvidia_get_acpi_id_from_uuid(uuid_t *uuid) +{ + if (uuid_equal(uuid, &nvidia_ec_battery_service_uuid)) + return "PNP0C0A"; + + if (uuid_equal(uuid, &nvidia_ec_time_alarm_service_uuid)) + return "ACPI000E"; + + if (uuid_equal(uuid, &nvidia_ec_fan_service_uuid)) + return "PNP0C0B"; + + if (uuid_equal(uuid, &nvidia_ec_ucsi_service_uuid)) + return "PNP0CA0"; + + return NULL; +} + /* * Handler function for FFH operation region offset 4. * When ACPI interpreter runs code with FFH operation region offset 4, @@ -161,6 +190,7 @@ static int nvidia_ffh_handler(struct acpi_ffh_info *info, acpi_integer *value, v static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) { struct nvidia_ec_ffa_device *nvidia_ec_ffa_dev; + const char *acpi_id = NULL; if (!ffa_pdev) { dev_err(&ffa_dev->dev, "nvidia ffa device not available\n"); @@ -182,6 +212,18 @@ static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) list_add(&nvidia_ec_ffa_dev->list, &nvidia_ec_ffa_dev_head); mutex_unlock(&nvidia_ffa_lock); + /* + * When acpi subsystem probe all ACPI devices, then it execute _STA + * method for each device. The _STA method fails at that time since + * custom FFA driver won't be ready. Get ACPI ID from UUID and + * rescan the device again. + */ + acpi_id = nvidia_get_acpi_id_from_uuid(&ffa_dev->uuid); + if (acpi_id) { + acpi_bus_for_each_dev(nvidia_ffa_rescan_acpi_device, + (void *)acpi_id); + } + return 0; } From 55683b67ff1d398e0a6f1c5c436fe6ee857a64c9 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Mon, 12 Aug 2024 22:39:25 -0500 Subject: [PATCH 050/464] NVIDIA: SAUCE: irqchip/gic-v3: Allow unused SGIs for drivers/modules BugLink: https://bugs.launchpad.net/bugs/2114230 The commit 897e9e60c016 ("firmware: arm_ffa: Initial support for scheduler receiver interrupt") adds support for SGI interrupts in the FFA driver. However, the validation for SGIs in the GICv3 is too strict, causing the driver probe to fail. This patch relaxes the SGI validation check, allowing callers to use SGIs if the requested SGI number is greater than or equal to MAX_IPI, which fixes the TFA driver probe failure. This issue is observed on NVIDIA server platform with FFA-v1.1. PTP clock support registered EDAC MC: Ver: 3.0.0 ARM FF-A: Driver version 1.1 ARM FF-A: Firmware version 1.1 found GICv3: [Firmware Bug]: Illegal GSI8 translation request ARM FF-A: Failed to create IRQ mapping! ARM FF-A: Notification setup failed -61, not enabled ARM FF-A: Failed to register driver sched callback -95 scmi_core: SCMI protocol bus registered This patch was sent in arm mailing list for upstream but it got rejected. https://patchwork.kernel.org/project/linux-arm-kernel/patch/20240813033925.925947-1-sdonthineni@nvidia.com/ The proper fix requires some kind of mechanism by which a SGI can be requested by module but that needs discussion with arm and it will take time. This patch will break only if MAX_IPI value gets changed. This patch adds a BUILD_BUG_ON() to catch that situation. Once proper solution is concluded then this patch will be reverted. Signed-off-by: Shanker Donthineni Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (backported from commit fd136cf979db) [maskedarray: removed enum ipi_msg_type definition as it appears in upstream commit "irqchip/gic-v5: Add GICv5 LPI/IPI support"] Signed-off-by: Abdur Rahman (cherry picked from commit df84d5ddd125e925e912c8af1c76a71d3beb1de6 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/irqchip/irq-gic-v3.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index 20f13b686ab22..3dd47cb01bb16 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1634,7 +1634,13 @@ static int gic_irq_domain_translate(struct irq_domain *d, if(fwspec->param_count != 2) return -EINVAL; - if (fwspec->param[0] < 16) { + /* + * Below check was added on assumption that MAX_IPI + * value will not be greater than 8. + */ + BUILD_BUG_ON(MAX_IPI > 8); + + if (fwspec->param[0] < MAX_IPI) { pr_err(FW_BUG "Illegal GSI%d translation request\n", fwspec->param[0]); return -EINVAL; From 47f0d79e9caf5a10d834d2aa980e5ec49066e73b Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Thu, 8 May 2025 20:26:27 +0000 Subject: [PATCH 051/464] NVIDIA: SAUCE: Add support for notifications from secure EC services BugLink: https://bugs.launchpad.net/bugs/2114230 Please refer https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md for details regarding FFA device details for secure EC services communication. 1. We need to get virtual IDs which a EC service supports. In the FFA node, the _DSD object contains this information. If we look the sample from above document, Name(_DSD, Package() { ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"), //Device Prop UUID Package() { Package(2) { "arm-arml0002-ffa-ntf-bind", Package() { 1, // Revision 2, // Count of following packages Package () { ToUUID("330c1273-fde5-4757-9819-5b6539037502"), // Service1 UUID Package () { 0x01, //Cookie1 (UINT32) 0x07, //Cookie2 } }, Package () { ToUUID("b510b3a3-59f6-4054-ba7a-ff2eb1eac765"), // Service2 UUID Package () { 0x01, //Cookie1 0x03, //Cookie2 } } } } } }) // _DSD() Then it uses a nexted package structure. nvidia_ffa_fill_notification_map() added in this commit parses the _DSD object and fill the notification id map for that service. 2. Once the virtual ID is get then it needs to map to physical ID by invoking function 1 in the notify service. 3. The UUID for notification service is B510B3A3-59F6-4054-BA7A-FF2EB1EAC765. An FFA device will be created for this notification service by ffa_module. This notify service needs to be probed first. To make that happen, a separate ffa_driver instance is created and it is getting registered first. 4. We can do 1:1 mapping between virtual ID and hardware ID. 5. We need to invoke notify_request() with hardware notification ID. It registers callback function for notification. 6. Once notification comes then we need to evaluate _DSM method with virtual ID (which will be mapped same as hardware ID). 7. The function 2 in the notify service should destroy the mapping. But it is nither implemented in the firmware not its documentation is available. A TODO comment is added in nvidia_ffa_notification_destroy(). Also, if we unload and reload the modules, the existing mapping still exists. In nvidia_ffa_notification_setup(), ignore the error for this case. When firmware is updated, then the error will be returned. 8. The notification service FFA device is needed by each EC secure services FFA device to get virtual notification list. Now following device dependency chain is created. FFA device <- notification service FFA device <- EC secure services FFA device To satisfy this, call driver registration in its dependent driver probe routine. Similarly, do the driver registration in its dependent driver removed routine. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 1287a1d24fd0 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 605dde1a0254e9eebe7ab4142891430a71f4da39 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 473 ++++++++++++++++++++++++- 1 file changed, 468 insertions(+), 5 deletions(-) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index b47a99a5d45c3..78068f1237b57 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -15,6 +15,12 @@ /* platform device for FFA ACPI device (HID MSFT000C) */ static struct platform_device *ffa_pdev; +/* FFA device for EC notification service */ +static struct ffa_device *notify_ffa_dev; + +static const uuid_t nvidia_ec_notify_service_uuid = + UUID_INIT(0xb510b3a3, 0x59f6, 0x4054, 0xba, 0x7a, 0xff, 0x2e, 0xb1, 0xea, 0xc7, 0x65); + static const uuid_t nvidia_ec_managment_service_uuid = UUID_INIT(0x330c1273, 0xfde5, 0x4757, 0x98, 0x19, 0x5b, 0x65, 0x39, 0x03, 0x75, 0x02); @@ -39,9 +45,19 @@ static const uuid_t nvidia_ec_input_service_uuid = static const uuid_t nvidia_ec_time_alarm_service_uuid = UUID_INIT(0x23ea63ed, 0xb593, 0x46ea, 0xb0, 0x27, 0x89, 0x24, 0xdf, 0x88, 0xe9, 0x2f); +static const guid_t nvidia_notify_bind_guid = + GUID_INIT(0xdaffd814, 0x6eba, 0x4d8c, 0x8a, 0x91, 0xbc, 0x9b, 0xbf, 0x4a, 0xa3, 0x01); + +static const guid_t nvidia_notify_dsm_guid = + GUID_INIT(0x7681541e, 0x8827, 0x4239, 0x8d, 0x9d, 0x36, 0xbe, 0x7f, 0xe1, 0x25, 0x42); + +#define NVIDIA_FFA_MAX_NOTIFICATIONS 64 + /* EC service FFA device structure */ struct nvidia_ec_ffa_device { struct ffa_device *ffa_dev; + u8 notification_count; + u8 notification_id[NVIDIA_FFA_MAX_NOTIFICATIONS]; struct list_head list; }; @@ -71,6 +87,27 @@ static uuid_t nvidia_get_uuid_from_aml_buf(const u8 *buf) buf[12], buf[13], buf[14], buf[15] }}; } +/* + * ACPI ASL code uses ToUUID() macro which encodes it in mixed-endian format. + * Convert UUID buffer to AML UUID. + */ +static void nvidia_uuid_to_aml_uuid_buf(const uuid_t *uuid, u8 *buf) +{ + const u8 *src = (u8 *)uuid; + + buf[0] = src[3]; + buf[1] = src[2]; + buf[2] = src[1]; + buf[3] = src[0]; + + buf[4] = src[5]; + buf[5] = src[4]; + buf[6] = src[7]; + buf[7] = src[6]; + + memcpy(buf + 8, src + 8, 8); +} + static int nvidia_ffa_rescan_acpi_device(struct device *dev, void *data) { struct acpi_device *adev = to_acpi_device(dev); @@ -100,6 +137,380 @@ static const char *nvidia_get_acpi_id_from_uuid(uuid_t *uuid) return NULL; } +/* + * Fill the virtual notification IDs array supported by the current FFA device. + * ACPI _DSD object contains notification mapping. It uses nexted package + * acpi object. + * + * From the example given in + * https://github.com/OpenDevicePartnership/documentation/blob/main/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md#register-notification + * + * pkg1 Name(_DSD, Package() { + * pkg1_guid ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"), // Device Prop UUID + * pkg2 Package() { + * pkg3 Package(2) { + * pkg3_prop "arm-arml0002-ffa-ntf-bind", + * pkg4 Package() { + * pkg4_rev 1, // Revision + * pkg4_count 1, // Count of following packages + * pkg5 Package () { + * pkg5_uuid ToUUID("330c1273-fde5-4757-9819-5b6539037502"), // Service1 UUID + * pkg6 Package () { + * pkg6_notify_id[] 0x01, // Cookie1 (UINT32) + * 0x07, // Cookie2 + * } + * }, + * } + * } + * } + * }) // _DSD() + * + * The variable names in this function are according to above. + */ +static int nvidia_ffa_fill_notification_map(struct nvidia_ec_ffa_device *ec_ffa_dev) +{ + struct acpi_device *adev = ACPI_COMPANION(&ffa_pdev->dev); + struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL }; + union acpi_object *pkg1, *pkg1_guid; + union acpi_object *pkg2; + union acpi_object *pkg3, *pkg3_prop; + union acpi_object *pkg4, *pkg4_rev, *pkg4_count; + acpi_status status; + int i; + + status = acpi_evaluate_object_typed(adev->handle, "_DSD", NULL, + &output, ACPI_TYPE_PACKAGE); + if (ACPI_FAILURE(status)) { + dev_err(&ffa_pdev->dev, "ACPI _DSD object not found\n"); + return -ENODEV; + } + + pkg1 = output.pointer; + + /* + * _DSD returns a Package() with one or more pairs of elements. + * The first element of each pair is a Universal Unique Identifier (UUID). + * The second element of each pair is another Package() Data Structure. + * + * The _DSD for FFA device will have only one pair of elements so + * pkg1 elements count should be 2. + */ + if (pkg1->package.count != 2) { + kfree(output.pointer); + return -EINVAL; + } + + pkg1_guid = &pkg1->package.elements[0]; + pkg2 = &pkg1->package.elements[1]; + if (pkg1_guid->type != ACPI_TYPE_BUFFER || + pkg1_guid->buffer.length != UUID_SIZE || + pkg2->type != ACPI_TYPE_PACKAGE) { + kfree(output.pointer); + return -EINVAL; + } + + /* Check if GUID macthes with notify device prop GUID */ + if (!guid_equal((guid_t *)pkg1_guid->buffer.pointer, + &nvidia_notify_bind_guid)) { + kfree(output.pointer); + return -EINVAL; + } + + /* pkg3 should conatin 1 element with package type */ + if (pkg2->package.count != 1) { + kfree(output.pointer); + return -EINVAL; + } + + pkg3 = &pkg2->package.elements[0]; + if (pkg3->type != ACPI_TYPE_PACKAGE) { + kfree(output.pointer); + return -EINVAL; + } + + pkg3_prop = &pkg3->package.elements[0]; + if (pkg3_prop->type != ACPI_TYPE_STRING || + strncmp(pkg3_prop->string.pointer, + "arm-arml0002-ffa-ntf-bind", + pkg3_prop->string.length)) { + kfree(output.pointer); + return -EINVAL; + } + + pkg4 = &pkg3->package.elements[1]; + /* + * pkg4 should have minimum 3 elements (revision, count and minimum + * one notification map package) + */ + if (pkg4->type != ACPI_TYPE_PACKAGE || + pkg4->package.count < 3) { + kfree(output.pointer); + return -EINVAL; + } + + pkg4_rev = &pkg4->package.elements[0]; + pkg4_count = &pkg4->package.elements[1]; + + /* Check if revision is 1 */ + if (pkg4_rev->type != ACPI_TYPE_INTEGER || + pkg4_rev->integer.value != 1) { + kfree(output.pointer); + return -EINVAL; + } + + /* + * The pkg4_count represents the count of following packages. + * pkg4_count + 1 (for revision) + 1 (for pkg4_count itself) should + * match total number of elements in pkg4. + */ + if (pkg4_count->type != ACPI_TYPE_INTEGER || + (pkg4_count->integer.value + 2) != pkg4->package.count) { + kfree(output.pointer); + return -EINVAL; + } + + /* + * Traverse the array of notification map packages. + * Each notification map package contains 2 elements, UUID + * and notification ID array package. Check if there is a notification + * map for the FFA device by comparing UUID and update the + * notification_id[] and notification_count. + */ + for (i = 2; i < pkg4->package.count; i++) { + union acpi_object *pkg5_uuid, *pkg5 = &pkg4->package.elements[2]; + union acpi_object *pkg6; + uuid_t uuid; + int j; + + if (pkg5->type != ACPI_TYPE_PACKAGE && + pkg5->package.count != 2) { + kfree(output.pointer); + return -EINVAL; + } + + pkg5_uuid = &pkg5->package.elements[0]; + pkg6 = &pkg5->package.elements[1]; + if (pkg5_uuid->type != ACPI_TYPE_BUFFER || + pkg5_uuid->buffer.length != UUID_SIZE || + pkg6->type != ACPI_TYPE_PACKAGE) { + kfree(output.pointer); + return -EINVAL; + } + + uuid = nvidia_get_uuid_from_aml_buf(pkg5_uuid->buffer.pointer); + if (!uuid_equal(&uuid, &ec_ffa_dev->ffa_dev->uuid)) + continue; + + for (j = 0; j < pkg6->package.count; j++) { + union acpi_object *pkg6_notify_id = &pkg6->package.elements[j]; + + if (pkg6_notify_id->type != ACPI_TYPE_INTEGER) { + kfree(output.pointer); + return -EINVAL; + } + + ec_ffa_dev->notification_id[j] = pkg6_notify_id->integer.value; + } + + ec_ffa_dev->notification_count = pkg6->package.count; + kfree(output.pointer); + return 0; + } + + kfree(output.pointer); + return 0; +} + +/* + * Notification EC service callback. + * Get the ffa device from callback data and invoke notification _DSM with + * notify_id. + * + * The details regarding _DSM is documented in + * https://github.com/OpenDevicePartnership/documentation/tree/main/bookshelf/Shelf%204%20Specifications#notification-events + */ +static void nvidia_ffa_ec_service_notif_callback(int notify_id, void *cb_data) +{ + struct acpi_device *adev = ACPI_COMPANION(&ffa_pdev->dev); + struct ffa_device *ffa_dev = (struct ffa_device *)cb_data; + union acpi_object args[2], input_pkg; + union acpi_object *output; + u8 uuid[UUID_SIZE]; + + nvidia_uuid_to_aml_uuid_buf(&ffa_dev->uuid, uuid); + + args[0].type = ACPI_TYPE_BUFFER; + args[0].buffer.length = sizeof(uuid); + args[0].buffer.pointer = uuid; + + args[1].type = ACPI_TYPE_INTEGER; + args[1].integer.value = notify_id; + + input_pkg.type = ACPI_TYPE_PACKAGE; + input_pkg.package.count = 2; + input_pkg.package.elements = args; + + output = acpi_evaluate_dsm(adev->handle, &nvidia_notify_dsm_guid, + 1, 1, &input_pkg); + if (!output) + dev_err(&ffa_pdev->dev, "Failed to execute notify\n"); + else + ACPI_FREE(output); +} + +/* + * Create notification setup for the notification_id. + * + * The details regarding notification setup is documented in + * https://github.com/OpenDevicePartnership/documentation/tree/main/bookshelf/Shelf%204%20Specifications#register-notification + * + * This function setup 1:1 mapping between hardware notification ID and + * virtual notification ID. + */ +static int nvidia_ffa_notification_setup(struct nvidia_ec_ffa_device *ec_ffa_dev, + u8 notification_id) +{ + struct ffa_send_direct_data2 ffa_data = { 0 }; + u8 *uuid = (u8 *)&ec_ffa_dev->ffa_dev->uuid; + int ret; + + /* X4 register, function 1 */ + ffa_data.data[0] = 1; + + BUILD_BUG_ON(UUID_SIZE != 16); + BUILD_BUG_ON(sizeof(ffa_data.data[1]) < 8); + + /* X5 and X6 registers contain UUID */ + memcpy(&ffa_data.data[1], uuid, 8); + memcpy(&ffa_data.data[2], uuid + 8, 8); + + /* X7 register, the number of notification mappings */ + ffa_data.data[3] = 1; + + /* X7 register, notification ID and notification bitmap bit number */ + ffa_data.data[4] = ((u64)notification_id << 32) | notification_id; + + if (!notify_ffa_dev->ops || + !notify_ffa_dev->ops->msg_ops || + !notify_ffa_dev->ops->msg_ops->sync_send_receive2) { + return -EINVAL; + } + + ret = notify_ffa_dev->ops->msg_ops->sync_send_receive2(notify_ffa_dev, + &ffa_data); + if (ret) { + dev_err(&ec_ffa_dev->ffa_dev->dev, + "Failed to send NOTIFY_SETUP id=%d error=%d\n", + notification_id, ret); + return ret; + } + + if (ffa_data.data[0]) { + dev_err(&ec_ffa_dev->ffa_dev->dev, + "NOTIFY_SETUP returned failure id=%d error=%ld\n", + notification_id, ffa_data.data[0]); + + /* + * TODO: destroy operation is not yet implemented in the firmware + * So, if driver is reloaded, then the previous notification + * still exists and failure will be returned. Once destroy + * is implemented in firmware, update code here to return error + */ + } + + return 0; +} + +/* Destroy notification setup for the notification_id */ +static void nvidia_ffa_notification_destroy(struct nvidia_ec_ffa_device *ec_ffa_dev, + u8 notification_id) +{ + /* + * TODO: destroy operation is not yet implemented in the firmware. + * Once implemented in firmware, update code here. + */ +} + +/* + * Create notifications for the FFA device. + * + * 1. Get notification map array for FFA device. + * 2. For each notification, setup notification with notify service and + * then invoke notify_request method to enable notification for FFA device. + */ +static int nvidia_ffa_create_notifications(struct nvidia_ec_ffa_device *ec_ffa_dev) +{ + int i, ret = 0; + + if (!ec_ffa_dev->ffa_dev->ops || + !ec_ffa_dev->ffa_dev->ops->notifier_ops || + !ec_ffa_dev->ffa_dev->ops->notifier_ops->notify_request || + !ec_ffa_dev->ffa_dev->ops->notifier_ops->notify_relinquish) { + return -EOPNOTSUPP; + } + + ret = nvidia_ffa_fill_notification_map(ec_ffa_dev); + if (ret) { + dev_err(&ffa_pdev->dev, "Error in filling notification map error=%d\n", ret); + return ret; + } + + for (i = 0; i < ec_ffa_dev->notification_count; i++) { + ret = nvidia_ffa_notification_setup(ec_ffa_dev, + ec_ffa_dev->notification_id[i]); + if (ret) { + dev_err(&ec_ffa_dev->ffa_dev->dev, + "Failed to setup notification id=%d error=%d\n", + ec_ffa_dev->notification_id[i], ret); + break; + } + + ret = ec_ffa_dev->ffa_dev->ops->notifier_ops->notify_request( + ec_ffa_dev->ffa_dev, false, + nvidia_ffa_ec_service_notif_callback, + ec_ffa_dev->ffa_dev, ec_ffa_dev->notification_id[i]); + if (ret) { + nvidia_ffa_notification_destroy(ec_ffa_dev, + ec_ffa_dev->notification_id[i]); + dev_err(&ec_ffa_dev->ffa_dev->dev, + "Failed to request notification id=%d error=%d\n", + ec_ffa_dev->notification_id[i], ret); + break; + } + } + + /* Remove already setup notification in case of error */ + if (ret) { + int j; + + for (j = 0; j < i; j++) { + ec_ffa_dev->ffa_dev->ops->notifier_ops->notify_relinquish( + ec_ffa_dev->ffa_dev, + ec_ffa_dev->notification_id[j]); + nvidia_ffa_notification_destroy(ec_ffa_dev, + ec_ffa_dev->notification_id[j]); + } + + ec_ffa_dev->notification_count = 0; + } + + return ret; +} + +/* Remove notifications for the FFA device. */ +static void nvidia_ffa_remove_notifications(struct nvidia_ec_ffa_device *ec_ffa_dev) +{ + int i; + + for (i = 0; i < ec_ffa_dev->notification_count; i++) { + ec_ffa_dev->ffa_dev->ops->notifier_ops->notify_relinquish( + ec_ffa_dev->ffa_dev, + ec_ffa_dev->notification_id[i]); + nvidia_ffa_notification_destroy(ec_ffa_dev, + ec_ffa_dev->notification_id[i]); + } +} + /* * Handler function for FFH operation region offset 4. * When ACPI interpreter runs code with FFH operation region offset 4, @@ -191,9 +602,10 @@ static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) { struct nvidia_ec_ffa_device *nvidia_ec_ffa_dev; const char *acpi_id = NULL; + int ret; - if (!ffa_pdev) { - dev_err(&ffa_dev->dev, "nvidia ffa device not available\n"); + if (!ffa_pdev || !notify_ffa_dev) { + dev_err(&ffa_dev->dev, "nvidia ffa or notify device not available\n"); return -ENODEV; } @@ -208,6 +620,15 @@ static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) nvidia_ec_ffa_dev->ffa_dev = ffa_dev; INIT_LIST_HEAD(&nvidia_ec_ffa_dev->list); + ret = nvidia_ffa_create_notifications(nvidia_ec_ffa_dev); + if (ret) { + dev_info(&ffa_dev->dev, + "Failed to create ffa notifications error=%d\n", + ret); + devm_kfree(&ffa_dev->dev, nvidia_ec_ffa_dev); + return ret; + } + mutex_lock(&nvidia_ffa_lock); list_add(&nvidia_ec_ffa_dev->list, &nvidia_ec_ffa_dev_head); mutex_unlock(&nvidia_ffa_lock); @@ -235,6 +656,7 @@ static void nvidia_ffa_ec_service_remove(struct ffa_device *ffa_dev) list_for_each_entry_safe(cur, tmp, &nvidia_ec_ffa_dev_head, list) { if (cur->ffa_dev == ffa_dev) { list_del(&cur->list); + nvidia_ffa_remove_notifications(cur); devm_kfree(&ffa_dev->dev, cur); break; } @@ -261,6 +683,46 @@ static struct ffa_driver nvidia_ffa_ec_service_driver = { .id_table = nvidia_ffa_ec_service_ids, }; +static int nvidia_ffa_notify_service_probe(struct ffa_device *ffa_dev) +{ + int ret; + + if (!ffa_pdev) { + dev_err(&ffa_dev->dev, "nvidia ffa device not available\n"); + return -ENODEV; + } + + notify_ffa_dev = ffa_dev; + + ret = ffa_driver_register(&nvidia_ffa_ec_service_driver, THIS_MODULE, DRV_NAME); + if (ret) { + dev_err(&ffa_dev->dev, + "Failed to register ec service driver error=%d\n", ret); + notify_ffa_dev = NULL; + return ret; + } + + return 0; +} + +static void nvidia_ffa_notify_service_remove(struct ffa_device *ffa_dev) +{ + ffa_driver_unregister(&nvidia_ffa_ec_service_driver); + notify_ffa_dev = NULL; +} + +static const struct ffa_device_id nvidia_ffa_notify_service_ids[] = { + { nvidia_ec_notify_service_uuid }, + {} +}; + +static struct ffa_driver nvidia_ffa_notify_service_driver = { + .name = "nvidia-ffa-notify", + .probe = nvidia_ffa_notify_service_probe, + .remove = nvidia_ffa_notify_service_remove, + .id_table = nvidia_ffa_notify_service_ids, +}; + static const struct acpi_device_id nvidia_ffa_device_ids[] = { /* * Please refer @@ -313,10 +775,11 @@ static int nvidia_ffa_probe(struct platform_device *pdev) ffa_pdev = pdev; - ret = ffa_driver_register(&nvidia_ffa_ec_service_driver, THIS_MODULE, DRV_NAME); + ret = ffa_driver_register(&nvidia_ffa_notify_service_driver, THIS_MODULE, DRV_NAME); if (ret) { dev_err(&pdev->dev, - "Failed to register ec service driver error=%d\n", ret); + "Failed to register notify service driver error=%d\n", ret); + acpi_arm64_ffh_update_custom_offset_handler(NULL); ffa_pdev = NULL; return ret; } @@ -326,7 +789,7 @@ static int nvidia_ffa_probe(struct platform_device *pdev) static void nvidia_ffa_remove(struct platform_device *pdev) { - ffa_driver_unregister(&nvidia_ffa_ec_service_driver); + ffa_driver_unregister(&nvidia_ffa_notify_service_driver); ffa_pdev = NULL; acpi_arm64_ffh_update_custom_offset_handler(NULL); } From b98ba8cbe0c9cc5d16d1b1e7273a407eb345b1f4 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Fri, 13 Jun 2025 05:42:10 +0000 Subject: [PATCH 052/464] UBUNTU: [Config] nvidia: Update annotations to enable NVIDIA FFA EC driver BugLink: https://bugs.launchpad.net/bugs/2114230 The NVIDIA FFA and EC secure services driver enables the communication with EC (Embedded Controller). Make this driver built-in to enable EC communication at early boot. Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 9ea0251a632d4455a4ae3e9878a928d65f48ad30) (cherry picked from commit 9ea0251a632d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 31b28eae7597f7e0a38375f933e4fd1af9e246d7 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 20ede527308a9..8404149f7f758 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -156,6 +156,9 @@ CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> +CONFIG_NVIDIA_FFA_EC policy<{'arm64': 'y'}> +CONFIG_NVIDIA_FFA_EC note<'LP: #2114230'> + CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> From 051091c733c9b9b66ea9182185803133b66a7caa Mon Sep 17 00:00:00 2001 From: Jonas Chen Date: Mon, 21 Apr 2025 17:17:38 +0800 Subject: [PATCH 053/464] NVIDIA: SAUCE: MEDIATEK: pinctrl: mediatek: Add gpio-range record in pinctrl driver BugLink: https://bugs.launchpad.net/bugs/2117784 Kernel GPIO subsystem mapping hardware pin number to a different range of gpio number. Add gpio-range structure to hold the mapped gpio range in pinctrl driver. That enables the kernel to search a range of mapped gpio range against a pinctrl device. Signed-off-by: Jonas Chen Signed-off-by: Yenchia Chen Signed-off-by: Abhishek Sahu Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Acked-by: nvmochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 1049985ca252 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c113e8d8f903eccbe4fa0c075af1aadd5f4fe014 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.h | 1 + drivers/pinctrl/mediatek/pinctrl-paris.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.h b/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.h index fa7c0ed493464..df8dce14744f9 100644 --- a/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.h +++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.h @@ -302,6 +302,7 @@ struct mtk_pinctrl { spinlock_t lock; /* identify rsel setting by si unit or rsel define in dts node */ bool rsel_si_unit; + struct pinctrl_gpio_range range; }; void mtk_rmw(struct mtk_pinctrl *pctl, u8 i, u32 reg, u32 mask, u32 set); diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c index 6bf37d8085fae..2cf61cfe809ed 100644 --- a/drivers/pinctrl/mediatek/pinctrl-paris.c +++ b/drivers/pinctrl/mediatek/pinctrl-paris.c @@ -3,7 +3,7 @@ * MediaTek Pinctrl Paris Driver, which implement the vendor per-pin * bindings for MediaTek SoC. * - * Copyright (C) 2018 MediaTek Inc. + * Copyright (C) 2018-2025 MediaTek Inc. * Author: Sean Wang * Zhiyong Tao * Hongzhou.Yang @@ -936,6 +936,15 @@ static int mtk_gpio_set_config(struct gpio_chip *chip, unsigned int offset, return mtk_eint_set_debounce(hw->eint, desc->eint.eint_n, debounce); } +static void mtk_pinctrl_gpio_range_init(struct mtk_pinctrl *hw, struct gpio_chip *chip) +{ + hw->range.name = "mtk_pinctrl_gpio_range"; + hw->range.id = 0; + hw->range.pin_base = 0; + hw->range.base = chip->base; + hw->range.npins = hw->soc->npins; +} + static int mtk_build_gpiochip(struct mtk_pinctrl *hw) { struct gpio_chip *chip = &hw->chip; @@ -959,6 +968,8 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw) if (ret < 0) return ret; + mtk_pinctrl_gpio_range_init(hw, chip); + return 0; } @@ -1077,6 +1088,8 @@ int mtk_paris_pinctrl_probe(struct platform_device *pdev) if (err) return dev_err_probe(dev, err, "Failed to add gpio_chip\n"); + pinctrl_add_gpio_range(hw->pctrl, &hw->range); + platform_set_drvdata(pdev, hw); return 0; From 2cf0dbb9fc763bb20ab1d80a671b7972233d8d93 Mon Sep 17 00:00:00 2001 From: Jonas Chen Date: Tue, 22 Apr 2025 09:18:17 +0800 Subject: [PATCH 054/464] NVIDIA: SAUCE: MEDIATEK: pinctrl: mediatek: Add acpi support BugLink: https://bugs.launchpad.net/bugs/2117784 Add acpi support in the shared part of pinctrl driver. Parsing hardware base addresses and irq naumber to initialize eint accroding to the acpi table data. Signed-off-by: Jonas Chen Signed-off-by: Yenchia Chen Signed-off-by: Abhishek Sahu Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Acked-by: nvmochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (backported from commit cdce65d91ea9 noble:linux-nvidia-6.14) [maskedarray: context adjusted due to commit 86dee87: "pinctrl: mediatek: Fix the invalid conditions"] Signed-off-by: Abdur Rahman (cherry picked from commit 84076e8dd9e065d3ab6ce59aec774a14c7f8ff32 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../pinctrl/mediatek/pinctrl-mtk-common-v2.c | 26 +++++++++++++++---- drivers/pinctrl/mediatek/pinctrl-paris.c | 11 +++++--- 2 files changed, 29 insertions(+), 8 deletions(-) diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.c b/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.c index 4918d38abfc29..fc71f9b267c52 100644 --- a/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.c +++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common-v2.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Copyright (C) 2018 MediaTek Inc. + * Copyright (C) 2018-2025 MediaTek Inc. * * Author: Sean Wang * @@ -369,18 +369,30 @@ int mtk_build_eint(struct mtk_pinctrl *hw, struct platform_device *pdev) { struct device_node *np = pdev->dev.of_node; int ret, i, j, count_reg_names; + struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev); + struct resource *res; if (!IS_ENABLED(CONFIG_EINT_MTK)) return 0; - if (!of_property_read_bool(np, "interrupt-controller")) + if (is_of_node(fwnode) && !of_property_read_bool(np, "interrupt-controller")) return -ENODEV; hw->eint = devm_kzalloc(hw->dev, sizeof(*hw->eint), GFP_KERNEL); if (!hw->eint) return -ENOMEM; - count_reg_names = of_property_count_strings(np, "reg-names"); + if (is_of_node(fwnode)) { + count_reg_names = of_property_count_strings(np, "reg-names"); + } else { + count_reg_names = 0; + for (i = 0; i < pdev->num_resources; i++) { + struct resource *r = &pdev->resource[i]; + + if (resource_type(r) == IORESOURCE_MEM) + count_reg_names++; + } + } if (count_reg_names < 0) return -EINVAL; @@ -396,14 +408,18 @@ int mtk_build_eint(struct mtk_pinctrl *hw, struct platform_device *pdev) } for (i = hw->soc->nbase_names, j = 0; i < count_reg_names; i++, j++) { - hw->eint->base[j] = of_iomap(np, i); + res = platform_get_resource(pdev, IORESOURCE_MEM, i); + hw->eint->base[j] = is_of_node(fwnode) ? of_iomap(np, i) : + ioremap(res->start, resource_size(res)); if (IS_ERR(hw->eint->base[j])) { ret = PTR_ERR(hw->eint->base[j]); goto err_free_eint; } } - hw->eint->irq = irq_of_parse_and_map(np, 0); + hw->eint->irq = is_of_node(fwnode) + ? irq_of_parse_and_map(np, 0) + : platform_get_irq(pdev, 0); if (!hw->eint->irq) { ret = -EINVAL; goto err_free_eint; diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c index 2cf61cfe809ed..f74221acba8cb 100644 --- a/drivers/pinctrl/mediatek/pinctrl-paris.c +++ b/drivers/pinctrl/mediatek/pinctrl-paris.c @@ -1008,6 +1008,7 @@ int mtk_paris_pinctrl_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct pinctrl_pin_desc *pins; struct mtk_pinctrl *hw; + struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev); int err, i; hw = devm_kzalloc(&pdev->dev, sizeof(*hw), GFP_KERNEL); @@ -1032,16 +1033,20 @@ int mtk_paris_pinctrl_probe(struct platform_device *pdev) return -ENOMEM; for (i = 0; i < hw->soc->nbase_names; i++) { - hw->base[i] = devm_platform_ioremap_resource_byname(pdev, - hw->soc->base_names[i]); + hw->base[i] = is_of_node(fwnode) + ? devm_platform_ioremap_resource_byname(pdev, hw->soc->base_names[i]) + : devm_platform_get_and_ioremap_resource(pdev, i, NULL); if (IS_ERR(hw->base[i])) return PTR_ERR(hw->base[i]); } hw->nbase = hw->soc->nbase_names; - hw->rsel_si_unit = of_property_read_bool(hw->dev->of_node, + if (is_of_node(fwnode)) + hw->rsel_si_unit = of_property_read_bool(hw->dev->of_node, "mediatek,rsel-resistance-in-si-unit"); + else + hw->rsel_si_unit = false; spin_lock_init(&hw->lock); From eaa518480f156b488736f62af3e7b78d16ab1ad4 Mon Sep 17 00:00:00 2001 From: Jonas Chen Date: Tue, 22 Apr 2025 09:30:44 +0800 Subject: [PATCH 055/464] NVIDIA: SAUCE: MEDIATEK: pinctrl: mt8901: Add pinctrl driver BugLink: https://bugs.launchpad.net/bugs/2117784 Add mt8901 pinctrl, gpio and eint driver implementation. Signed-off-by: Jonas Chen Signed-off-by: Yenchia Chen Signed-off-by: Abhishek Sahu Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Acked-by: nvmochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (backported from commit 1fc7a586c54c noble:linux-nvidia-6.14) [maskedarray: context adjusted for missing commit a3fe132: "pinctrl: mediatek: Add pinctrl driver for mt8189"] Signed-off-by: Abdur Rahman (backported from commit 81bfb0635b75667abd430efc1d2ea50813b0fdf5 noble:linux-nvidia-6.17) [jacobmartin: context adjusted for new pinctrl-mt8901 driver from upstream] Signed-off-by: Jacob Martin --- drivers/pinctrl/mediatek/Kconfig | 12 + drivers/pinctrl/mediatek/Makefile | 1 + drivers/pinctrl/mediatek/mtk-eint.c | 4 + drivers/pinctrl/mediatek/mtk-eint.h | 1 + drivers/pinctrl/mediatek/pinctrl-mt8901.c | 1460 +++++++++++ drivers/pinctrl/mediatek/pinctrl-mtk-mt8901.h | 2130 +++++++++++++++++ 6 files changed, 3608 insertions(+) create mode 100644 drivers/pinctrl/mediatek/pinctrl-mt8901.c create mode 100644 drivers/pinctrl/mediatek/pinctrl-mtk-mt8901.h diff --git a/drivers/pinctrl/mediatek/Kconfig b/drivers/pinctrl/mediatek/Kconfig index 4819617d93683..92f4f394b71e7 100644 --- a/drivers/pinctrl/mediatek/Kconfig +++ b/drivers/pinctrl/mediatek/Kconfig @@ -281,6 +281,18 @@ config PINCTRL_MT8189 In MTK platform, we support virtual gpio and use it to map specific eint which doesn't have real gpio pin. +config PINCTRL_MT8901 + bool "MediaTek MT8901 pin control" + depends on ACPI + depends on ARM64 || COMPILE_TEST + default ARM64 && ARCH_MEDIATEK + select PINCTRL_MTK_PARIS + help + Say yes here to support pin controller and gpio driver + on MediaTek MT8901 SoC. + In MTK platform, we support virtual gpio and use it to + map specific eint which doesn't have real gpio pin. + config PINCTRL_MT8192 bool "MediaTek MT8192 pin control" depends on OF diff --git a/drivers/pinctrl/mediatek/Makefile b/drivers/pinctrl/mediatek/Makefile index ae765bd999657..57c69b1e5c2d4 100644 --- a/drivers/pinctrl/mediatek/Makefile +++ b/drivers/pinctrl/mediatek/Makefile @@ -43,3 +43,4 @@ obj-$(CONFIG_PINCTRL_MT8196) += pinctrl-mt8196.o obj-$(CONFIG_PINCTRL_MT8365) += pinctrl-mt8365.o obj-$(CONFIG_PINCTRL_MT8516) += pinctrl-mt8516.o obj-$(CONFIG_PINCTRL_MT6397) += pinctrl-mt6397.o +obj-$(CONFIG_PINCTRL_MT8901) += pinctrl-mt8901.o diff --git a/drivers/pinctrl/mediatek/mtk-eint.c b/drivers/pinctrl/mediatek/mtk-eint.c index 2a3c04eedc5f3..3e6b121cf593a 100644 --- a/drivers/pinctrl/mediatek/mtk-eint.c +++ b/drivers/pinctrl/mediatek/mtk-eint.c @@ -71,6 +71,10 @@ const unsigned int debounce_time_mt6878[] = { }; EXPORT_SYMBOL_GPL(debounce_time_mt6878); +const unsigned int debounce_time_mt8901[] = { + 156, 313, 625, 1250, 20000, 40000, 80000, 160000, 320000, 640000, 0}; +EXPORT_SYMBOL_GPL(debounce_time_mt8901); + static void __iomem *mtk_eint_get_offset(struct mtk_eint *eint, unsigned int eint_num, unsigned int offset) diff --git a/drivers/pinctrl/mediatek/mtk-eint.h b/drivers/pinctrl/mediatek/mtk-eint.h index 3cdd6f6310cd0..1b185f660affa 100644 --- a/drivers/pinctrl/mediatek/mtk-eint.h +++ b/drivers/pinctrl/mediatek/mtk-eint.h @@ -53,6 +53,7 @@ extern const unsigned int debounce_time_mt2701[]; extern const unsigned int debounce_time_mt6765[]; extern const unsigned int debounce_time_mt6795[]; extern const unsigned int debounce_time_mt6878[]; +extern const unsigned int debounce_time_mt8901[]; struct mtk_eint; diff --git a/drivers/pinctrl/mediatek/pinctrl-mt8901.c b/drivers/pinctrl/mediatek/pinctrl-mt8901.c new file mode 100644 index 0000000000000..623cd0cd58f05 --- /dev/null +++ b/drivers/pinctrl/mediatek/pinctrl-mt8901.c @@ -0,0 +1,1460 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2025 MediaTek Inc. + * + */ + +#include +#include +#include "pinctrl-mtk-mt8901.h" +#include "pinctrl-paris.h" + +#define PIN_FIELD_BASE(s_pin, e_pin, i_base, s_addr, x_addrs, s_bit, x_bits) \ + PIN_FIELD_CALC(s_pin, e_pin, i_base, s_addr, x_addrs, s_bit, x_bits, 32, 0) +#define PINS_FIELD_BASE(s_pin, e_pin, i_base, s_addr, x_addrs, s_bit, x_bits) \ + PIN_FIELD_CALC(s_pin, e_pin, i_base, s_addr, x_addrs, s_bit, x_bits, 32, 1) + +static const struct mtk_pin_field_calc mt8901_pin_mode_range[] = { + PIN_FIELD(0, 181, 0x0300, 0x10, 0, 4), +}; + +static const struct mtk_pin_field_calc mt8901_pin_dir_range[] = { + PIN_FIELD(0, 181, 0x0000, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_di_range[] = { + PIN_FIELD(0, 181, 0x0200, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_do_range[] = { + PIN_FIELD(0, 181, 0x0100, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_smt_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(1, 1, 8, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(2, 2, 8, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(3, 3, 8, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(4, 4, 8, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(5, 5, 8, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(6, 6, 8, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(7, 7, 8, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(8, 8, 2, 0x0120, 0x10, 13, 1), + PIN_FIELD_BASE(9, 9, 2, 0x0120, 0x10, 14, 1), + PIN_FIELD_BASE(10, 10, 2, 0x0120, 0x10, 15, 1), + PIN_FIELD_BASE(11, 11, 2, 0x0120, 0x10, 16, 1), + PIN_FIELD_BASE(12, 12, 1, 0x0140, 0x10, 17, 1), + PIN_FIELD_BASE(13, 13, 1, 0x0140, 0x10, 18, 1), + PIN_FIELD_BASE(14, 14, 1, 0x0140, 0x10, 14, 1), + PIN_FIELD_BASE(15, 15, 1, 0x0140, 0x10, 16, 1), + PIN_FIELD_BASE(16, 16, 1, 0x0140, 0x10, 19, 1), + PIN_FIELD_BASE(17, 17, 1, 0x0140, 0x10, 20, 1), + PIN_FIELD_BASE(18, 18, 1, 0x0140, 0x10, 21, 1), + PIN_FIELD_BASE(19, 19, 1, 0x0140, 0x10, 27, 1), + PIN_FIELD_BASE(20, 20, 1, 0x0140, 0x10, 28, 1), + PIN_FIELD_BASE(21, 21, 1, 0x0140, 0x10, 26, 1), + PIN_FIELD_BASE(22, 22, 1, 0x0140, 0x10, 25, 1), + PIN_FIELD_BASE(23, 23, 1, 0x0140, 0x10, 0, 1), + PIN_FIELD_BASE(24, 24, 1, 0x0140, 0x10, 1, 1), + PIN_FIELD_BASE(25, 25, 1, 0x0140, 0x10, 2, 1), + PIN_FIELD_BASE(26, 26, 1, 0x0140, 0x10, 3, 1), + PIN_FIELD_BASE(27, 27, 1, 0x0140, 0x10, 4, 1), + PIN_FIELD_BASE(28, 28, 1, 0x0140, 0x10, 5, 1), + PIN_FIELD_BASE(29, 29, 1, 0x0140, 0x10, 6, 1), + PIN_FIELD_BASE(30, 30, 1, 0x0140, 0x10, 7, 1), + PIN_FIELD_BASE(31, 31, 1, 0x0140, 0x10, 8, 1), + PIN_FIELD_BASE(32, 32, 9, 0x0100, 0x10, 0, 1), + PIN_FIELD_BASE(33, 33, 1, 0x0140, 0x10, 9, 1), + PIN_FIELD_BASE(34, 34, 1, 0x0140, 0x10, 10, 1), + PIN_FIELD_BASE(35, 35, 1, 0x0140, 0x10, 11, 1), + PIN_FIELD_BASE(36, 36, 9, 0x0100, 0x10, 8, 1), + PIN_FIELD_BASE(37, 37, 2, 0x0120, 0x10, 17, 1), + PIN_FIELD_BASE(38, 38, 2, 0x0120, 0x10, 18, 1), + PIN_FIELD_BASE(39, 39, 1, 0x0140, 0x10, 12, 1), + PIN_FIELD_BASE(40, 40, 2, 0x0120, 0x10, 1, 1), + PIN_FIELD_BASE(41, 41, 2, 0x0120, 0x10, 2, 1), + PIN_FIELD_BASE(42, 42, 2, 0x0120, 0x10, 3, 1), + PIN_FIELD_BASE(43, 43, 2, 0x0120, 0x10, 4, 1), + PIN_FIELD_BASE(44, 44, 2, 0x0120, 0x10, 5, 1), + PIN_FIELD_BASE(45, 45, 2, 0x0120, 0x10, 6, 1), + PIN_FIELD_BASE(46, 46, 1, 0x0140, 0x10, 13, 1), + PIN_FIELD_BASE(47, 47, 1, 0x0140, 0x10, 15, 1), + PIN_FIELD_BASE(48, 48, 2, 0x0120, 0x10, 7, 1), + PIN_FIELD_BASE(49, 49, 2, 0x0120, 0x10, 8, 1), + PIN_FIELD_BASE(50, 50, 2, 0x0120, 0x10, 9, 1), + PIN_FIELD_BASE(51, 51, 2, 0x0120, 0x10, 10, 1), + PIN_FIELD_BASE(52, 52, 2, 0x0120, 0x10, 11, 1), + PIN_FIELD_BASE(53, 53, 2, 0x0120, 0x10, 12, 1), + PIN_FIELD_BASE(54, 54, 5, 0x0120, 0x10, 10, 1), + PIN_FIELD_BASE(55, 55, 5, 0x0120, 0x10, 11, 1), + PIN_FIELD_BASE(56, 56, 1, 0x0140, 0x10, 22, 1), + PIN_FIELD_BASE(57, 57, 1, 0x0140, 0x10, 23, 1), + PIN_FIELD_BASE(58, 58, 1, 0x0140, 0x10, 24, 1), + PIN_FIELD_BASE(59, 59, 2, 0x0120, 0x10, 0, 1), + PIN_FIELD_BASE(60, 60, 9, 0x0100, 0x10, 1, 1), + PIN_FIELD_BASE(61, 61, 9, 0x0100, 0x10, 2, 1), + PIN_FIELD_BASE(62, 62, 9, 0x0100, 0x10, 3, 1), + PIN_FIELD_BASE(63, 63, 9, 0x0100, 0x10, 4, 1), + PIN_FIELD_BASE(64, 64, 9, 0x0100, 0x10, 5, 1), + PIN_FIELD_BASE(65, 65, 9, 0x0100, 0x10, 6, 1), + PIN_FIELD_BASE(66, 66, 5, 0x0120, 0x10, 0, 1), + PIN_FIELD_BASE(67, 67, 5, 0x0120, 0x10, 1, 1), + PIN_FIELD_BASE(68, 68, 9, 0x0100, 0x10, 7, 1), + PIN_FIELD_BASE(69, 69, 7, 0x0110, 0x10, 0, 1), + PIN_FIELD_BASE(70, 70, 7, 0x0110, 0x10, 1, 1), + PIN_FIELD_BASE(71, 71, 7, 0x0110, 0x10, 2, 1), + PIN_FIELD_BASE(72, 72, 7, 0x0110, 0x10, 3, 1), + PIN_FIELD_BASE(73, 73, 7, 0x0110, 0x10, 4, 1), + PIN_FIELD_BASE(74, 74, 7, 0x0110, 0x10, 5, 1), + PIN_FIELD_BASE(75, 75, 7, 0x0110, 0x10, 6, 1), + PIN_FIELD_BASE(76, 76, 7, 0x0110, 0x10, 7, 1), + PIN_FIELD_BASE(77, 77, 7, 0x0110, 0x10, 8, 1), + PIN_FIELD_BASE(78, 78, 7, 0x0110, 0x10, 9, 1), + PIN_FIELD_BASE(79, 79, 7, 0x0110, 0x10, 10, 1), + PIN_FIELD_BASE(80, 80, 7, 0x0110, 0x10, 11, 1), + PIN_FIELD_BASE(81, 81, 7, 0x0110, 0x10, 12, 1), + PIN_FIELD_BASE(82, 82, 7, 0x0110, 0x10, 13, 1), + PIN_FIELD_BASE(83, 83, 7, 0x0110, 0x10, 14, 1), + PIN_FIELD_BASE(84, 84, 7, 0x0110, 0x10, 15, 1), + PIN_FIELD_BASE(85, 85, 7, 0x0110, 0x10, 16, 1), + PIN_FIELD_BASE(86, 86, 7, 0x0110, 0x10, 17, 1), + PIN_FIELD_BASE(87, 87, 7, 0x0110, 0x10, 18, 1), + PIN_FIELD_BASE(88, 88, 7, 0x0110, 0x10, 19, 1), + PIN_FIELD_BASE(89, 89, 7, 0x0110, 0x10, 20, 1), + PIN_FIELD_BASE(90, 90, 7, 0x0110, 0x10, 21, 1), + PIN_FIELD_BASE(91, 91, 7, 0x0110, 0x10, 22, 1), + PIN_FIELD_BASE(92, 92, 3, 0x0130, 0x10, 16, 1), + PIN_FIELD_BASE(93, 93, 3, 0x0130, 0x10, 17, 1), + PIN_FIELD_BASE(94, 94, 3, 0x0130, 0x10, 18, 1), + PIN_FIELD_BASE(95, 95, 3, 0x0130, 0x10, 19, 1), + PIN_FIELD_BASE(96, 96, 3, 0x0130, 0x10, 20, 1), + PIN_FIELD_BASE(97, 97, 3, 0x0130, 0x10, 21, 1), + PIN_FIELD_BASE(98, 98, 3, 0x0130, 0x10, 22, 1), + PIN_FIELD_BASE(99, 99, 3, 0x0130, 0x10, 23, 1), + PIN_FIELD_BASE(100, 100, 3, 0x0130, 0x10, 24, 1), + PIN_FIELD_BASE(101, 101, 3, 0x0130, 0x10, 25, 1), + PIN_FIELD_BASE(102, 102, 3, 0x0130, 0x10, 26, 1), + PIN_FIELD_BASE(103, 103, 3, 0x0130, 0x10, 27, 1), + PIN_FIELD_BASE(104, 104, 3, 0x0130, 0x10, 28, 1), + PIN_FIELD_BASE(105, 105, 3, 0x0130, 0x10, 29, 1), + PIN_FIELD_BASE(106, 106, 8, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(107, 107, 8, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(108, 108, 8, 0x00c0, 0x10, 10, 1), + PIN_FIELD_BASE(109, 109, 8, 0x00c0, 0x10, 11, 1), + PIN_FIELD_BASE(110, 110, 8, 0x00c0, 0x10, 12, 1), + PIN_FIELD_BASE(111, 111, 8, 0x00c0, 0x10, 13, 1), + PIN_FIELD_BASE(112, 112, 5, 0x0120, 0x10, 15, 1), + PIN_FIELD_BASE(113, 113, 5, 0x0120, 0x10, 16, 1), + PIN_FIELD_BASE(114, 114, 5, 0x0120, 0x10, 17, 1), + PIN_FIELD_BASE(115, 115, 5, 0x0120, 0x10, 18, 1), + PIN_FIELD_BASE(116, 116, 5, 0x0120, 0x10, 19, 1), + PIN_FIELD_BASE(117, 117, 4, 0x0110, 0x10, 8, 1), + PIN_FIELD_BASE(118, 118, 4, 0x0110, 0x10, 9, 1), + PIN_FIELD_BASE(119, 119, 4, 0x0110, 0x10, 10, 1), + PIN_FIELD_BASE(120, 120, 4, 0x0110, 0x10, 11, 1), + PIN_FIELD_BASE(121, 121, 4, 0x0110, 0x10, 12, 1), + PIN_FIELD_BASE(122, 122, 4, 0x0110, 0x10, 13, 1), + PIN_FIELD_BASE(123, 123, 4, 0x0110, 0x10, 14, 1), + PIN_FIELD_BASE(124, 124, 4, 0x0110, 0x10, 15, 1), + PIN_FIELD_BASE(125, 125, 4, 0x0110, 0x10, 16, 1), + PIN_FIELD_BASE(126, 126, 5, 0x0120, 0x10, 6, 1), + PIN_FIELD_BASE(127, 127, 5, 0x0120, 0x10, 7, 1), + PIN_FIELD_BASE(128, 128, 5, 0x0120, 0x10, 8, 1), + PIN_FIELD_BASE(129, 129, 5, 0x0120, 0x10, 9, 1), + PIN_FIELD_BASE(130, 130, 4, 0x0110, 0x10, 17, 1), + PIN_FIELD_BASE(131, 131, 4, 0x0110, 0x10, 18, 1), + PIN_FIELD_BASE(132, 132, 4, 0x0110, 0x10, 19, 1), + PIN_FIELD_BASE(133, 133, 4, 0x0110, 0x10, 20, 1), + PIN_FIELD_BASE(134, 134, 3, 0x0130, 0x10, 0, 1), + PIN_FIELD_BASE(135, 135, 3, 0x0130, 0x10, 1, 1), + PIN_FIELD_BASE(136, 136, 3, 0x0130, 0x10, 2, 1), + PIN_FIELD_BASE(137, 137, 3, 0x0130, 0x10, 3, 1), + PIN_FIELD_BASE(138, 138, 3, 0x0130, 0x10, 4, 1), + PIN_FIELD_BASE(139, 139, 3, 0x0130, 0x10, 5, 1), + PIN_FIELD_BASE(140, 140, 3, 0x0130, 0x10, 6, 1), + PIN_FIELD_BASE(141, 141, 3, 0x0130, 0x10, 7, 1), + PIN_FIELD_BASE(142, 142, 3, 0x0130, 0x10, 8, 1), + PIN_FIELD_BASE(143, 143, 3, 0x0130, 0x10, 9, 1), + PIN_FIELD_BASE(144, 144, 3, 0x0130, 0x10, 10, 1), + PIN_FIELD_BASE(145, 145, 3, 0x0130, 0x10, 11, 1), + PIN_FIELD_BASE(146, 146, 3, 0x0130, 0x10, 12, 1), + PIN_FIELD_BASE(147, 147, 3, 0x0130, 0x10, 13, 1), + PIN_FIELD_BASE(148, 148, 3, 0x0130, 0x10, 14, 1), + PIN_FIELD_BASE(149, 149, 3, 0x0130, 0x10, 15, 1), + PIN_FIELD_BASE(150, 150, 4, 0x0110, 0x10, 21, 1), + PIN_FIELD_BASE(151, 151, 4, 0x0110, 0x10, 26, 1), + PIN_FIELD_BASE(152, 152, 4, 0x0110, 0x10, 25, 1), + PIN_FIELD_BASE(153, 153, 4, 0x0110, 0x10, 24, 1), + PIN_FIELD_BASE(154, 154, 4, 0x0110, 0x10, 22, 1), + PIN_FIELD_BASE(155, 155, 4, 0x0110, 0x10, 23, 1), + PIN_FIELD_BASE(156, 156, 4, 0x0110, 0x10, 0, 1), + PIN_FIELD_BASE(157, 157, 4, 0x0110, 0x10, 1, 1), + PIN_FIELD_BASE(158, 158, 4, 0x0110, 0x10, 2, 1), + PIN_FIELD_BASE(159, 159, 4, 0x0110, 0x10, 3, 1), + PIN_FIELD_BASE(160, 160, 4, 0x0110, 0x10, 4, 1), + PIN_FIELD_BASE(161, 161, 4, 0x0110, 0x10, 5, 1), + PIN_FIELD_BASE(162, 162, 4, 0x0110, 0x10, 6, 1), + PIN_FIELD_BASE(163, 163, 4, 0x0110, 0x10, 7, 1), + PIN_FIELD_BASE(164, 164, 5, 0x0120, 0x10, 12, 1), + PIN_FIELD_BASE(165, 165, 5, 0x0120, 0x10, 3, 1), + PIN_FIELD_BASE(166, 166, 5, 0x0120, 0x10, 4, 1), + PIN_FIELD_BASE(167, 167, 5, 0x0120, 0x10, 2, 1), + PIN_FIELD_BASE(168, 168, 5, 0x0120, 0x10, 13, 1), + PIN_FIELD_BASE(169, 169, 5, 0x0120, 0x10, 14, 1), + PIN_FIELD_BASE(170, 170, 5, 0x0120, 0x10, 5, 1), + PIN_FIELD_BASE(171, 171, 6, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(172, 172, 6, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(173, 173, 6, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(174, 174, 6, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(175, 175, 6, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(176, 176, 6, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(177, 177, 6, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(178, 178, 6, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(179, 179, 6, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(180, 180, 6, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(181, 181, 10, 0x0080, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_ies_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x0050, 0x10, 0, 1), + PIN_FIELD_BASE(1, 1, 8, 0x0050, 0x10, 1, 1), + PIN_FIELD_BASE(2, 2, 8, 0x0050, 0x10, 2, 1), + PIN_FIELD_BASE(3, 3, 8, 0x0050, 0x10, 3, 1), + PIN_FIELD_BASE(4, 4, 8, 0x0050, 0x10, 4, 1), + PIN_FIELD_BASE(5, 5, 8, 0x0050, 0x10, 5, 1), + PIN_FIELD_BASE(6, 6, 8, 0x0050, 0x10, 6, 1), + PIN_FIELD_BASE(7, 7, 8, 0x0050, 0x10, 7, 1), + PIN_FIELD_BASE(8, 8, 2, 0x0070, 0x10, 13, 1), + PIN_FIELD_BASE(9, 9, 2, 0x0070, 0x10, 14, 1), + PIN_FIELD_BASE(10, 10, 2, 0x0070, 0x10, 15, 1), + PIN_FIELD_BASE(11, 11, 2, 0x0070, 0x10, 16, 1), + PIN_FIELD_BASE(12, 12, 1, 0x0080, 0x10, 17, 1), + PIN_FIELD_BASE(13, 13, 1, 0x0080, 0x10, 18, 1), + PIN_FIELD_BASE(14, 14, 1, 0x0080, 0x10, 14, 1), + PIN_FIELD_BASE(15, 15, 1, 0x0080, 0x10, 16, 1), + PIN_FIELD_BASE(16, 16, 1, 0x0080, 0x10, 19, 1), + PIN_FIELD_BASE(17, 17, 1, 0x0080, 0x10, 20, 1), + PIN_FIELD_BASE(18, 18, 1, 0x0080, 0x10, 21, 1), + PIN_FIELD_BASE(19, 19, 1, 0x0080, 0x10, 27, 1), + PIN_FIELD_BASE(20, 20, 1, 0x0080, 0x10, 28, 1), + PIN_FIELD_BASE(21, 21, 1, 0x0080, 0x10, 26, 1), + PIN_FIELD_BASE(22, 22, 1, 0x0080, 0x10, 25, 1), + PIN_FIELD_BASE(23, 23, 1, 0x0080, 0x10, 0, 1), + PIN_FIELD_BASE(24, 24, 1, 0x0080, 0x10, 1, 1), + PIN_FIELD_BASE(25, 25, 1, 0x0080, 0x10, 2, 1), + PIN_FIELD_BASE(26, 26, 1, 0x0080, 0x10, 3, 1), + PIN_FIELD_BASE(27, 27, 1, 0x0080, 0x10, 4, 1), + PIN_FIELD_BASE(28, 28, 1, 0x0080, 0x10, 5, 1), + PIN_FIELD_BASE(29, 29, 1, 0x0080, 0x10, 6, 1), + PIN_FIELD_BASE(30, 30, 1, 0x0080, 0x10, 7, 1), + PIN_FIELD_BASE(31, 31, 1, 0x0080, 0x10, 8, 1), + PIN_FIELD_BASE(32, 32, 9, 0x0060, 0x10, 0, 1), + PIN_FIELD_BASE(33, 33, 1, 0x0080, 0x10, 9, 1), + PIN_FIELD_BASE(34, 34, 1, 0x0080, 0x10, 10, 1), + PIN_FIELD_BASE(35, 35, 1, 0x0080, 0x10, 11, 1), + PIN_FIELD_BASE(36, 36, 9, 0x0060, 0x10, 8, 1), + PIN_FIELD_BASE(37, 37, 2, 0x0070, 0x10, 17, 1), + PIN_FIELD_BASE(38, 38, 2, 0x0070, 0x10, 18, 1), + PIN_FIELD_BASE(39, 39, 1, 0x0080, 0x10, 12, 1), + PIN_FIELD_BASE(40, 40, 2, 0x0070, 0x10, 1, 1), + PIN_FIELD_BASE(41, 41, 2, 0x0070, 0x10, 2, 1), + PIN_FIELD_BASE(42, 42, 2, 0x0070, 0x10, 3, 1), + PIN_FIELD_BASE(43, 43, 2, 0x0070, 0x10, 4, 1), + PIN_FIELD_BASE(44, 44, 2, 0x0070, 0x10, 5, 1), + PIN_FIELD_BASE(45, 45, 2, 0x0070, 0x10, 6, 1), + PIN_FIELD_BASE(46, 46, 1, 0x0080, 0x10, 13, 1), + PIN_FIELD_BASE(47, 47, 1, 0x0080, 0x10, 15, 1), + PIN_FIELD_BASE(48, 48, 2, 0x0070, 0x10, 7, 1), + PIN_FIELD_BASE(49, 49, 2, 0x0070, 0x10, 8, 1), + PIN_FIELD_BASE(50, 50, 2, 0x0070, 0x10, 9, 1), + PIN_FIELD_BASE(51, 51, 2, 0x0070, 0x10, 10, 1), + PIN_FIELD_BASE(52, 52, 2, 0x0070, 0x10, 11, 1), + PIN_FIELD_BASE(53, 53, 2, 0x0070, 0x10, 12, 1), + PIN_FIELD_BASE(54, 54, 5, 0x0060, 0x10, 10, 1), + PIN_FIELD_BASE(55, 55, 5, 0x0060, 0x10, 11, 1), + PIN_FIELD_BASE(56, 56, 1, 0x0080, 0x10, 22, 1), + PIN_FIELD_BASE(57, 57, 1, 0x0080, 0x10, 23, 1), + PIN_FIELD_BASE(58, 58, 1, 0x0080, 0x10, 24, 1), + PIN_FIELD_BASE(59, 59, 2, 0x0070, 0x10, 0, 1), + PIN_FIELD_BASE(60, 60, 9, 0x0060, 0x10, 1, 1), + PIN_FIELD_BASE(61, 61, 9, 0x0060, 0x10, 2, 1), + PIN_FIELD_BASE(62, 62, 9, 0x0060, 0x10, 3, 1), + PIN_FIELD_BASE(63, 63, 9, 0x0060, 0x10, 4, 1), + PIN_FIELD_BASE(64, 64, 9, 0x0060, 0x10, 5, 1), + PIN_FIELD_BASE(65, 65, 9, 0x0060, 0x10, 6, 1), + PIN_FIELD_BASE(66, 66, 5, 0x0060, 0x10, 0, 1), + PIN_FIELD_BASE(67, 67, 5, 0x0060, 0x10, 1, 1), + PIN_FIELD_BASE(68, 68, 9, 0x0060, 0x10, 7, 1), + PIN_FIELD_BASE(69, 69, 7, 0x0070, 0x10, 0, 1), + PIN_FIELD_BASE(70, 70, 7, 0x0070, 0x10, 1, 1), + PIN_FIELD_BASE(71, 71, 7, 0x0070, 0x10, 2, 1), + PIN_FIELD_BASE(72, 72, 7, 0x0070, 0x10, 3, 1), + PIN_FIELD_BASE(73, 73, 7, 0x0070, 0x10, 4, 1), + PIN_FIELD_BASE(74, 74, 7, 0x0070, 0x10, 5, 1), + PIN_FIELD_BASE(75, 75, 7, 0x0070, 0x10, 6, 1), + PIN_FIELD_BASE(76, 76, 7, 0x0070, 0x10, 7, 1), + PIN_FIELD_BASE(77, 77, 7, 0x0070, 0x10, 8, 1), + PIN_FIELD_BASE(78, 78, 7, 0x0070, 0x10, 9, 1), + PIN_FIELD_BASE(79, 79, 7, 0x0070, 0x10, 10, 1), + PIN_FIELD_BASE(80, 80, 7, 0x0070, 0x10, 11, 1), + PIN_FIELD_BASE(81, 81, 7, 0x0070, 0x10, 12, 1), + PIN_FIELD_BASE(82, 82, 7, 0x0070, 0x10, 13, 1), + PIN_FIELD_BASE(83, 83, 7, 0x0070, 0x10, 14, 1), + PIN_FIELD_BASE(84, 84, 7, 0x0070, 0x10, 15, 1), + PIN_FIELD_BASE(85, 85, 7, 0x0070, 0x10, 16, 1), + PIN_FIELD_BASE(86, 86, 7, 0x0070, 0x10, 17, 1), + PIN_FIELD_BASE(87, 87, 7, 0x0070, 0x10, 18, 1), + PIN_FIELD_BASE(88, 88, 7, 0x0070, 0x10, 19, 1), + PIN_FIELD_BASE(89, 89, 7, 0x0070, 0x10, 20, 1), + PIN_FIELD_BASE(90, 90, 7, 0x0070, 0x10, 21, 1), + PIN_FIELD_BASE(91, 91, 7, 0x0070, 0x10, 22, 1), + PIN_FIELD_BASE(92, 92, 3, 0x0080, 0x10, 16, 1), + PIN_FIELD_BASE(93, 93, 3, 0x0080, 0x10, 17, 1), + PIN_FIELD_BASE(94, 94, 3, 0x0080, 0x10, 18, 1), + PIN_FIELD_BASE(95, 95, 3, 0x0080, 0x10, 19, 1), + PIN_FIELD_BASE(96, 96, 3, 0x0080, 0x10, 20, 1), + PIN_FIELD_BASE(97, 97, 3, 0x0080, 0x10, 21, 1), + PIN_FIELD_BASE(98, 98, 3, 0x0080, 0x10, 22, 1), + PIN_FIELD_BASE(99, 99, 3, 0x0080, 0x10, 23, 1), + PIN_FIELD_BASE(100, 100, 3, 0x0080, 0x10, 24, 1), + PIN_FIELD_BASE(101, 101, 3, 0x0080, 0x10, 25, 1), + PIN_FIELD_BASE(102, 102, 3, 0x0080, 0x10, 26, 1), + PIN_FIELD_BASE(103, 103, 3, 0x0080, 0x10, 27, 1), + PIN_FIELD_BASE(104, 104, 3, 0x0080, 0x10, 28, 1), + PIN_FIELD_BASE(105, 105, 3, 0x0080, 0x10, 29, 1), + PIN_FIELD_BASE(106, 106, 8, 0x0050, 0x10, 8, 1), + PIN_FIELD_BASE(107, 107, 8, 0x0050, 0x10, 9, 1), + PIN_FIELD_BASE(108, 108, 8, 0x0050, 0x10, 10, 1), + PIN_FIELD_BASE(109, 109, 8, 0x0050, 0x10, 11, 1), + PIN_FIELD_BASE(110, 110, 8, 0x0050, 0x10, 12, 1), + PIN_FIELD_BASE(111, 111, 8, 0x0050, 0x10, 13, 1), + PIN_FIELD_BASE(112, 112, 5, 0x0060, 0x10, 15, 1), + PIN_FIELD_BASE(113, 113, 5, 0x0060, 0x10, 16, 1), + PIN_FIELD_BASE(114, 114, 5, 0x0060, 0x10, 17, 1), + PIN_FIELD_BASE(115, 115, 5, 0x0060, 0x10, 18, 1), + PIN_FIELD_BASE(116, 116, 5, 0x0060, 0x10, 19, 1), + PIN_FIELD_BASE(117, 117, 4, 0x0060, 0x10, 8, 1), + PIN_FIELD_BASE(118, 118, 4, 0x0060, 0x10, 9, 1), + PIN_FIELD_BASE(119, 119, 4, 0x0060, 0x10, 10, 1), + PIN_FIELD_BASE(120, 120, 4, 0x0060, 0x10, 11, 1), + PIN_FIELD_BASE(121, 121, 4, 0x0060, 0x10, 12, 1), + PIN_FIELD_BASE(122, 122, 4, 0x0060, 0x10, 13, 1), + PIN_FIELD_BASE(123, 123, 4, 0x0060, 0x10, 14, 1), + PIN_FIELD_BASE(124, 124, 4, 0x0060, 0x10, 15, 1), + PIN_FIELD_BASE(125, 125, 4, 0x0060, 0x10, 16, 1), + PIN_FIELD_BASE(126, 126, 5, 0x0060, 0x10, 6, 1), + PIN_FIELD_BASE(127, 127, 5, 0x0060, 0x10, 7, 1), + PIN_FIELD_BASE(128, 128, 5, 0x0060, 0x10, 8, 1), + PIN_FIELD_BASE(129, 129, 5, 0x0060, 0x10, 9, 1), + PIN_FIELD_BASE(130, 130, 4, 0x0060, 0x10, 17, 1), + PIN_FIELD_BASE(131, 131, 4, 0x0060, 0x10, 18, 1), + PIN_FIELD_BASE(132, 132, 4, 0x0060, 0x10, 19, 1), + PIN_FIELD_BASE(133, 133, 4, 0x0060, 0x10, 20, 1), + PIN_FIELD_BASE(134, 134, 3, 0x0080, 0x10, 0, 1), + PIN_FIELD_BASE(135, 135, 3, 0x0080, 0x10, 1, 1), + PIN_FIELD_BASE(136, 136, 3, 0x0080, 0x10, 2, 1), + PIN_FIELD_BASE(137, 137, 3, 0x0080, 0x10, 3, 1), + PIN_FIELD_BASE(138, 138, 3, 0x0080, 0x10, 4, 1), + PIN_FIELD_BASE(139, 139, 3, 0x0080, 0x10, 5, 1), + PIN_FIELD_BASE(140, 140, 3, 0x0080, 0x10, 6, 1), + PIN_FIELD_BASE(141, 141, 3, 0x0080, 0x10, 7, 1), + PIN_FIELD_BASE(142, 142, 3, 0x0080, 0x10, 8, 1), + PIN_FIELD_BASE(143, 143, 3, 0x0080, 0x10, 9, 1), + PIN_FIELD_BASE(144, 144, 3, 0x0080, 0x10, 10, 1), + PIN_FIELD_BASE(145, 145, 3, 0x0080, 0x10, 11, 1), + PIN_FIELD_BASE(146, 146, 3, 0x0080, 0x10, 12, 1), + PIN_FIELD_BASE(147, 147, 3, 0x0080, 0x10, 13, 1), + PIN_FIELD_BASE(148, 148, 3, 0x0080, 0x10, 14, 1), + PIN_FIELD_BASE(149, 149, 3, 0x0080, 0x10, 15, 1), + PIN_FIELD_BASE(150, 150, 4, 0x0060, 0x10, 21, 1), + PIN_FIELD_BASE(151, 151, 4, 0x0060, 0x10, 26, 1), + PIN_FIELD_BASE(152, 152, 4, 0x0060, 0x10, 25, 1), + PIN_FIELD_BASE(153, 153, 4, 0x0060, 0x10, 24, 1), + PIN_FIELD_BASE(154, 154, 4, 0x0060, 0x10, 22, 1), + PIN_FIELD_BASE(155, 155, 4, 0x0060, 0x10, 23, 1), + PIN_FIELD_BASE(156, 156, 4, 0x0060, 0x10, 0, 1), + PIN_FIELD_BASE(157, 157, 4, 0x0060, 0x10, 1, 1), + PIN_FIELD_BASE(158, 158, 4, 0x0060, 0x10, 2, 1), + PIN_FIELD_BASE(159, 159, 4, 0x0060, 0x10, 3, 1), + PIN_FIELD_BASE(160, 160, 4, 0x0060, 0x10, 4, 1), + PIN_FIELD_BASE(161, 161, 4, 0x0060, 0x10, 5, 1), + PIN_FIELD_BASE(162, 162, 4, 0x0060, 0x10, 6, 1), + PIN_FIELD_BASE(163, 163, 4, 0x0060, 0x10, 7, 1), + PIN_FIELD_BASE(164, 164, 5, 0x0060, 0x10, 12, 1), + PIN_FIELD_BASE(165, 165, 5, 0x0060, 0x10, 3, 1), + PIN_FIELD_BASE(166, 166, 5, 0x0060, 0x10, 4, 1), + PIN_FIELD_BASE(167, 167, 5, 0x0060, 0x10, 2, 1), + PIN_FIELD_BASE(168, 168, 5, 0x0060, 0x10, 13, 1), + PIN_FIELD_BASE(169, 169, 5, 0x0060, 0x10, 14, 1), + PIN_FIELD_BASE(170, 170, 5, 0x0060, 0x10, 5, 1), + PIN_FIELD_BASE(171, 171, 6, 0x0050, 0x10, 0, 1), + PIN_FIELD_BASE(172, 172, 6, 0x0050, 0x10, 1, 1), + PIN_FIELD_BASE(173, 173, 6, 0x0050, 0x10, 2, 1), + PIN_FIELD_BASE(174, 174, 6, 0x0050, 0x10, 3, 1), + PIN_FIELD_BASE(175, 175, 6, 0x0050, 0x10, 4, 1), + PIN_FIELD_BASE(176, 176, 6, 0x0050, 0x10, 5, 1), + PIN_FIELD_BASE(177, 177, 6, 0x0050, 0x10, 6, 1), + PIN_FIELD_BASE(178, 178, 6, 0x0050, 0x10, 7, 1), + PIN_FIELD_BASE(179, 179, 6, 0x0050, 0x10, 8, 1), + PIN_FIELD_BASE(180, 180, 6, 0x0050, 0x10, 9, 1), + PIN_FIELD_BASE(181, 181, 10, 0x0020, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_pupd_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x0080, 0x10, 0, 1), + PIN_FIELD_BASE(1, 1, 8, 0x0080, 0x10, 1, 1), + PIN_FIELD_BASE(2, 2, 8, 0x0080, 0x10, 2, 1), + PIN_FIELD_BASE(3, 3, 8, 0x0080, 0x10, 3, 1), + PIN_FIELD_BASE(4, 4, 8, 0x0080, 0x10, 4, 1), + PIN_FIELD_BASE(5, 5, 8, 0x0080, 0x10, 5, 1), + PIN_FIELD_BASE(6, 6, 8, 0x0080, 0x10, 6, 1), + PIN_FIELD_BASE(7, 7, 8, 0x0080, 0x10, 7, 1), + PIN_FIELD_BASE(14, 14, 1, 0x00c0, 0x10, 14, 1), + PIN_FIELD_BASE(15, 15, 1, 0x00c0, 0x10, 16, 1), + PIN_FIELD_BASE(16, 16, 1, 0x00c0, 0x10, 17, 1), + PIN_FIELD_BASE(19, 19, 1, 0x00c0, 0x10, 21, 1), + PIN_FIELD_BASE(20, 20, 1, 0x00c0, 0x10, 22, 1), + PIN_FIELD_BASE(21, 21, 1, 0x00c0, 0x10, 20, 1), + PIN_FIELD_BASE(22, 22, 1, 0x00c0, 0x10, 19, 1), + PIN_FIELD_BASE(23, 23, 1, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(24, 24, 1, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(25, 25, 1, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(26, 26, 1, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(27, 27, 1, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(28, 28, 1, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(29, 29, 1, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(30, 30, 1, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(31, 31, 1, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(32, 32, 9, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(33, 33, 1, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(34, 34, 1, 0x00c0, 0x10, 10, 1), + PIN_FIELD_BASE(35, 35, 1, 0x00c0, 0x10, 11, 1), + PIN_FIELD_BASE(36, 36, 9, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(37, 37, 2, 0x00b0, 0x10, 10, 1), + PIN_FIELD_BASE(38, 38, 2, 0x00b0, 0x10, 11, 1), + PIN_FIELD_BASE(39, 39, 1, 0x00c0, 0x10, 12, 1), + PIN_FIELD_BASE(40, 40, 2, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(41, 41, 2, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(42, 42, 2, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(43, 43, 2, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(44, 44, 2, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(45, 45, 2, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(46, 46, 1, 0x00c0, 0x10, 13, 1), + PIN_FIELD_BASE(47, 47, 1, 0x00c0, 0x10, 15, 1), + PIN_FIELD_BASE(48, 48, 2, 0x00b0, 0x10, 6, 1), + PIN_FIELD_BASE(49, 49, 2, 0x00b0, 0x10, 7, 1), + PIN_FIELD_BASE(50, 50, 2, 0x00b0, 0x10, 8, 1), + PIN_FIELD_BASE(51, 51, 2, 0x00b0, 0x10, 9, 1), + PIN_FIELD_BASE(58, 58, 1, 0x00c0, 0x10, 18, 1), + PIN_FIELD_BASE(62, 62, 9, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(63, 63, 9, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(64, 64, 9, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(65, 65, 9, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(68, 68, 9, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(74, 74, 7, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(75, 75, 7, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(76, 76, 7, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(77, 77, 7, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(78, 78, 7, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(79, 79, 7, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(80, 80, 7, 0x00b0, 0x10, 6, 1), + PIN_FIELD_BASE(81, 81, 7, 0x00b0, 0x10, 7, 1), + PIN_FIELD_BASE(82, 82, 7, 0x00b0, 0x10, 8, 1), + PIN_FIELD_BASE(83, 83, 7, 0x00b0, 0x10, 9, 1), + PIN_FIELD_BASE(84, 84, 7, 0x00b0, 0x10, 10, 1), + PIN_FIELD_BASE(85, 85, 7, 0x00b0, 0x10, 11, 1), + PIN_FIELD_BASE(86, 86, 7, 0x00b0, 0x10, 12, 1), + PIN_FIELD_BASE(87, 87, 7, 0x00b0, 0x10, 13, 1), + PIN_FIELD_BASE(90, 90, 7, 0x00b0, 0x10, 14, 1), + PIN_FIELD_BASE(91, 91, 7, 0x00b0, 0x10, 15, 1), + PIN_FIELD_BASE(94, 94, 3, 0x00c0, 0x10, 12, 1), + PIN_FIELD_BASE(95, 95, 3, 0x00c0, 0x10, 13, 1), + PIN_FIELD_BASE(96, 96, 3, 0x00c0, 0x10, 14, 1), + PIN_FIELD_BASE(97, 97, 3, 0x00c0, 0x10, 15, 1), + PIN_FIELD_BASE(98, 98, 3, 0x00c0, 0x10, 16, 1), + PIN_FIELD_BASE(99, 99, 3, 0x00c0, 0x10, 17, 1), + PIN_FIELD_BASE(100, 100, 3, 0x00c0, 0x10, 18, 1), + PIN_FIELD_BASE(101, 101, 3, 0x00c0, 0x10, 19, 1), + PIN_FIELD_BASE(102, 102, 3, 0x00c0, 0x10, 20, 1), + PIN_FIELD_BASE(103, 103, 3, 0x00c0, 0x10, 21, 1), + PIN_FIELD_BASE(104, 104, 3, 0x00c0, 0x10, 22, 1), + PIN_FIELD_BASE(105, 105, 3, 0x00c0, 0x10, 23, 1), + PIN_FIELD_BASE(106, 106, 8, 0x0080, 0x10, 8, 1), + PIN_FIELD_BASE(107, 107, 8, 0x0080, 0x10, 9, 1), + PIN_FIELD_BASE(108, 108, 8, 0x0080, 0x10, 10, 1), + PIN_FIELD_BASE(109, 109, 8, 0x0080, 0x10, 11, 1), + PIN_FIELD_BASE(110, 110, 8, 0x0080, 0x10, 12, 1), + PIN_FIELD_BASE(111, 111, 8, 0x0080, 0x10, 13, 1), + PIN_FIELD_BASE(112, 112, 5, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(113, 113, 5, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(114, 114, 5, 0x00a0, 0x10, 7, 1), + PIN_FIELD_BASE(115, 115, 5, 0x00a0, 0x10, 8, 1), + PIN_FIELD_BASE(116, 116, 5, 0x00a0, 0x10, 9, 1), + PIN_FIELD_BASE(125, 125, 4, 0x00a0, 0x10, 8, 1), + PIN_FIELD_BASE(130, 130, 4, 0x00a0, 0x10, 9, 1), + PIN_FIELD_BASE(131, 131, 4, 0x00a0, 0x10, 10, 1), + PIN_FIELD_BASE(132, 132, 4, 0x00a0, 0x10, 11, 1), + PIN_FIELD_BASE(133, 133, 4, 0x00a0, 0x10, 12, 1), + PIN_FIELD_BASE(138, 138, 3, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(139, 139, 3, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(140, 140, 3, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(141, 141, 3, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(142, 142, 3, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(143, 143, 3, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(144, 144, 3, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(145, 145, 3, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(146, 146, 3, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(147, 147, 3, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(148, 148, 3, 0x00c0, 0x10, 10, 1), + PIN_FIELD_BASE(149, 149, 3, 0x00c0, 0x10, 11, 1), + PIN_FIELD_BASE(150, 150, 4, 0x00a0, 0x10, 13, 1), + PIN_FIELD_BASE(151, 151, 4, 0x00a0, 0x10, 18, 1), + PIN_FIELD_BASE(152, 152, 4, 0x00a0, 0x10, 17, 1), + PIN_FIELD_BASE(153, 153, 4, 0x00a0, 0x10, 16, 1), + PIN_FIELD_BASE(154, 154, 4, 0x00a0, 0x10, 14, 1), + PIN_FIELD_BASE(155, 155, 4, 0x00a0, 0x10, 15, 1), + PIN_FIELD_BASE(156, 156, 4, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(157, 157, 4, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(158, 158, 4, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(159, 159, 4, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(160, 160, 4, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(161, 161, 4, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(162, 162, 4, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(163, 163, 4, 0x00a0, 0x10, 7, 1), + PIN_FIELD_BASE(164, 164, 5, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(167, 167, 5, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(168, 168, 5, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(169, 169, 5, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(170, 170, 5, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(171, 171, 6, 0x0080, 0x10, 0, 1), + PIN_FIELD_BASE(172, 172, 6, 0x0080, 0x10, 1, 1), + PIN_FIELD_BASE(173, 173, 6, 0x0080, 0x10, 2, 1), + PIN_FIELD_BASE(174, 174, 6, 0x0080, 0x10, 3, 1), + PIN_FIELD_BASE(175, 175, 6, 0x0080, 0x10, 4, 1), + PIN_FIELD_BASE(176, 176, 6, 0x0080, 0x10, 5, 1), + PIN_FIELD_BASE(177, 177, 6, 0x0080, 0x10, 6, 1), + PIN_FIELD_BASE(178, 178, 6, 0x0080, 0x10, 7, 1), + PIN_FIELD_BASE(179, 179, 6, 0x0080, 0x10, 8, 1), + PIN_FIELD_BASE(180, 180, 6, 0x0080, 0x10, 9, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_r0_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x0090, 0x10, 0, 1), + PIN_FIELD_BASE(1, 1, 8, 0x0090, 0x10, 1, 1), + PIN_FIELD_BASE(2, 2, 8, 0x0090, 0x10, 2, 1), + PIN_FIELD_BASE(3, 3, 8, 0x0090, 0x10, 3, 1), + PIN_FIELD_BASE(4, 4, 8, 0x0090, 0x10, 4, 1), + PIN_FIELD_BASE(5, 5, 8, 0x0090, 0x10, 5, 1), + PIN_FIELD_BASE(6, 6, 8, 0x0090, 0x10, 6, 1), + PIN_FIELD_BASE(7, 7, 8, 0x0090, 0x10, 7, 1), + PIN_FIELD_BASE(14, 14, 1, 0x00e0, 0x10, 14, 1), + PIN_FIELD_BASE(15, 15, 1, 0x00e0, 0x10, 16, 1), + PIN_FIELD_BASE(16, 16, 1, 0x00e0, 0x10, 17, 1), + PIN_FIELD_BASE(19, 19, 1, 0x00e0, 0x10, 21, 1), + PIN_FIELD_BASE(20, 20, 1, 0x00e0, 0x10, 22, 1), + PIN_FIELD_BASE(21, 21, 1, 0x00e0, 0x10, 20, 1), + PIN_FIELD_BASE(22, 22, 1, 0x00e0, 0x10, 19, 1), + PIN_FIELD_BASE(23, 23, 1, 0x00e0, 0x10, 0, 1), + PIN_FIELD_BASE(24, 24, 1, 0x00e0, 0x10, 1, 1), + PIN_FIELD_BASE(25, 25, 1, 0x00e0, 0x10, 2, 1), + PIN_FIELD_BASE(26, 26, 1, 0x00e0, 0x10, 3, 1), + PIN_FIELD_BASE(27, 27, 1, 0x00e0, 0x10, 4, 1), + PIN_FIELD_BASE(28, 28, 1, 0x00e0, 0x10, 5, 1), + PIN_FIELD_BASE(29, 29, 1, 0x00e0, 0x10, 6, 1), + PIN_FIELD_BASE(30, 30, 1, 0x00e0, 0x10, 7, 1), + PIN_FIELD_BASE(31, 31, 1, 0x00e0, 0x10, 8, 1), + PIN_FIELD_BASE(32, 32, 9, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(33, 33, 1, 0x00e0, 0x10, 9, 1), + PIN_FIELD_BASE(34, 34, 1, 0x00e0, 0x10, 10, 1), + PIN_FIELD_BASE(35, 35, 1, 0x00e0, 0x10, 11, 1), + PIN_FIELD_BASE(36, 36, 9, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(37, 37, 2, 0x00d0, 0x10, 10, 1), + PIN_FIELD_BASE(38, 38, 2, 0x00d0, 0x10, 11, 1), + PIN_FIELD_BASE(39, 39, 1, 0x00e0, 0x10, 12, 1), + PIN_FIELD_BASE(40, 40, 2, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(41, 41, 2, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(42, 42, 2, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(43, 43, 2, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(44, 44, 2, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(45, 45, 2, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(46, 46, 1, 0x00e0, 0x10, 13, 1), + PIN_FIELD_BASE(47, 47, 1, 0x00e0, 0x10, 15, 1), + PIN_FIELD_BASE(48, 48, 2, 0x00d0, 0x10, 6, 1), + PIN_FIELD_BASE(49, 49, 2, 0x00d0, 0x10, 7, 1), + PIN_FIELD_BASE(50, 50, 2, 0x00d0, 0x10, 8, 1), + PIN_FIELD_BASE(51, 51, 2, 0x00d0, 0x10, 9, 1), + PIN_FIELD_BASE(58, 58, 1, 0x00e0, 0x10, 18, 1), + PIN_FIELD_BASE(62, 62, 9, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(63, 63, 9, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(64, 64, 9, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(65, 65, 9, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(68, 68, 9, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(74, 74, 7, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(75, 75, 7, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(76, 76, 7, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(77, 77, 7, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(78, 78, 7, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(79, 79, 7, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(80, 80, 7, 0x00d0, 0x10, 6, 1), + PIN_FIELD_BASE(81, 81, 7, 0x00d0, 0x10, 7, 1), + PIN_FIELD_BASE(82, 82, 7, 0x00d0, 0x10, 8, 1), + PIN_FIELD_BASE(83, 83, 7, 0x00d0, 0x10, 9, 1), + PIN_FIELD_BASE(84, 84, 7, 0x00d0, 0x10, 10, 1), + PIN_FIELD_BASE(85, 85, 7, 0x00d0, 0x10, 11, 1), + PIN_FIELD_BASE(86, 86, 7, 0x00d0, 0x10, 12, 1), + PIN_FIELD_BASE(87, 87, 7, 0x00d0, 0x10, 13, 1), + PIN_FIELD_BASE(90, 90, 7, 0x00d0, 0x10, 14, 1), + PIN_FIELD_BASE(91, 91, 7, 0x00d0, 0x10, 15, 1), + PIN_FIELD_BASE(94, 94, 3, 0x00e0, 0x10, 12, 1), + PIN_FIELD_BASE(95, 95, 3, 0x00e0, 0x10, 13, 1), + PIN_FIELD_BASE(96, 96, 3, 0x00e0, 0x10, 14, 1), + PIN_FIELD_BASE(97, 97, 3, 0x00e0, 0x10, 15, 1), + PIN_FIELD_BASE(98, 98, 3, 0x00e0, 0x10, 16, 1), + PIN_FIELD_BASE(99, 99, 3, 0x00e0, 0x10, 17, 1), + PIN_FIELD_BASE(100, 100, 3, 0x00e0, 0x10, 18, 1), + PIN_FIELD_BASE(101, 101, 3, 0x00e0, 0x10, 19, 1), + PIN_FIELD_BASE(102, 102, 3, 0x00e0, 0x10, 20, 1), + PIN_FIELD_BASE(103, 103, 3, 0x00e0, 0x10, 21, 1), + PIN_FIELD_BASE(104, 104, 3, 0x00e0, 0x10, 22, 1), + PIN_FIELD_BASE(105, 105, 3, 0x00e0, 0x10, 23, 1), + PIN_FIELD_BASE(106, 106, 8, 0x0090, 0x10, 8, 1), + PIN_FIELD_BASE(107, 107, 8, 0x0090, 0x10, 9, 1), + PIN_FIELD_BASE(108, 108, 8, 0x0090, 0x10, 10, 1), + PIN_FIELD_BASE(109, 109, 8, 0x0090, 0x10, 11, 1), + PIN_FIELD_BASE(110, 110, 8, 0x0090, 0x10, 12, 1), + PIN_FIELD_BASE(111, 111, 8, 0x0090, 0x10, 13, 1), + PIN_FIELD_BASE(112, 112, 5, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(113, 113, 5, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(114, 114, 5, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(115, 115, 5, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(116, 116, 5, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(125, 125, 4, 0x00c0, 0x10, 8, 1), + PIN_FIELD_BASE(130, 130, 4, 0x00c0, 0x10, 9, 1), + PIN_FIELD_BASE(131, 131, 4, 0x00c0, 0x10, 10, 1), + PIN_FIELD_BASE(132, 132, 4, 0x00c0, 0x10, 11, 1), + PIN_FIELD_BASE(133, 133, 4, 0x00c0, 0x10, 12, 1), + PIN_FIELD_BASE(138, 138, 3, 0x00e0, 0x10, 0, 1), + PIN_FIELD_BASE(139, 139, 3, 0x00e0, 0x10, 1, 1), + PIN_FIELD_BASE(140, 140, 3, 0x00e0, 0x10, 2, 1), + PIN_FIELD_BASE(141, 141, 3, 0x00e0, 0x10, 3, 1), + PIN_FIELD_BASE(142, 142, 3, 0x00e0, 0x10, 4, 1), + PIN_FIELD_BASE(143, 143, 3, 0x00e0, 0x10, 5, 1), + PIN_FIELD_BASE(144, 144, 3, 0x00e0, 0x10, 6, 1), + PIN_FIELD_BASE(145, 145, 3, 0x00e0, 0x10, 7, 1), + PIN_FIELD_BASE(146, 146, 3, 0x00e0, 0x10, 8, 1), + PIN_FIELD_BASE(147, 147, 3, 0x00e0, 0x10, 9, 1), + PIN_FIELD_BASE(148, 148, 3, 0x00e0, 0x10, 10, 1), + PIN_FIELD_BASE(149, 149, 3, 0x00e0, 0x10, 11, 1), + PIN_FIELD_BASE(150, 150, 4, 0x00c0, 0x10, 13, 1), + PIN_FIELD_BASE(151, 151, 4, 0x00c0, 0x10, 18, 1), + PIN_FIELD_BASE(152, 152, 4, 0x00c0, 0x10, 17, 1), + PIN_FIELD_BASE(153, 153, 4, 0x00c0, 0x10, 16, 1), + PIN_FIELD_BASE(154, 154, 4, 0x00c0, 0x10, 14, 1), + PIN_FIELD_BASE(155, 155, 4, 0x00c0, 0x10, 15, 1), + PIN_FIELD_BASE(156, 156, 4, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(157, 157, 4, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(158, 158, 4, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(159, 159, 4, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(160, 160, 4, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(161, 161, 4, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(162, 162, 4, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(163, 163, 4, 0x00c0, 0x10, 7, 1), + PIN_FIELD_BASE(164, 164, 5, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(167, 167, 5, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(168, 168, 5, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(169, 169, 5, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(170, 170, 5, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(171, 171, 6, 0x0090, 0x10, 0, 1), + PIN_FIELD_BASE(172, 172, 6, 0x0090, 0x10, 1, 1), + PIN_FIELD_BASE(173, 173, 6, 0x0090, 0x10, 2, 1), + PIN_FIELD_BASE(174, 174, 6, 0x0090, 0x10, 3, 1), + PIN_FIELD_BASE(175, 175, 6, 0x0090, 0x10, 4, 1), + PIN_FIELD_BASE(176, 176, 6, 0x0090, 0x10, 5, 1), + PIN_FIELD_BASE(177, 177, 6, 0x0090, 0x10, 6, 1), + PIN_FIELD_BASE(178, 178, 6, 0x0090, 0x10, 7, 1), + PIN_FIELD_BASE(179, 179, 6, 0x0090, 0x10, 8, 1), + PIN_FIELD_BASE(180, 180, 6, 0x0090, 0x10, 9, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_r1_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(1, 1, 8, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(2, 2, 8, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(3, 3, 8, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(4, 4, 8, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(5, 5, 8, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(6, 6, 8, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(7, 7, 8, 0x00a0, 0x10, 7, 1), + PIN_FIELD_BASE(14, 14, 1, 0x00f0, 0x10, 14, 1), + PIN_FIELD_BASE(15, 15, 1, 0x00f0, 0x10, 16, 1), + PIN_FIELD_BASE(16, 16, 1, 0x00f0, 0x10, 17, 1), + PIN_FIELD_BASE(19, 19, 1, 0x00f0, 0x10, 21, 1), + PIN_FIELD_BASE(20, 20, 1, 0x00f0, 0x10, 22, 1), + PIN_FIELD_BASE(21, 21, 1, 0x00f0, 0x10, 20, 1), + PIN_FIELD_BASE(22, 22, 1, 0x00f0, 0x10, 19, 1), + PIN_FIELD_BASE(23, 23, 1, 0x00f0, 0x10, 0, 1), + PIN_FIELD_BASE(24, 24, 1, 0x00f0, 0x10, 1, 1), + PIN_FIELD_BASE(25, 25, 1, 0x00f0, 0x10, 2, 1), + PIN_FIELD_BASE(26, 26, 1, 0x00f0, 0x10, 3, 1), + PIN_FIELD_BASE(27, 27, 1, 0x00f0, 0x10, 4, 1), + PIN_FIELD_BASE(28, 28, 1, 0x00f0, 0x10, 5, 1), + PIN_FIELD_BASE(29, 29, 1, 0x00f0, 0x10, 6, 1), + PIN_FIELD_BASE(30, 30, 1, 0x00f0, 0x10, 7, 1), + PIN_FIELD_BASE(31, 31, 1, 0x00f0, 0x10, 8, 1), + PIN_FIELD_BASE(32, 32, 9, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(33, 33, 1, 0x00f0, 0x10, 9, 1), + PIN_FIELD_BASE(34, 34, 1, 0x00f0, 0x10, 10, 1), + PIN_FIELD_BASE(35, 35, 1, 0x00f0, 0x10, 11, 1), + PIN_FIELD_BASE(36, 36, 9, 0x00d0, 0x10, 6, 1), + PIN_FIELD_BASE(37, 37, 2, 0x00e0, 0x10, 10, 1), + PIN_FIELD_BASE(38, 38, 2, 0x00e0, 0x10, 11, 1), + PIN_FIELD_BASE(39, 39, 1, 0x00f0, 0x10, 12, 1), + PIN_FIELD_BASE(40, 40, 2, 0x00e0, 0x10, 0, 1), + PIN_FIELD_BASE(41, 41, 2, 0x00e0, 0x10, 1, 1), + PIN_FIELD_BASE(42, 42, 2, 0x00e0, 0x10, 2, 1), + PIN_FIELD_BASE(43, 43, 2, 0x00e0, 0x10, 3, 1), + PIN_FIELD_BASE(44, 44, 2, 0x00e0, 0x10, 4, 1), + PIN_FIELD_BASE(45, 45, 2, 0x00e0, 0x10, 5, 1), + PIN_FIELD_BASE(46, 46, 1, 0x00f0, 0x10, 13, 1), + PIN_FIELD_BASE(47, 47, 1, 0x00f0, 0x10, 15, 1), + PIN_FIELD_BASE(48, 48, 2, 0x00e0, 0x10, 6, 1), + PIN_FIELD_BASE(49, 49, 2, 0x00e0, 0x10, 7, 1), + PIN_FIELD_BASE(50, 50, 2, 0x00e0, 0x10, 8, 1), + PIN_FIELD_BASE(51, 51, 2, 0x00e0, 0x10, 9, 1), + PIN_FIELD_BASE(58, 58, 1, 0x00f0, 0x10, 18, 1), + PIN_FIELD_BASE(62, 62, 9, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(63, 63, 9, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(64, 64, 9, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(65, 65, 9, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(68, 68, 9, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(74, 74, 7, 0x00e0, 0x10, 0, 1), + PIN_FIELD_BASE(75, 75, 7, 0x00e0, 0x10, 1, 1), + PIN_FIELD_BASE(76, 76, 7, 0x00e0, 0x10, 2, 1), + PIN_FIELD_BASE(77, 77, 7, 0x00e0, 0x10, 3, 1), + PIN_FIELD_BASE(78, 78, 7, 0x00e0, 0x10, 4, 1), + PIN_FIELD_BASE(79, 79, 7, 0x00e0, 0x10, 5, 1), + PIN_FIELD_BASE(80, 80, 7, 0x00e0, 0x10, 6, 1), + PIN_FIELD_BASE(81, 81, 7, 0x00e0, 0x10, 7, 1), + PIN_FIELD_BASE(82, 82, 7, 0x00e0, 0x10, 8, 1), + PIN_FIELD_BASE(83, 83, 7, 0x00e0, 0x10, 9, 1), + PIN_FIELD_BASE(84, 84, 7, 0x00e0, 0x10, 10, 1), + PIN_FIELD_BASE(85, 85, 7, 0x00e0, 0x10, 11, 1), + PIN_FIELD_BASE(86, 86, 7, 0x00e0, 0x10, 12, 1), + PIN_FIELD_BASE(87, 87, 7, 0x00e0, 0x10, 13, 1), + PIN_FIELD_BASE(90, 90, 7, 0x00e0, 0x10, 14, 1), + PIN_FIELD_BASE(91, 91, 7, 0x00e0, 0x10, 15, 1), + PIN_FIELD_BASE(94, 94, 3, 0x00f0, 0x10, 12, 1), + PIN_FIELD_BASE(95, 95, 3, 0x00f0, 0x10, 13, 1), + PIN_FIELD_BASE(96, 96, 3, 0x00f0, 0x10, 14, 1), + PIN_FIELD_BASE(97, 97, 3, 0x00f0, 0x10, 15, 1), + PIN_FIELD_BASE(98, 98, 3, 0x00f0, 0x10, 16, 1), + PIN_FIELD_BASE(99, 99, 3, 0x00f0, 0x10, 17, 1), + PIN_FIELD_BASE(100, 100, 3, 0x00f0, 0x10, 18, 1), + PIN_FIELD_BASE(101, 101, 3, 0x00f0, 0x10, 19, 1), + PIN_FIELD_BASE(102, 102, 3, 0x00f0, 0x10, 20, 1), + PIN_FIELD_BASE(103, 103, 3, 0x00f0, 0x10, 21, 1), + PIN_FIELD_BASE(104, 104, 3, 0x00f0, 0x10, 22, 1), + PIN_FIELD_BASE(105, 105, 3, 0x00f0, 0x10, 23, 1), + PIN_FIELD_BASE(106, 106, 8, 0x00a0, 0x10, 8, 1), + PIN_FIELD_BASE(107, 107, 8, 0x00a0, 0x10, 9, 1), + PIN_FIELD_BASE(108, 108, 8, 0x00a0, 0x10, 10, 1), + PIN_FIELD_BASE(109, 109, 8, 0x00a0, 0x10, 11, 1), + PIN_FIELD_BASE(110, 110, 8, 0x00a0, 0x10, 12, 1), + PIN_FIELD_BASE(111, 111, 8, 0x00a0, 0x10, 13, 1), + PIN_FIELD_BASE(112, 112, 5, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(113, 113, 5, 0x00d0, 0x10, 6, 1), + PIN_FIELD_BASE(114, 114, 5, 0x00d0, 0x10, 7, 1), + PIN_FIELD_BASE(115, 115, 5, 0x00d0, 0x10, 8, 1), + PIN_FIELD_BASE(116, 116, 5, 0x00d0, 0x10, 9, 1), + PIN_FIELD_BASE(125, 125, 4, 0x00d0, 0x10, 8, 1), + PIN_FIELD_BASE(130, 130, 4, 0x00d0, 0x10, 9, 1), + PIN_FIELD_BASE(131, 131, 4, 0x00d0, 0x10, 10, 1), + PIN_FIELD_BASE(132, 132, 4, 0x00d0, 0x10, 11, 1), + PIN_FIELD_BASE(133, 133, 4, 0x00d0, 0x10, 12, 1), + PIN_FIELD_BASE(138, 138, 3, 0x00f0, 0x10, 0, 1), + PIN_FIELD_BASE(139, 139, 3, 0x00f0, 0x10, 1, 1), + PIN_FIELD_BASE(140, 140, 3, 0x00f0, 0x10, 2, 1), + PIN_FIELD_BASE(141, 141, 3, 0x00f0, 0x10, 3, 1), + PIN_FIELD_BASE(142, 142, 3, 0x00f0, 0x10, 4, 1), + PIN_FIELD_BASE(143, 143, 3, 0x00f0, 0x10, 5, 1), + PIN_FIELD_BASE(144, 144, 3, 0x00f0, 0x10, 6, 1), + PIN_FIELD_BASE(145, 145, 3, 0x00f0, 0x10, 7, 1), + PIN_FIELD_BASE(146, 146, 3, 0x00f0, 0x10, 8, 1), + PIN_FIELD_BASE(147, 147, 3, 0x00f0, 0x10, 9, 1), + PIN_FIELD_BASE(148, 148, 3, 0x00f0, 0x10, 10, 1), + PIN_FIELD_BASE(149, 149, 3, 0x00f0, 0x10, 11, 1), + PIN_FIELD_BASE(150, 150, 4, 0x00d0, 0x10, 13, 1), + PIN_FIELD_BASE(151, 151, 4, 0x00d0, 0x10, 18, 1), + PIN_FIELD_BASE(152, 152, 4, 0x00d0, 0x10, 17, 1), + PIN_FIELD_BASE(153, 153, 4, 0x00d0, 0x10, 16, 1), + PIN_FIELD_BASE(154, 154, 4, 0x00d0, 0x10, 14, 1), + PIN_FIELD_BASE(155, 155, 4, 0x00d0, 0x10, 15, 1), + PIN_FIELD_BASE(156, 156, 4, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(157, 157, 4, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(158, 158, 4, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(159, 159, 4, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(160, 160, 4, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(161, 161, 4, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(162, 162, 4, 0x00d0, 0x10, 6, 1), + PIN_FIELD_BASE(163, 163, 4, 0x00d0, 0x10, 7, 1), + PIN_FIELD_BASE(164, 164, 5, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(167, 167, 5, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(168, 168, 5, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(169, 169, 5, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(170, 170, 5, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(171, 171, 6, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(172, 172, 6, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(173, 173, 6, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(174, 174, 6, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(175, 175, 6, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(176, 176, 6, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(177, 177, 6, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(178, 178, 6, 0x00a0, 0x10, 7, 1), + PIN_FIELD_BASE(179, 179, 6, 0x00a0, 0x10, 8, 1), + PIN_FIELD_BASE(180, 180, 6, 0x00a0, 0x10, 9, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_pu_range[] = { + PIN_FIELD_BASE(8, 8, 2, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(9, 9, 2, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(10, 10, 2, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(11, 11, 2, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(12, 12, 1, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(13, 13, 1, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(17, 17, 1, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(18, 18, 1, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(52, 52, 2, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(53, 53, 2, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(54, 54, 5, 0x00b0, 0x10, 8, 1), + PIN_FIELD_BASE(55, 55, 5, 0x00b0, 0x10, 9, 1), + PIN_FIELD_BASE(56, 56, 1, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(57, 57, 1, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(59, 59, 2, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(60, 60, 9, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(61, 61, 9, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(66, 66, 5, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(67, 67, 5, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(69, 69, 7, 0x00c0, 0x10, 0, 1), + PIN_FIELD_BASE(70, 70, 7, 0x00c0, 0x10, 1, 1), + PIN_FIELD_BASE(71, 71, 7, 0x00c0, 0x10, 2, 1), + PIN_FIELD_BASE(72, 72, 7, 0x00c0, 0x10, 3, 1), + PIN_FIELD_BASE(73, 73, 7, 0x00c0, 0x10, 4, 1), + PIN_FIELD_BASE(88, 88, 7, 0x00c0, 0x10, 5, 1), + PIN_FIELD_BASE(89, 89, 7, 0x00c0, 0x10, 6, 1), + PIN_FIELD_BASE(92, 92, 3, 0x00d0, 0x10, 4, 1), + PIN_FIELD_BASE(93, 93, 3, 0x00d0, 0x10, 5, 1), + PIN_FIELD_BASE(117, 117, 4, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(118, 118, 4, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(119, 119, 4, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(120, 120, 4, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(121, 121, 4, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(122, 122, 4, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(123, 123, 4, 0x00b0, 0x10, 6, 1), + PIN_FIELD_BASE(124, 124, 4, 0x00b0, 0x10, 7, 1), + PIN_FIELD_BASE(126, 126, 5, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(127, 127, 5, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(128, 128, 5, 0x00b0, 0x10, 6, 1), + PIN_FIELD_BASE(129, 129, 5, 0x00b0, 0x10, 7, 1), + PIN_FIELD_BASE(134, 134, 3, 0x00d0, 0x10, 0, 1), + PIN_FIELD_BASE(135, 135, 3, 0x00d0, 0x10, 1, 1), + PIN_FIELD_BASE(136, 136, 3, 0x00d0, 0x10, 2, 1), + PIN_FIELD_BASE(137, 137, 3, 0x00d0, 0x10, 3, 1), + PIN_FIELD_BASE(165, 165, 5, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(166, 166, 5, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(181, 181, 10, 0x0060, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_pd_range[] = { + PIN_FIELD_BASE(8, 8, 2, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(9, 9, 2, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(10, 10, 2, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(11, 11, 2, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(12, 12, 1, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(13, 13, 1, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(17, 17, 1, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(18, 18, 1, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(52, 52, 2, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(53, 53, 2, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(54, 54, 5, 0x0090, 0x10, 8, 1), + PIN_FIELD_BASE(55, 55, 5, 0x0090, 0x10, 9, 1), + PIN_FIELD_BASE(56, 56, 1, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(57, 57, 1, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(59, 59, 2, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(60, 60, 9, 0x0090, 0x10, 0, 1), + PIN_FIELD_BASE(61, 61, 9, 0x0090, 0x10, 1, 1), + PIN_FIELD_BASE(66, 66, 5, 0x0090, 0x10, 0, 1), + PIN_FIELD_BASE(67, 67, 5, 0x0090, 0x10, 1, 1), + PIN_FIELD_BASE(69, 69, 7, 0x00a0, 0x10, 0, 1), + PIN_FIELD_BASE(70, 70, 7, 0x00a0, 0x10, 1, 1), + PIN_FIELD_BASE(71, 71, 7, 0x00a0, 0x10, 2, 1), + PIN_FIELD_BASE(72, 72, 7, 0x00a0, 0x10, 3, 1), + PIN_FIELD_BASE(73, 73, 7, 0x00a0, 0x10, 4, 1), + PIN_FIELD_BASE(88, 88, 7, 0x00a0, 0x10, 5, 1), + PIN_FIELD_BASE(89, 89, 7, 0x00a0, 0x10, 6, 1), + PIN_FIELD_BASE(92, 92, 3, 0x00b0, 0x10, 4, 1), + PIN_FIELD_BASE(93, 93, 3, 0x00b0, 0x10, 5, 1), + PIN_FIELD_BASE(117, 117, 4, 0x0090, 0x10, 0, 1), + PIN_FIELD_BASE(118, 118, 4, 0x0090, 0x10, 1, 1), + PIN_FIELD_BASE(119, 119, 4, 0x0090, 0x10, 2, 1), + PIN_FIELD_BASE(120, 120, 4, 0x0090, 0x10, 3, 1), + PIN_FIELD_BASE(121, 121, 4, 0x0090, 0x10, 4, 1), + PIN_FIELD_BASE(122, 122, 4, 0x0090, 0x10, 5, 1), + PIN_FIELD_BASE(123, 123, 4, 0x0090, 0x10, 6, 1), + PIN_FIELD_BASE(124, 124, 4, 0x0090, 0x10, 7, 1), + PIN_FIELD_BASE(126, 126, 5, 0x0090, 0x10, 4, 1), + PIN_FIELD_BASE(127, 127, 5, 0x0090, 0x10, 5, 1), + PIN_FIELD_BASE(128, 128, 5, 0x0090, 0x10, 6, 1), + PIN_FIELD_BASE(129, 129, 5, 0x0090, 0x10, 7, 1), + PIN_FIELD_BASE(134, 134, 3, 0x00b0, 0x10, 0, 1), + PIN_FIELD_BASE(135, 135, 3, 0x00b0, 0x10, 1, 1), + PIN_FIELD_BASE(136, 136, 3, 0x00b0, 0x10, 2, 1), + PIN_FIELD_BASE(137, 137, 3, 0x00b0, 0x10, 3, 1), + PIN_FIELD_BASE(165, 165, 5, 0x0090, 0x10, 2, 1), + PIN_FIELD_BASE(166, 166, 5, 0x0090, 0x10, 3, 1), + PIN_FIELD_BASE(181, 181, 10, 0x0050, 0x10, 0, 1), +}; + +static const struct mtk_pin_field_calc mt8901_pin_drv_range[] = { + PIN_FIELD_BASE(0, 0, 8, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(1, 1, 8, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(2, 2, 8, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(3, 3, 8, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(4, 4, 8, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(5, 5, 8, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(6, 6, 8, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(7, 7, 8, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(8, 8, 2, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(9, 9, 2, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(10, 10, 2, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(11, 11, 2, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(12, 12, 1, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(13, 13, 1, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(14, 14, 1, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(15, 15, 1, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(16, 16, 1, 0x0010, 0x10, 27, 3), + PIN_FIELD_BASE(17, 17, 1, 0x0020, 0x10, 0, 3), + PIN_FIELD_BASE(18, 18, 1, 0x0020, 0x10, 3, 3), + PIN_FIELD_BASE(19, 19, 1, 0x0020, 0x10, 21, 3), + PIN_FIELD_BASE(20, 20, 1, 0x0020, 0x10, 24, 3), + PIN_FIELD_BASE(21, 21, 1, 0x0020, 0x10, 18, 3), + PIN_FIELD_BASE(22, 22, 1, 0x0020, 0x10, 15, 3), + PIN_FIELD_BASE(23, 23, 1, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(24, 24, 1, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(25, 25, 1, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(26, 26, 1, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(27, 27, 1, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(28, 28, 1, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(29, 29, 1, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(30, 30, 1, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(31, 31, 1, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(32, 32, 9, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(33, 33, 1, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(34, 34, 1, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(35, 35, 1, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(36, 36, 9, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(37, 37, 2, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(38, 38, 2, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(39, 39, 1, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(40, 40, 2, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(41, 41, 2, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(42, 42, 2, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(43, 43, 2, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(44, 44, 2, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(45, 45, 2, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(46, 46, 1, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(47, 47, 1, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(48, 48, 2, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(49, 49, 2, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(50, 50, 2, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(51, 51, 2, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(52, 52, 2, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(53, 53, 2, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(54, 54, 5, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(55, 55, 5, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(56, 56, 1, 0x0020, 0x10, 6, 3), + PIN_FIELD_BASE(57, 57, 1, 0x0020, 0x10, 9, 3), + PIN_FIELD_BASE(58, 58, 1, 0x0020, 0x10, 12, 3), + PIN_FIELD_BASE(59, 59, 2, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(60, 60, 9, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(61, 61, 9, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(62, 62, 9, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(63, 63, 9, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(64, 64, 9, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(65, 65, 9, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(66, 66, 5, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(67, 67, 5, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(68, 68, 9, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(69, 69, 7, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(70, 70, 7, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(71, 71, 7, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(72, 72, 7, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(73, 73, 7, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(74, 74, 7, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(75, 75, 7, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(76, 76, 7, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(77, 77, 7, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(78, 78, 7, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(79, 79, 7, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(80, 80, 7, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(81, 81, 7, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(82, 82, 7, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(83, 83, 7, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(84, 84, 7, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(85, 85, 7, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(86, 86, 7, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(87, 87, 7, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(88, 88, 7, 0x0010, 0x10, 27, 3), + PIN_FIELD_BASE(89, 89, 7, 0x0020, 0x10, 0, 3), + PIN_FIELD_BASE(90, 90, 7, 0x0020, 0x10, 3, 3), + PIN_FIELD_BASE(91, 91, 7, 0x0020, 0x10, 6, 3), + PIN_FIELD_BASE(92, 92, 3, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(93, 93, 3, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(94, 94, 3, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(95, 95, 3, 0x0010, 0x10, 27, 3), + PIN_FIELD_BASE(96, 96, 3, 0x0020, 0x10, 0, 3), + PIN_FIELD_BASE(97, 97, 3, 0x0020, 0x10, 3, 3), + PIN_FIELD_BASE(98, 98, 3, 0x0020, 0x10, 6, 3), + PIN_FIELD_BASE(99, 99, 3, 0x0020, 0x10, 9, 3), + PIN_FIELD_BASE(100, 100, 3, 0x0020, 0x10, 12, 3), + PIN_FIELD_BASE(101, 101, 3, 0x0020, 0x10, 15, 3), + PIN_FIELD_BASE(102, 102, 3, 0x0020, 0x10, 18, 3), + PIN_FIELD_BASE(103, 103, 3, 0x0020, 0x10, 21, 3), + PIN_FIELD_BASE(104, 104, 3, 0x0020, 0x10, 24, 3), + PIN_FIELD_BASE(105, 105, 3, 0x0020, 0x10, 27, 3), + PIN_FIELD_BASE(106, 106, 8, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(107, 107, 8, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(108, 108, 8, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(109, 109, 8, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(110, 110, 8, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(111, 111, 8, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(112, 112, 5, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(113, 113, 5, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(114, 114, 5, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(115, 115, 5, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(116, 116, 5, 0x0010, 0x10, 27, 3), + PIN_FIELD_BASE(117, 117, 4, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(118, 118, 4, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(119, 119, 4, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(120, 120, 4, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(121, 121, 4, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(122, 122, 4, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(123, 123, 4, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(124, 124, 4, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(125, 125, 4, 0x0010, 0x10, 18, 3), + PIN_FIELD_BASE(126, 126, 5, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(127, 127, 5, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(128, 128, 5, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(129, 129, 5, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(130, 130, 4, 0x0010, 0x10, 21, 3), + PIN_FIELD_BASE(131, 131, 4, 0x0010, 0x10, 24, 3), + PIN_FIELD_BASE(132, 132, 4, 0x0010, 0x10, 27, 3), + PIN_FIELD_BASE(133, 133, 4, 0x0020, 0x10, 0, 3), + PIN_FIELD_BASE(134, 134, 3, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(135, 135, 3, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(136, 136, 3, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(137, 137, 3, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(138, 138, 3, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(139, 139, 3, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(140, 140, 3, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(141, 141, 3, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(142, 142, 3, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(143, 143, 3, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(144, 144, 3, 0x0010, 0x10, 0, 3), + PIN_FIELD_BASE(145, 145, 3, 0x0010, 0x10, 3, 3), + PIN_FIELD_BASE(146, 146, 3, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(147, 147, 3, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(148, 148, 3, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(149, 149, 3, 0x0010, 0x10, 15, 3), + PIN_FIELD_BASE(150, 150, 4, 0x0020, 0x10, 3, 3), + PIN_FIELD_BASE(151, 151, 4, 0x0020, 0x10, 18, 3), + PIN_FIELD_BASE(152, 152, 4, 0x0020, 0x10, 15, 3), + PIN_FIELD_BASE(153, 153, 4, 0x0020, 0x10, 12, 3), + PIN_FIELD_BASE(154, 154, 4, 0x0020, 0x10, 6, 3), + PIN_FIELD_BASE(155, 155, 4, 0x0020, 0x10, 9, 3), + PIN_FIELD_BASE(156, 156, 4, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(157, 157, 4, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(158, 158, 4, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(159, 159, 4, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(160, 160, 4, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(161, 161, 4, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(162, 162, 4, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(163, 163, 4, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(164, 164, 5, 0x0010, 0x10, 6, 3), + PIN_FIELD_BASE(165, 165, 5, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(166, 166, 5, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(167, 167, 5, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(168, 168, 5, 0x0010, 0x10, 9, 3), + PIN_FIELD_BASE(169, 169, 5, 0x0010, 0x10, 12, 3), + PIN_FIELD_BASE(170, 170, 5, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(171, 171, 6, 0x0000, 0x10, 0, 3), + PIN_FIELD_BASE(172, 172, 6, 0x0000, 0x10, 3, 3), + PIN_FIELD_BASE(173, 173, 6, 0x0000, 0x10, 6, 3), + PIN_FIELD_BASE(174, 174, 6, 0x0000, 0x10, 9, 3), + PIN_FIELD_BASE(175, 175, 6, 0x0000, 0x10, 12, 3), + PIN_FIELD_BASE(176, 176, 6, 0x0000, 0x10, 15, 3), + PIN_FIELD_BASE(177, 177, 6, 0x0000, 0x10, 18, 3), + PIN_FIELD_BASE(178, 178, 6, 0x0000, 0x10, 21, 3), + PIN_FIELD_BASE(179, 179, 6, 0x0000, 0x10, 24, 3), + PIN_FIELD_BASE(180, 180, 6, 0x0000, 0x10, 27, 3), + PIN_FIELD_BASE(181, 181, 10, 0x0000, 0x10, 0, 3), +}; + +static const struct mtk_pin_field_calc mt8901_pin_drv_adv_range[] = { + PIN_FIELD_BASE(8, 8, 2, 0x0030, 0x10, 6, 3), + PIN_FIELD_BASE(9, 9, 2, 0x0030, 0x10, 9, 3), + PIN_FIELD_BASE(10, 10, 2, 0x0030, 0x10, 12, 3), + PIN_FIELD_BASE(11, 11, 2, 0x0030, 0x10, 15, 3), + PIN_FIELD_BASE(12, 12, 1, 0x0040, 0x10, 0, 3), + PIN_FIELD_BASE(13, 13, 1, 0x0040, 0x10, 3, 3), + PIN_FIELD_BASE(17, 17, 1, 0x0040, 0x10, 6, 3), + PIN_FIELD_BASE(18, 18, 1, 0x0040, 0x10, 9, 3), + PIN_FIELD_BASE(52, 52, 2, 0x0030, 0x10, 0, 3), + PIN_FIELD_BASE(53, 53, 2, 0x0030, 0x10, 3, 3), + PIN_FIELD_BASE(54, 54, 5, 0x0030, 0x10, 24, 3), + PIN_FIELD_BASE(55, 55, 5, 0x0030, 0x10, 27, 3), + PIN_FIELD_BASE(56, 56, 1, 0x0040, 0x10, 12, 3), + PIN_FIELD_BASE(57, 57, 1, 0x0040, 0x10, 15, 3), + PIN_FIELD_BASE(60, 60, 9, 0x0020, 0x10, 0, 3), + PIN_FIELD_BASE(61, 61, 9, 0x0020, 0x10, 3, 3), + PIN_FIELD_BASE(66, 66, 5, 0x0030, 0x10, 0, 3), + PIN_FIELD_BASE(67, 67, 5, 0x0030, 0x10, 3, 3), + PIN_FIELD_BASE(70, 70, 7, 0x0030, 0x10, 0, 3), + PIN_FIELD_BASE(71, 71, 7, 0x0030, 0x10, 3, 3), + PIN_FIELD_BASE(72, 72, 7, 0x0030, 0x10, 6, 3), + PIN_FIELD_BASE(73, 73, 7, 0x0030, 0x10, 9, 3), + PIN_FIELD_BASE(88, 88, 7, 0x0030, 0x10, 12, 3), + PIN_FIELD_BASE(89, 89, 7, 0x0030, 0x10, 15, 3), + PIN_FIELD_BASE(92, 92, 3, 0x0040, 0x10, 12, 3), + PIN_FIELD_BASE(93, 93, 3, 0x0040, 0x10, 15, 3), + PIN_FIELD_BASE(117, 117, 4, 0x0030, 0x10, 0, 3), + PIN_FIELD_BASE(118, 118, 4, 0x0030, 0x10, 3, 3), + PIN_FIELD_BASE(119, 119, 4, 0x0030, 0x10, 6, 3), + PIN_FIELD_BASE(120, 120, 4, 0x0030, 0x10, 9, 3), + PIN_FIELD_BASE(121, 121, 4, 0x0030, 0x10, 12, 3), + PIN_FIELD_BASE(122, 122, 4, 0x0030, 0x10, 15, 3), + PIN_FIELD_BASE(123, 123, 4, 0x0030, 0x10, 18, 3), + PIN_FIELD_BASE(124, 124, 4, 0x0030, 0x10, 21, 3), + PIN_FIELD_BASE(126, 126, 5, 0x0030, 0x10, 12, 3), + PIN_FIELD_BASE(127, 127, 5, 0x0030, 0x10, 15, 3), + PIN_FIELD_BASE(128, 128, 5, 0x0030, 0x10, 18, 3), + PIN_FIELD_BASE(129, 129, 5, 0x0030, 0x10, 21, 3), + PIN_FIELD_BASE(134, 134, 3, 0x0040, 0x10, 0, 3), + PIN_FIELD_BASE(135, 135, 3, 0x0040, 0x10, 3, 3), + PIN_FIELD_BASE(136, 136, 3, 0x0040, 0x10, 6, 3), + PIN_FIELD_BASE(137, 137, 3, 0x0040, 0x10, 9, 3), + PIN_FIELD_BASE(165, 165, 5, 0x0030, 0x10, 6, 3), + PIN_FIELD_BASE(166, 166, 5, 0x0030, 0x10, 9, 3), +}; + +static const struct mtk_pin_field_calc mt8901_pin_rsel_range[] = { + PIN_FIELD_BASE(8, 8, 2, 0x0110, 0x10, 6, 3), + PIN_FIELD_BASE(9, 9, 2, 0x0110, 0x10, 9, 3), + PIN_FIELD_BASE(10, 10, 2, 0x0110, 0x10, 12, 3), + PIN_FIELD_BASE(11, 11, 2, 0x0110, 0x10, 15, 3), + PIN_FIELD_BASE(12, 12, 1, 0x0130, 0x10, 0, 1), + PIN_FIELD_BASE(13, 13, 1, 0x0130, 0x10, 1, 1), + PIN_FIELD_BASE(17, 17, 1, 0x0130, 0x10, 2, 1), + PIN_FIELD_BASE(18, 18, 1, 0x0130, 0x10, 3, 1), + PIN_FIELD_BASE(52, 52, 2, 0x0110, 0x10, 0, 3), + PIN_FIELD_BASE(53, 53, 2, 0x0110, 0x10, 3, 3), + PIN_FIELD_BASE(54, 54, 5, 0x0110, 0x10, 12, 3), + PIN_FIELD_BASE(55, 55, 5, 0x0110, 0x10, 15, 3), + PIN_FIELD_BASE(56, 56, 1, 0x0130, 0x10, 4, 1), + PIN_FIELD_BASE(57, 57, 1, 0x0130, 0x10, 5, 1), + PIN_FIELD_BASE(60, 60, 9, 0x00f0, 0x10, 0, 3), + PIN_FIELD_BASE(61, 61, 9, 0x00f0, 0x10, 3, 3), + PIN_FIELD_BASE(66, 66, 5, 0x0110, 0x10, 0, 1), + PIN_FIELD_BASE(67, 67, 5, 0x0110, 0x10, 1, 1), + PIN_FIELD_BASE(70, 70, 7, 0x0100, 0x10, 0, 3), + PIN_FIELD_BASE(71, 71, 7, 0x0100, 0x10, 3, 3), + PIN_FIELD_BASE(72, 72, 7, 0x0100, 0x10, 6, 3), + PIN_FIELD_BASE(73, 73, 7, 0x0100, 0x10, 9, 3), + PIN_FIELD_BASE(88, 88, 7, 0x0100, 0x10, 12, 3), + PIN_FIELD_BASE(89, 89, 7, 0x0100, 0x10, 15, 3), + PIN_FIELD_BASE(92, 92, 3, 0x0120, 0x10, 12, 3), + PIN_FIELD_BASE(93, 93, 3, 0x0120, 0x10, 15, 3), + PIN_FIELD_BASE(117, 117, 4, 0x0100, 0x10, 0, 3), + PIN_FIELD_BASE(118, 118, 4, 0x0100, 0x10, 3, 3), + PIN_FIELD_BASE(119, 119, 4, 0x0100, 0x10, 6, 3), + PIN_FIELD_BASE(120, 120, 4, 0x0100, 0x10, 9, 3), + PIN_FIELD_BASE(121, 121, 4, 0x0100, 0x10, 12, 3), + PIN_FIELD_BASE(122, 122, 4, 0x0100, 0x10, 15, 3), + PIN_FIELD_BASE(123, 123, 4, 0x0100, 0x10, 18, 3), + PIN_FIELD_BASE(124, 124, 4, 0x0100, 0x10, 21, 3), + PIN_FIELD_BASE(126, 126, 5, 0x0110, 0x10, 8, 1), + PIN_FIELD_BASE(127, 127, 5, 0x0110, 0x10, 9, 1), + PIN_FIELD_BASE(128, 128, 5, 0x0110, 0x10, 10, 1), + PIN_FIELD_BASE(129, 129, 5, 0x0110, 0x10, 11, 1), + PIN_FIELD_BASE(134, 134, 3, 0x0120, 0x10, 0, 3), + PIN_FIELD_BASE(135, 135, 3, 0x0120, 0x10, 3, 3), + PIN_FIELD_BASE(136, 136, 3, 0x0120, 0x10, 6, 3), + PIN_FIELD_BASE(137, 137, 3, 0x0120, 0x10, 9, 3), + PIN_FIELD_BASE(165, 165, 5, 0x0110, 0x10, 2, 3), + PIN_FIELD_BASE(166, 166, 5, 0x0110, 0x10, 5, 3), +}; + +static const struct mtk_pin_rsel mt8901_pin_rsel_val_range[] = { + 0 +}; + +static const unsigned int mt8901_pull_type[] = { + MTK_PULL_PUPD_R1R0_TYPE, /*0*/ + MTK_PULL_PUPD_R1R0_TYPE, /*1*/ + MTK_PULL_PUPD_R1R0_TYPE, /*2*/ + MTK_PULL_PUPD_R1R0_TYPE, /*3*/ + MTK_PULL_PUPD_R1R0_TYPE, /*4*/ + MTK_PULL_PUPD_R1R0_TYPE, /*5*/ + MTK_PULL_PUPD_R1R0_TYPE, /*6*/ + MTK_PULL_PUPD_R1R0_TYPE, /*7*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*8*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*9*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*10*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*11*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*12*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*13*/ + MTK_PULL_PUPD_R1R0_TYPE, /*14*/ + MTK_PULL_PUPD_R1R0_TYPE, /*15*/ + MTK_PULL_PUPD_R1R0_TYPE, /*16*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*17*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*18*/ + MTK_PULL_PUPD_R1R0_TYPE, /*19*/ + MTK_PULL_PUPD_R1R0_TYPE, /*20*/ + MTK_PULL_PUPD_R1R0_TYPE, /*21*/ + MTK_PULL_PUPD_R1R0_TYPE, /*22*/ + MTK_PULL_PUPD_R1R0_TYPE, /*23*/ + MTK_PULL_PUPD_R1R0_TYPE, /*24*/ + MTK_PULL_PUPD_R1R0_TYPE, /*25*/ + MTK_PULL_PUPD_R1R0_TYPE, /*26*/ + MTK_PULL_PUPD_R1R0_TYPE, /*27*/ + MTK_PULL_PUPD_R1R0_TYPE, /*28*/ + MTK_PULL_PUPD_R1R0_TYPE, /*29*/ + MTK_PULL_PUPD_R1R0_TYPE, /*30*/ + MTK_PULL_PUPD_R1R0_TYPE, /*31*/ + MTK_PULL_PUPD_R1R0_TYPE, /*32*/ + MTK_PULL_PUPD_R1R0_TYPE, /*33*/ + MTK_PULL_PUPD_R1R0_TYPE, /*34*/ + MTK_PULL_PUPD_R1R0_TYPE, /*35*/ + MTK_PULL_PUPD_R1R0_TYPE, /*36*/ + MTK_PULL_PUPD_R1R0_TYPE, /*37*/ + MTK_PULL_PUPD_R1R0_TYPE, /*38*/ + MTK_PULL_PUPD_R1R0_TYPE, /*39*/ + MTK_PULL_PUPD_R1R0_TYPE, /*40*/ + MTK_PULL_PUPD_R1R0_TYPE, /*41*/ + MTK_PULL_PUPD_R1R0_TYPE, /*42*/ + MTK_PULL_PUPD_R1R0_TYPE, /*43*/ + MTK_PULL_PUPD_R1R0_TYPE, /*44*/ + MTK_PULL_PUPD_R1R0_TYPE, /*45*/ + MTK_PULL_PUPD_R1R0_TYPE, /*46*/ + MTK_PULL_PUPD_R1R0_TYPE, /*47*/ + MTK_PULL_PUPD_R1R0_TYPE, /*48*/ + MTK_PULL_PUPD_R1R0_TYPE, /*49*/ + MTK_PULL_PUPD_R1R0_TYPE, /*50*/ + MTK_PULL_PUPD_R1R0_TYPE, /*51*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*52*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*53*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*54*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*55*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*56*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*57*/ + MTK_PULL_PUPD_R1R0_TYPE, /*58*/ + MTK_PULL_PU_PD_TYPE, /*59*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*60*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*61*/ + MTK_PULL_PUPD_R1R0_TYPE, /*62*/ + MTK_PULL_PUPD_R1R0_TYPE, /*63*/ + MTK_PULL_PUPD_R1R0_TYPE, /*64*/ + MTK_PULL_PUPD_R1R0_TYPE, /*65*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*66*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*67*/ + MTK_PULL_PUPD_R1R0_TYPE, /*68*/ + MTK_PULL_PU_PD_TYPE, /*69*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*70*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*71*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*72*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*73*/ + MTK_PULL_PUPD_R1R0_TYPE, /*74*/ + MTK_PULL_PUPD_R1R0_TYPE, /*75*/ + MTK_PULL_PUPD_R1R0_TYPE, /*76*/ + MTK_PULL_PUPD_R1R0_TYPE, /*77*/ + MTK_PULL_PUPD_R1R0_TYPE, /*78*/ + MTK_PULL_PUPD_R1R0_TYPE, /*79*/ + MTK_PULL_PUPD_R1R0_TYPE, /*80*/ + MTK_PULL_PUPD_R1R0_TYPE, /*81*/ + MTK_PULL_PUPD_R1R0_TYPE, /*82*/ + MTK_PULL_PUPD_R1R0_TYPE, /*83*/ + MTK_PULL_PUPD_R1R0_TYPE, /*84*/ + MTK_PULL_PUPD_R1R0_TYPE, /*85*/ + MTK_PULL_PUPD_R1R0_TYPE, /*86*/ + MTK_PULL_PUPD_R1R0_TYPE, /*87*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*88*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*89*/ + MTK_PULL_PUPD_R1R0_TYPE, /*90*/ + MTK_PULL_PUPD_R1R0_TYPE, /*91*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*92*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*93*/ + MTK_PULL_PUPD_R1R0_TYPE, /*94*/ + MTK_PULL_PUPD_R1R0_TYPE, /*95*/ + MTK_PULL_PUPD_R1R0_TYPE, /*96*/ + MTK_PULL_PUPD_R1R0_TYPE, /*97*/ + MTK_PULL_PUPD_R1R0_TYPE, /*98*/ + MTK_PULL_PUPD_R1R0_TYPE, /*99*/ + MTK_PULL_PUPD_R1R0_TYPE, /*100*/ + MTK_PULL_PUPD_R1R0_TYPE, /*101*/ + MTK_PULL_PUPD_R1R0_TYPE, /*102*/ + MTK_PULL_PUPD_R1R0_TYPE, /*103*/ + MTK_PULL_PUPD_R1R0_TYPE, /*104*/ + MTK_PULL_PUPD_R1R0_TYPE, /*105*/ + MTK_PULL_PUPD_R1R0_TYPE, /*106*/ + MTK_PULL_PUPD_R1R0_TYPE, /*107*/ + MTK_PULL_PUPD_R1R0_TYPE, /*108*/ + MTK_PULL_PUPD_R1R0_TYPE, /*109*/ + MTK_PULL_PUPD_R1R0_TYPE, /*110*/ + MTK_PULL_PUPD_R1R0_TYPE, /*111*/ + MTK_PULL_PUPD_R1R0_TYPE, /*112*/ + MTK_PULL_PUPD_R1R0_TYPE, /*113*/ + MTK_PULL_PUPD_R1R0_TYPE, /*114*/ + MTK_PULL_PUPD_R1R0_TYPE, /*115*/ + MTK_PULL_PUPD_R1R0_TYPE, /*116*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*117*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*118*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*119*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*120*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*121*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*122*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*123*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*124*/ + MTK_PULL_PUPD_R1R0_TYPE, /*125*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*126*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*127*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*128*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*129*/ + MTK_PULL_PUPD_R1R0_TYPE, /*130*/ + MTK_PULL_PUPD_R1R0_TYPE, /*131*/ + MTK_PULL_PUPD_R1R0_TYPE, /*132*/ + MTK_PULL_PUPD_R1R0_TYPE, /*133*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*134*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*135*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*136*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*137*/ + MTK_PULL_PUPD_R1R0_TYPE, /*138*/ + MTK_PULL_PUPD_R1R0_TYPE, /*139*/ + MTK_PULL_PUPD_R1R0_TYPE, /*140*/ + MTK_PULL_PUPD_R1R0_TYPE, /*141*/ + MTK_PULL_PUPD_R1R0_TYPE, /*142*/ + MTK_PULL_PUPD_R1R0_TYPE, /*143*/ + MTK_PULL_PUPD_R1R0_TYPE, /*144*/ + MTK_PULL_PUPD_R1R0_TYPE, /*145*/ + MTK_PULL_PUPD_R1R0_TYPE, /*146*/ + MTK_PULL_PUPD_R1R0_TYPE, /*147*/ + MTK_PULL_PUPD_R1R0_TYPE, /*148*/ + MTK_PULL_PUPD_R1R0_TYPE, /*149*/ + MTK_PULL_PUPD_R1R0_TYPE, /*150*/ + MTK_PULL_PUPD_R1R0_TYPE, /*151*/ + MTK_PULL_PUPD_R1R0_TYPE, /*152*/ + MTK_PULL_PUPD_R1R0_TYPE, /*153*/ + MTK_PULL_PUPD_R1R0_TYPE, /*154*/ + MTK_PULL_PUPD_R1R0_TYPE, /*155*/ + MTK_PULL_PUPD_R1R0_TYPE, /*156*/ + MTK_PULL_PUPD_R1R0_TYPE, /*157*/ + MTK_PULL_PUPD_R1R0_TYPE, /*158*/ + MTK_PULL_PUPD_R1R0_TYPE, /*159*/ + MTK_PULL_PUPD_R1R0_TYPE, /*160*/ + MTK_PULL_PUPD_R1R0_TYPE, /*161*/ + MTK_PULL_PUPD_R1R0_TYPE, /*162*/ + MTK_PULL_PUPD_R1R0_TYPE, /*163*/ + MTK_PULL_PUPD_R1R0_TYPE, /*164*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*165*/ + MTK_PULL_PU_PD_RSEL_TYPE, /*166*/ + MTK_PULL_PUPD_R1R0_TYPE, /*167*/ + MTK_PULL_PUPD_R1R0_TYPE, /*168*/ + MTK_PULL_PUPD_R1R0_TYPE, /*169*/ + MTK_PULL_PUPD_R1R0_TYPE, /*170*/ + MTK_PULL_PUPD_R1R0_TYPE, /*171*/ + MTK_PULL_PUPD_R1R0_TYPE, /*172*/ + MTK_PULL_PUPD_R1R0_TYPE, /*173*/ + MTK_PULL_PUPD_R1R0_TYPE, /*174*/ + MTK_PULL_PUPD_R1R0_TYPE, /*175*/ + MTK_PULL_PUPD_R1R0_TYPE, /*176*/ + MTK_PULL_PUPD_R1R0_TYPE, /*177*/ + MTK_PULL_PUPD_R1R0_TYPE, /*178*/ + MTK_PULL_PUPD_R1R0_TYPE, /*179*/ + MTK_PULL_PUPD_R1R0_TYPE, /*180*/ + MTK_PULL_PU_PD_TYPE, /*181*/ +}; + +static const struct mtk_pin_reg_calc mt8901_reg_cals[PINCTRL_PIN_REG_MAX] = { + [PINCTRL_PIN_REG_MODE] = MTK_RANGE(mt8901_pin_mode_range), + [PINCTRL_PIN_REG_DIR] = MTK_RANGE(mt8901_pin_dir_range), + [PINCTRL_PIN_REG_DI] = MTK_RANGE(mt8901_pin_di_range), + [PINCTRL_PIN_REG_DO] = MTK_RANGE(mt8901_pin_do_range), + [PINCTRL_PIN_REG_SMT] = MTK_RANGE(mt8901_pin_smt_range), + [PINCTRL_PIN_REG_IES] = MTK_RANGE(mt8901_pin_ies_range), + [PINCTRL_PIN_REG_PUPD] = MTK_RANGE(mt8901_pin_pupd_range), + [PINCTRL_PIN_REG_R0] = MTK_RANGE(mt8901_pin_r0_range), + [PINCTRL_PIN_REG_R1] = MTK_RANGE(mt8901_pin_r1_range), + [PINCTRL_PIN_REG_PU] = MTK_RANGE(mt8901_pin_pu_range), + [PINCTRL_PIN_REG_PD] = MTK_RANGE(mt8901_pin_pd_range), + [PINCTRL_PIN_REG_DRV] = MTK_RANGE(mt8901_pin_drv_range), + [PINCTRL_PIN_REG_DRV_ADV] = MTK_RANGE(mt8901_pin_drv_adv_range), + [PINCTRL_PIN_REG_RSEL] = MTK_RANGE(mt8901_pin_rsel_range), +}; + +static const char * const mt8901_pinctrl_register_base_name[] = { + "iocfg0", "iocfg_lt2", "iocfg_lt3", "iocfg_rt1", "iocfg_rt2", "iocfg_rt3", + "iocfg_tr", "iocfg_rt0", "iocfg_lt1", "iocfg_lb", "iocfg_rb", +}; + +static const struct mtk_eint_hw mt8901_eint_hw = { + .port_mask = 0xf, + .ports = 7, + .ap_num = 209, + .db_cnt = 32, + .db_time = debounce_time_mt8901, +}; + +static const struct mtk_pin_soc mt8901_data = { + .reg_cal = mt8901_reg_cals, + .pins = mtk_pins_mt8901, + .npins = ARRAY_SIZE(mtk_pins_mt8901), + .ngrps = ARRAY_SIZE(mtk_pins_mt8901), + .eint_hw = &mt8901_eint_hw, + .eint_pin = eint_pins_mt8901, + .nfuncs = 8, + .gpio_m = 0, + .base_names = mt8901_pinctrl_register_base_name, + .nbase_names = ARRAY_SIZE(mt8901_pinctrl_register_base_name), + .pull_type = mt8901_pull_type, + .pin_rsel = mt8901_pin_rsel_val_range, + .npin_rsel = ARRAY_SIZE(mt8901_pin_rsel_val_range), /*numsel*/ + .bias_set_combo = mtk_pinconf_bias_set_combo, + .bias_get_combo = mtk_pinconf_bias_get_combo, + .drive_set = mtk_pinconf_drive_set_rev1, + .drive_get = mtk_pinconf_drive_get_rev1, + .adv_drive_set = mtk_pinconf_adv_drive_set_raw, + .adv_drive_get = mtk_pinconf_adv_drive_get_raw, +}; + +static const struct acpi_device_id mt8901_pinctrl_acpi_match[] = { + {"NVDA9221", (kernel_ulong_t)&mt8901_data }, + { } +}; +MODULE_DEVICE_TABLE(acpi, mt8901_pinctrl_acpi_match); + +static struct platform_driver mt8901_pinctrl_driver = { + .driver = { + .name = "mt8901-pinctrl", + .acpi_match_table = ACPI_PTR(mt8901_pinctrl_acpi_match), + .pm = pm_sleep_ptr(&mtk_paris_pinctrl_pm_ops) + }, + .probe = mtk_paris_pinctrl_probe, +}; + +static int __init mt8901_pinctrl_init(void) +{ + return platform_driver_register(&mt8901_pinctrl_driver); +} + +arch_initcall(mt8901_pinctrl_init); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("MediaTek MT8901 Pinctrl Driver"); diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-mt8901.h b/drivers/pinctrl/mediatek/pinctrl-mtk-mt8901.h new file mode 100644 index 0000000000000..fc64fc6ff5f9b --- /dev/null +++ b/drivers/pinctrl/mediatek/pinctrl-mtk-mt8901.h @@ -0,0 +1,2130 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2025 MediaTek Inc. + * + */ + +#ifndef __PINCTRL_MTK_MT8901_H +#define __PINCTRL_MTK_MT8901_H + +#include "pinctrl-paris.h" + +#define INVALID_BASE 0xFF + +static const struct mtk_pin_desc mtk_pins_mt8901[] = { + MTK_PIN( + 0, "GPIO0", + MTK_EINT_FUNCTION(0, 0), + DRV_GRP4, + MTK_FUNCTION(0, "B:GPIO0"), + MTK_FUNCTION(1, "O:ESPI_SCK") + ), + MTK_PIN( + 1, "GPIO1", + MTK_EINT_FUNCTION(0, 1), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO1"), + MTK_FUNCTION(1, "B1_ESPI_IO0") + ), + MTK_PIN( + 2, "GPIO2", + MTK_EINT_FUNCTION(0, 2), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO2"), + MTK_FUNCTION(1, "B1_ESPI_IO1") + ), + MTK_PIN( + 3, "GPIO3", + MTK_EINT_FUNCTION(0, 3), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO3"), + MTK_FUNCTION(1, "B1_ESPI_IO2") + ), + MTK_PIN( + 4, "GPIO4", + MTK_EINT_FUNCTION(0, 4), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO4"), + MTK_FUNCTION(1, "B1_ESPI_IO3") + ), + MTK_PIN( + 5, "GPIO5", + MTK_EINT_FUNCTION(0, 5), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO5"), + MTK_FUNCTION(1, "O_ESPI_CSN") + ), + MTK_PIN( + 6, "GPIO6", + MTK_EINT_FUNCTION(0, 6), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO6"), + MTK_FUNCTION(1, "O_ESPI_RESET_O"), + MTK_FUNCTION(2, "I1_ESPI_RESET_I") + ), + MTK_PIN( + 7, "GPIO7", + MTK_EINT_FUNCTION(0, 7), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO7"), + MTK_FUNCTION(1, "I1_ESPI_ALERT") + ), + MTK_PIN( + 8, "GPIO8", + MTK_EINT_FUNCTION(0, 8), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO8"), + MTK_FUNCTION(1, "B1_I2C_SCL3"), + MTK_FUNCTION(2, "B1_DISP_SCL2"), + MTK_FUNCTION(4, "O_PMSR_SMAP"), + MTK_FUNCTION(6, "O_MD32_0_TXD"), + MTK_FUNCTION(7, "O_MD32_1_TXD") + ), + MTK_PIN( + 9, "GPIO9", + MTK_EINT_FUNCTION(0, 9), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO9"), + MTK_FUNCTION(1, "B1_I2C_SDA3"), + MTK_FUNCTION(2, "B1_DISP_SDA2"), + MTK_FUNCTION(4, "O_PMSR_SMAP_MAX"), + MTK_FUNCTION(6, "I1_MD32_0_RXD"), + MTK_FUNCTION(7, "I1_MD32_1_RXD") + ), + MTK_PIN( + 10, "GPIO10", + MTK_EINT_FUNCTION(0, 10), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO10"), + MTK_FUNCTION(1, "B1_I2C_SCL4"), + MTK_FUNCTION(2, "B1_DISP_SCL2"), + MTK_FUNCTION(4, "O_PMSR_SMAP_MAX_W"), + MTK_FUNCTION(6, "O_MD32_0_GPIO0"), + MTK_FUNCTION(7, "O_MD32_1_GPIO0") + ), + MTK_PIN( + 11, "GPIO11", + MTK_EINT_FUNCTION(0, 11), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO11"), + MTK_FUNCTION(1, "B1_I2C_SDA4"), + MTK_FUNCTION(2, "B1_DISP_SDA2") + ), + MTK_PIN( + 12, "GPIO12", + MTK_EINT_FUNCTION(0, 12), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO12"), + MTK_FUNCTION(1, "B0_SPMI_M_SCL"), + MTK_FUNCTION(2, "B0_TP_GPIO31_AO") + ), + MTK_PIN( + 13, "GPIO13", + MTK_EINT_FUNCTION(0, 13), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO13"), + MTK_FUNCTION(1, "B0_SPMI_M_SDA"), + MTK_FUNCTION(2, "B0_TP_GPIO6_AO") + ), + MTK_PIN( + 14, "GPIO14", + MTK_EINT_FUNCTION(0, 14), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO14"), + MTK_FUNCTION(1, "I0_DPAUX_HPD_IN_2"), + MTK_FUNCTION(7, "O_DBG_MON_A0") + ), + MTK_PIN( + 15, "GPIO15", + MTK_EINT_FUNCTION(0, 15), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO15"), + MTK_FUNCTION(1, "I0_DPAUX_HPD_IN_3"), + MTK_FUNCTION(2, "B0_TP_GPIO25_AO"), + MTK_FUNCTION(7, "O_DBG_MON_A1") + ), + MTK_PIN( + 16, "GPIO16", + MTK_EINT_FUNCTION(0, 16), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO16"), + MTK_FUNCTION(1, "O_USB4_L_TCPC_RESET"), + MTK_FUNCTION(7, "O_DBG_MON_A18") + ), + MTK_PIN( + 17, "GPIO17", + MTK_EINT_FUNCTION(0, 17), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO17"), + MTK_FUNCTION(1, "B0_SPMI_P_SCL") + ), + MTK_PIN( + 18, "GPIO18", + MTK_EINT_FUNCTION(0, 18), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO18"), + MTK_FUNCTION(1, "B0_SPMI_P_SDA") + ), + MTK_PIN( + 19, "GPIO19", + MTK_EINT_FUNCTION(0, 19), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO19"), + MTK_FUNCTION(1, "B0_TP_GPIO29_AO") + ), + MTK_PIN( + 20, "GPIO20", + MTK_EINT_FUNCTION(0, 20), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO20"), + MTK_FUNCTION(1, "B0_TP_GPIO30_AO") + ), + MTK_PIN( + 21, "GPIO21", + MTK_EINT_FUNCTION(0, 21), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO21"), + MTK_FUNCTION(1, "B1_PROCHOT") + ), + MTK_PIN( + 22, "GPIO22", + MTK_EINT_FUNCTION(0, 22), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO22"), + MTK_FUNCTION(1, "I0_RTC32K_CK") + ), + MTK_PIN( + 23, "GPIO23", + MTK_EINT_FUNCTION(0, 23), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO23"), + MTK_FUNCTION(1, "B0_TP_GPIO0_AO") + ), + MTK_PIN( + 24, "GPIO24", + MTK_EINT_FUNCTION(0, 24), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO24"), + MTK_FUNCTION(2, "B0_TP_GPIO1_AO"), + MTK_FUNCTION(3, "O_CMMCLK1"), + MTK_FUNCTION(4, "O_SROOT_GPIO_O"), + MTK_FUNCTION(5, "O_MD32_10_TXD"), + MTK_FUNCTION(6, "O_MD32_11_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_A3") + ), + MTK_PIN( + 25, "GPIO25", + MTK_EINT_FUNCTION(0, 25), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO25"), + MTK_FUNCTION(1, "B0_TP_GPIO2_AO"), + MTK_FUNCTION(2, "I0_VBUSVALID_0P"), + MTK_FUNCTION(4, "I0_SROOT_GPIO_I"), + MTK_FUNCTION(5, "I1_MD32_10_RXD"), + MTK_FUNCTION(6, "I1_MD32_11_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_A4") + ), + MTK_PIN( + 26, "GPIO26", + MTK_EINT_FUNCTION(0, 26), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO26"), + MTK_FUNCTION(1, "B0_TP_GPIO3_AO"), + MTK_FUNCTION(5, "O_MD32_12_TXD"), + MTK_FUNCTION(6, "O_MD32_13_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_A5") + ), + MTK_PIN( + 27, "GPIO27", + MTK_EINT_FUNCTION(0, 27), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO27"), + MTK_FUNCTION(1, "B0_TP_GPIO4_AO"), + MTK_FUNCTION(5, "I1_MD32_12_RXD"), + MTK_FUNCTION(6, "I1_MD32_13_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_A6") + ), + MTK_PIN( + 28, "GPIO28", + MTK_EINT_FUNCTION(0, 28), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO28"), + MTK_FUNCTION(1, "B0_TP_GPIO5_AO"), + MTK_FUNCTION(5, "O_MD32_12_GPIO0"), + MTK_FUNCTION(6, "O_MD32_13_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_A7") + ), + MTK_PIN( + 29, "GPIO29", + MTK_EINT_FUNCTION(0, 29), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO29"), + MTK_FUNCTION(1, "B1_THERMTRIP") + ), + MTK_PIN( + 30, "GPIO30", + MTK_EINT_FUNCTION(0, 30), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO30"), + MTK_FUNCTION(1, "B0_TP_GPIO7_AO"), + MTK_FUNCTION(2, "O_CMMCLK0"), + MTK_FUNCTION(4, "I0_CLUSTER0_SLV_CPUEB_JTAG_TRSTN"), + MTK_FUNCTION(5, "I0_CLUSTER1_SLV_CPUEB_JTAG_TRSTN"), + MTK_FUNCTION(6, "I0_OSROOT_GPIO_I"), + MTK_FUNCTION(7, "O_DBG_MON_A8") + ), + MTK_PIN( + 31, "GPIO31", + MTK_EINT_FUNCTION(0, 31), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO31"), + MTK_FUNCTION(1, "B0_TP_GPIO8_AO"), + MTK_FUNCTION(2, "O_CMMCLK1"), + MTK_FUNCTION(4, "I1_CLUSTER0_SLV_CPUEB_JTAG_TMS"), + MTK_FUNCTION(5, "I1_CLUSTER1_SLV_CPUEB_JTAG_TMS"), + MTK_FUNCTION(6, "O_OSROOT_GPIO_O"), + MTK_FUNCTION(7, "O_DBG_MON_A9") + ), + MTK_PIN( + 32, "GPIO32", + MTK_EINT_FUNCTION(0, 32), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO32"), + MTK_FUNCTION(1, "B0_TP_GPIO9_AO"), + MTK_FUNCTION(5, "O_SROOT_UTX"), + MTK_FUNCTION(6, "I1_TP_UCTS1_VLP"), + MTK_FUNCTION(7, "O_DBG_MON_A10") + ), + MTK_PIN( + 33, "GPIO33", + MTK_EINT_FUNCTION(0, 33), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO33"), + MTK_FUNCTION(1, "B0_TP_GPIO10_AO"), + MTK_FUNCTION(2, "O_CMMCLK2"), + MTK_FUNCTION(3, "I0_VBUSVALID_1P"), + MTK_FUNCTION(4, "I1_CLUSTER0_SLV_CPUEB_JTAG_TCK"), + MTK_FUNCTION(5, "I1_CLUSTER1_SLV_CPUEB_JTAG_TCK"), + MTK_FUNCTION(6, "I0_SROOT_TCK"), + MTK_FUNCTION(7, "I0_OSROOT_TCK") + ), + MTK_PIN( + 34, "GPIO34", + MTK_EINT_FUNCTION(0, 34), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO34"), + MTK_FUNCTION(1, "B0_TP_GPIO11_AO"), + MTK_FUNCTION(2, "O_CMMCLK4"), + MTK_FUNCTION(3, "I0_VBUSVALID_3P"), + MTK_FUNCTION(4, "I1_CLUSTER0_SLV_CPUEB_JTAG_TDI"), + MTK_FUNCTION(5, "I1_CLUSTER1_SLV_CPUEB_JTAG_TDI"), + MTK_FUNCTION(6, "I0_SROOT_TDI"), + MTK_FUNCTION(7, "I0_OSROOT_TDI") + ), + MTK_PIN( + 35, "GPIO35", + MTK_EINT_FUNCTION(0, 35), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO35"), + MTK_FUNCTION(1, "B0_TP_GPIO12_AO"), + MTK_FUNCTION(2, "O_SCP_PWM_1_VLP"), + MTK_FUNCTION(4, "O_CLUSTER0_SLV_CPUEB_JTAG_TDO"), + MTK_FUNCTION(5, "O_CLUSTER1_SLV_CPUEB_JTAG_TDO"), + MTK_FUNCTION(6, "O_SROOT_TDO"), + MTK_FUNCTION(7, "O_OSROOT_TDO") + ), + MTK_PIN( + 36, "GPIO36", + MTK_EINT_FUNCTION(0, 36), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO36"), + MTK_FUNCTION(1, "B0_TP_GPIO22_AO"), + MTK_FUNCTION(5, "I1_SROOT_URX"), + MTK_FUNCTION(6, "O_TP_URTS1_VLP"), + MTK_FUNCTION(7, "O_DBG_MON_A31") + ), + MTK_PIN( + 37, "GPIO37", + MTK_EINT_FUNCTION(0, 37), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO37"), + MTK_FUNCTION(1, "B0_TP_GPIO23_AO"), + MTK_FUNCTION(2, "O_CMMCLK3"), + MTK_FUNCTION(4, "O_SCP_PWM_2_VLP"), + MTK_FUNCTION(6, "O_MD32_5_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_A11") + ), + MTK_PIN( + 38, "GPIO38", + MTK_EINT_FUNCTION(0, 38), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO38"), + MTK_FUNCTION(1, "B0_TP_GPIO24_AO"), + MTK_FUNCTION(2, "O_SCP_VREQ_VAO") + ), + MTK_PIN( + 39, "GPIO39", + MTK_EINT_FUNCTION(0, 39), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO39"), + MTK_FUNCTION(1, "B0_TP_GPIO13_AO"), + MTK_FUNCTION(2, "O_SCP_PWM_2_VLP"), + MTK_FUNCTION(3, "O_CMMCLK0"), + MTK_FUNCTION(4, "I0_VBUSVALID_2P"), + MTK_FUNCTION(5, "O_MD32_10_GPIO0"), + MTK_FUNCTION(6, "I0_SROOT_TMS"), + MTK_FUNCTION(7, "I0_OSROOT_TMS") + ), + MTK_PIN( + 40, "GPIO40", + MTK_EINT_FUNCTION(0, 40), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO40"), + MTK_FUNCTION(1, "B0_TP_GPIO14_AO"), + MTK_FUNCTION(2, "I0_VBUSVALID_1P"), + MTK_FUNCTION(3, "O_URTS2"), + MTK_FUNCTION(4, "O_TP_URTS2_VLP"), + MTK_FUNCTION(5, "O_SPMI_P_TRIG_FLAG"), + MTK_FUNCTION(6, "I1_MD32_5_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_A13") + ), + MTK_PIN( + 41, "GPIO41", + MTK_EINT_FUNCTION(0, 41), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO41"), + MTK_FUNCTION(1, "B0_TP_GPIO15_AO"), + MTK_FUNCTION(2, "I0_VBUSVALID_0P"), + MTK_FUNCTION(3, "I1_UCTS2"), + MTK_FUNCTION(4, "I1_TP_UCTS2_VLP"), + MTK_FUNCTION(5, "O_SPMI_S_TRIG_FLAG"), + MTK_FUNCTION(6, "O_MD32_5_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_A12") + ), + MTK_PIN( + 42, "GPIO42", + MTK_EINT_FUNCTION(0, 42), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO42"), + MTK_FUNCTION(1, "B0_TP_GPIO16_AO"), + MTK_FUNCTION(2, "O_CMMCLK3"), + MTK_FUNCTION(3, "O_UTXD2"), + MTK_FUNCTION(4, "O_TP_UTXD2_VLP"), + MTK_FUNCTION(5, "O_SPMI_M_TRIG_FLAG"), + MTK_FUNCTION(6, "I0_SROOT_NTRST"), + MTK_FUNCTION(7, "I0_OSROOT_NTRST") + ), + MTK_PIN( + 43, "GPIO43", + MTK_EINT_FUNCTION(0, 43), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO43"), + MTK_FUNCTION(1, "B0_TP_GPIO17_AO"), + MTK_FUNCTION(2, "O_CMMCLK4"), + MTK_FUNCTION(3, "I1_URXD2"), + MTK_FUNCTION(4, "I1_TP_URXD2_VLP"), + MTK_FUNCTION(5, "O_MD32_4_TXD"), + MTK_FUNCTION(6, "O_MD32PCM_UTXD_AO_VLP") + ), + MTK_PIN( + 44, "GPIO44", + MTK_EINT_FUNCTION(0, 44), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO44"), + MTK_FUNCTION(1, "B0_TP_GPIO18_AO"), + MTK_FUNCTION(2, "I1_TP_UCTS2_VLP"), + MTK_FUNCTION(3, "I1_UCTS2"), + MTK_FUNCTION(5, "I1_MD32_4_RXD"), + MTK_FUNCTION(6, "I1_MD32PCM_URXD_AO_VLP") + ), + MTK_PIN( + 45, "GPIO45", + MTK_EINT_FUNCTION(0, 45), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO45"), + MTK_FUNCTION(1, "B0_TP_GPIO19_AO"), + MTK_FUNCTION(2, "O_TP_URTS2_VLP"), + MTK_FUNCTION(3, "O_URTS2"), + MTK_FUNCTION(5, "O_MD32_4_GPIO0"), + MTK_FUNCTION(6, "O_MD32_11_GPIO0") + ), + MTK_PIN( + 46, "GPIO46", + MTK_EINT_FUNCTION(0, 46), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO46"), + MTK_FUNCTION(1, "B0_TP_GPIO20_AO"), + MTK_FUNCTION(2, "I0_VBUSVALID_2P"), + MTK_FUNCTION(3, "O_SCP_VREQ_VAO"), + MTK_FUNCTION(4, "O_SCP_PWM_1_VLP"), + MTK_FUNCTION(5, "O_SROOT_GPIO_O"), + MTK_FUNCTION(6, "O_OSROOT_GPIO_O") + ), + MTK_PIN( + 47, "GPIO47", + MTK_EINT_FUNCTION(0, 47), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO47"), + MTK_FUNCTION(1, "B0_TP_GPIO21_AO"), + MTK_FUNCTION(2, "I0_VBUSVALID_3P"), + MTK_FUNCTION(3, "O_CMMCLK2"), + MTK_FUNCTION(5, "I0_SROOT_GPIO_I"), + MTK_FUNCTION(6, "I0_OSROOT_GPIO_I"), + MTK_FUNCTION(7, "O_DBG_MON_A2") + ), + MTK_PIN( + 48, "GPIO48", + MTK_EINT_FUNCTION(0, 48), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO48"), + MTK_FUNCTION(1, "O_UTXD0"), + MTK_FUNCTION(2, "O_TP_UTXD1_VLP"), + MTK_FUNCTION(6, "O_ADSP_UTXD0"), + MTK_FUNCTION(7, "O_DBG_MON_A19") + ), + MTK_PIN( + 49, "GPIO49", + MTK_EINT_FUNCTION(0, 49), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO49"), + MTK_FUNCTION(1, "I1_URXD0"), + MTK_FUNCTION(2, "I1_TP_URXD1_VLP"), + MTK_FUNCTION(6, "I1_ADSP_URXD0"), + MTK_FUNCTION(7, "O_DBG_MON_A20") + ), + MTK_PIN( + 50, "GPIO50", + MTK_EINT_FUNCTION(0, 50), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO50"), + MTK_FUNCTION(1, "O_TP_UTXD2_VLP"), + MTK_FUNCTION(2, "O_UTXD2"), + MTK_FUNCTION(4, "B0_TP_GPIO26_AO"), + MTK_FUNCTION(5, "O_TP_UTXD1_VLP"), + MTK_FUNCTION(7, "O_SROOT_UTX") + ), + MTK_PIN( + 51, "GPIO51", + MTK_EINT_FUNCTION(0, 51), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO51"), + MTK_FUNCTION(1, "I1_TP_URXD2_VLP"), + MTK_FUNCTION(2, "I1_URXD2"), + MTK_FUNCTION(4, "B0_TP_GPIO27_AO"), + MTK_FUNCTION(5, "I1_TP_URXD1_VLP"), + MTK_FUNCTION(7, "I1_SROOT_URX") + ), + MTK_PIN( + 52, "GPIO52", + MTK_EINT_FUNCTION(0, 52), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO52"), + MTK_FUNCTION(1, "B1_USB4_L_PD_SCL"), + MTK_FUNCTION(7, "O_ADSP_UTXD0") + ), + MTK_PIN( + 53, "GPIO53", + MTK_EINT_FUNCTION(0, 53), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO53"), + MTK_FUNCTION(1, "B1_USB4_L_PD_SDA"), + MTK_FUNCTION(5, "O_MD32_7_TXD"), + MTK_FUNCTION(6, "O_MD32_6_TXD"), + MTK_FUNCTION(7, "I1_ADSP_URXD0") + ), + MTK_PIN( + 54, "GPIO54", + MTK_EINT_FUNCTION(0, 54), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO54"), + MTK_FUNCTION(1, "B1_USB4_R_PD_SCL"), + MTK_FUNCTION(4, "I1_CKM_SCL"), + MTK_FUNCTION(5, "I1_MD32_7_RXD"), + MTK_FUNCTION(6, "I1_USB4_L_PAR_SCL"), + MTK_FUNCTION(7, "O_PBUD_CTRL_UTXD_AO_VLP") + ), + MTK_PIN( + 55, "GPIO55", + MTK_EINT_FUNCTION(0, 55), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO55"), + MTK_FUNCTION(1, "B1_USB4_R_PD_SDA"), + MTK_FUNCTION(4, "B1_CKM_SDA"), + MTK_FUNCTION(5, "O_MD32_7_GPIO0"), + MTK_FUNCTION(6, "B1_USB4_L_PAR_SDA"), + MTK_FUNCTION(7, "I1_PBUD_CTRL_URXD_AO_VLP") + ), + MTK_PIN( + 56, "GPIO56", + MTK_EINT_FUNCTION(0, 56), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO56"), + MTK_FUNCTION(1, "B0_SPMI_S_SCL") + ), + MTK_PIN( + 57, "GPIO57", + MTK_EINT_FUNCTION(0, 57), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO57"), + MTK_FUNCTION(1, "B0_SPMI_S_SDA") + ), + MTK_PIN( + 58, "GPIO58", + MTK_EINT_FUNCTION(0, 58), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO58"), + MTK_FUNCTION(1, "O_WATCHDOG") + ), + MTK_PIN( + 59, "GPIO59", + MTK_EINT_FUNCTION(0, 59), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO59"), + MTK_FUNCTION(1, "B0_PAD_RESET_DRAM_0") + ), + MTK_PIN( + 60, "GPIO60", + MTK_EINT_FUNCTION(0, 60), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO60"), + MTK_FUNCTION(1, "B1_I2C_SCL0"), + MTK_FUNCTION(5, "O_MD32_2_TXD"), + MTK_FUNCTION(6, "O_MD32_3_TXD") + ), + MTK_PIN( + 61, "GPIO61", + MTK_EINT_FUNCTION(0, 61), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO61"), + MTK_FUNCTION(1, "B1_I2C_SDA0"), + MTK_FUNCTION(5, "I1_MD32_2_RXD"), + MTK_FUNCTION(6, "I1_MD32_3_RXD") + ), + MTK_PIN( + 62, "GPIO62", + MTK_EINT_FUNCTION(0, 62), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO62"), + MTK_FUNCTION(1, "I0_DMIC0_DAT"), + MTK_FUNCTION(6, "O_TP_UTXD1_VLP"), + MTK_FUNCTION(7, "O_DBG_MON_B0") + ), + MTK_PIN( + 63, "GPIO63", + MTK_EINT_FUNCTION(0, 63), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO63"), + MTK_FUNCTION(1, "O_DMIC0_CLK"), + MTK_FUNCTION(6, "I1_TP_URXD1_VLP"), + MTK_FUNCTION(7, "O_DBG_MON_B1") + ), + MTK_PIN( + 64, "GPIO64", + MTK_EINT_FUNCTION(0, 64), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO64"), + MTK_FUNCTION(1, "I0_DMIC1_DAT"), + MTK_FUNCTION(5, "O_MD32_2_GPIO0"), + MTK_FUNCTION(6, "O_MD32_3_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_B2") + ), + MTK_PIN( + 65, "GPIO65", + MTK_EINT_FUNCTION(0, 65), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO65"), + MTK_FUNCTION(1, "O_DMIC1_CLK"), + MTK_FUNCTION(7, "O_DBG_MON_B3") + ), + MTK_PIN( + 66, "GPIO66", + MTK_EINT_FUNCTION(0, 66), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO66"), + MTK_FUNCTION(1, "O_SOUNDWIRE0_CK"), + MTK_FUNCTION(3, "I1_TP_UCTS0_VLP"), + MTK_FUNCTION(4, "O_SPI_HID_IRQ_S_MON0"), + MTK_FUNCTION(7, "O_VADSP_UTXD0") + ), + MTK_PIN( + 67, "GPIO67", + MTK_EINT_FUNCTION(0, 67), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO67"), + MTK_FUNCTION(1, "B0_SOUNDWIRE0_D0"), + MTK_FUNCTION(3, "O_TP_URTS0_VLP"), + MTK_FUNCTION(7, "I1_VADSP_URXD0") + ), + MTK_PIN( + 68, "GPIO68", + MTK_EINT_FUNCTION(0, 68), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO68"), + MTK_FUNCTION(1, "O_SCP_PWM_0_VLP"), + MTK_FUNCTION(2, "O_PWM_VLP"), + MTK_FUNCTION(3, "B0_TP_GPIO28_AO") + ), + MTK_PIN( + 69, "GPIO69", + MTK_EINT_FUNCTION(0, 69), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO69"), + MTK_FUNCTION(1, "B0_PAD_RESET_DRAM_8") + ), + MTK_PIN( + 70, "GPIO70", + MTK_EINT_FUNCTION(0, 70), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO70"), + MTK_FUNCTION(1, "B1_SCP_SCL0"), + MTK_FUNCTION(4, "I0_VADSP_JTAG0_TCK"), + MTK_FUNCTION(5, "I1_PCIE4_USB3_PAR_SCL"), + MTK_FUNCTION(6, "I1_SCP_JTAG0_TCK_VLP"), + MTK_FUNCTION(7, "B1_SROOT_SCL") + ), + MTK_PIN( + 71, "GPIO71", + MTK_EINT_FUNCTION(0, 71), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO71"), + MTK_FUNCTION(1, "B1_SCP_SDA0"), + MTK_FUNCTION(4, "I1_VADSP_JTAG0_TMS"), + MTK_FUNCTION(5, "B1_PCIE4_USB3_PAR_SDA"), + MTK_FUNCTION(6, "B1_SCP_JTAG0_TMS_VLP"), + MTK_FUNCTION(7, "B1_SROOT_SDA") + ), + MTK_PIN( + 72, "GPIO72", + MTK_EINT_FUNCTION(0, 72), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO72"), + MTK_FUNCTION(1, "B1_SCP_SCL2"), + MTK_FUNCTION(2, "B1_I3C_HCI_AO_SCL"), + MTK_FUNCTION(4, "I1_VADSP_JTAG0_TDI"), + MTK_FUNCTION(5, "I1_PCIE5_PAR_SCL"), + MTK_FUNCTION(6, "I1_SCP_JTAG0_TDI_VLP"), + MTK_FUNCTION(7, "O_OSROOT_UTX") + ), + MTK_PIN( + 73, "GPIO73", + MTK_EINT_FUNCTION(0, 73), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO73"), + MTK_FUNCTION(1, "B1_SCP_SDA2"), + MTK_FUNCTION(2, "B1_I3C_HCI_AO_SDA"), + MTK_FUNCTION(4, "O_VADSP_JTAG0_TDO"), + MTK_FUNCTION(5, "B1_PCIE5_PAR_SDA"), + MTK_FUNCTION(6, "O_SCP_JTAG0_TDO_VLP"), + MTK_FUNCTION(7, "I1_OSROOT_URX") + ), + MTK_PIN( + 74, "GPIO74", + MTK_EINT_FUNCTION(0, 74), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO74"), + MTK_FUNCTION(1, "I0_SCP_SPIS0_SCL"), + MTK_FUNCTION(2, "O_SPI2_CLK"), + MTK_FUNCTION(3, "O_SCP_SPIM0_CK"), + MTK_FUNCTION(4, "I1_SPM_JTAG_TCK_VLP"), + MTK_FUNCTION(5, "I1_SSPM_JTAG_TCK_VLP"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TCK_VLP"), + MTK_FUNCTION(7, "I0_OSROOT_TCK") + ), + MTK_PIN( + 75, "GPIO75", + MTK_EINT_FUNCTION(0, 75), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO75"), + MTK_FUNCTION(1, "B0_SCP_SPIS0_SIO0"), + MTK_FUNCTION(2, "B0_SPI2_MI"), + MTK_FUNCTION(3, "B0_SCP_SPIM0_SIO0"), + MTK_FUNCTION(4, "I1_SPM_JTAG_TDI_VLP"), + MTK_FUNCTION(5, "I1_SSPM_JTAG_TDI_VLP"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TDI_VLP"), + MTK_FUNCTION(7, "I0_OSROOT_TDI") + ), + MTK_PIN( + 76, "GPIO76", + MTK_EINT_FUNCTION(0, 76), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO76"), + MTK_FUNCTION(1, "B0_SCP_SPIS0_SIO1"), + MTK_FUNCTION(2, "B0_SPI2_MO"), + MTK_FUNCTION(3, "B0_SCP_SPIM0_SIO1"), + MTK_FUNCTION(4, "B1_SPM_JTAG_TDO_VLP"), + MTK_FUNCTION(5, "O_SSPM_JTAG_TDO_VLP"), + MTK_FUNCTION(6, "O_PBUD_CTRL_JTAG_TDO_VLP"), + MTK_FUNCTION(7, "O_OSROOT_TDO") + ), + MTK_PIN( + 77, "GPIO77", + MTK_EINT_FUNCTION(0, 77), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO77"), + MTK_FUNCTION(1, "B0_SCP_SPIS0_SIO2"), + MTK_FUNCTION(2, "B0_SPI2_WP"), + MTK_FUNCTION(3, "B0_SCP_SPIM0_SIO2"), + MTK_FUNCTION(4, "I1_SPM_JTAG_TMS_VLP"), + MTK_FUNCTION(5, "I1_SSPM_JTAG_TMS_VLP"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TMS_VLP"), + MTK_FUNCTION(7, "I0_OSROOT_TMS") + ), + MTK_PIN( + 78, "GPIO78", + MTK_EINT_FUNCTION(0, 78), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO78"), + MTK_FUNCTION(1, "B0_SCP_SPIS0_SIO3"), + MTK_FUNCTION(2, "B0_SPI2_HOLD"), + MTK_FUNCTION(3, "B0_SCP_SPIM0_SIO3"), + MTK_FUNCTION(4, "I0_SPM_JTAG_TRSTN_VLP"), + MTK_FUNCTION(5, "I0_SSPM_JTAG_TRSTN_VLP"), + MTK_FUNCTION(6, "I0_PBUD_CTRL_JTAG_TRSTN_VLP"), + MTK_FUNCTION(7, "I0_OSROOT_NTRST") + ), + MTK_PIN( + 79, "GPIO79", + MTK_EINT_FUNCTION(0, 79), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO79"), + MTK_FUNCTION(1, "I1_SCP_SPIS0_CS"), + MTK_FUNCTION(2, "O_SPI2_CSB"), + MTK_FUNCTION(3, "O_SCP_SPIM0_CS"), + MTK_FUNCTION(4, "I1_VADSP_JTAG0_TRSTN"), + MTK_FUNCTION(6, "I0_SCP_JTAG0_TRSTN_VLP") + ), + MTK_PIN( + 80, "GPIO80", + MTK_EINT_FUNCTION(0, 80), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO80"), + MTK_FUNCTION(1, "O_SPI0_CLK"), + MTK_FUNCTION(2, "B0_SPI0_OSROOT_CLK"), + MTK_FUNCTION(4, "I1_SSPM_JTAG_TCK_VLP"), + MTK_FUNCTION(5, "I0_VADSP_JTAG0_TCK"), + MTK_FUNCTION(6, "I1_SPM_JTAG_TCK_VLP"), + MTK_FUNCTION(7, "I1_SCP_JTAG0_TCK_VLP") + ), + MTK_PIN( + 81, "GPIO81", + MTK_EINT_FUNCTION(0, 81), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO81"), + MTK_FUNCTION(1, "B0_SPI0_MI"), + MTK_FUNCTION(2, "I0_SPI0_OSROOT_MI"), + MTK_FUNCTION(4, "I1_SSPM_JTAG_TDI_VLP"), + MTK_FUNCTION(5, "I1_VADSP_JTAG0_TDI"), + MTK_FUNCTION(6, "I1_SPM_JTAG_TDI_VLP"), + MTK_FUNCTION(7, "I1_SCP_JTAG0_TDI_VLP") + ), + MTK_PIN( + 82, "GPIO82", + MTK_EINT_FUNCTION(0, 82), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO82"), + MTK_FUNCTION(1, "B0_SPI0_MO"), + MTK_FUNCTION(2, "O_SPI0_OSROOT_MO"), + MTK_FUNCTION(4, "O_SSPM_JTAG_TDO_VLP"), + MTK_FUNCTION(5, "O_VADSP_JTAG0_TDO"), + MTK_FUNCTION(6, "B1_SPM_JTAG_TDO_VLP"), + MTK_FUNCTION(7, "O_SCP_JTAG0_TDO_VLP") + ), + MTK_PIN( + 83, "GPIO83", + MTK_EINT_FUNCTION(0, 83), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO83"), + MTK_FUNCTION(1, "O_SPI0_CSB0"), + MTK_FUNCTION(2, "O_SPI0_OSROOT_CSB"), + MTK_FUNCTION(4, "I1_SSPM_JTAG_TMS_VLP"), + MTK_FUNCTION(5, "I1_VADSP_JTAG0_TMS"), + MTK_FUNCTION(6, "I1_SPM_JTAG_TMS_VLP"), + MTK_FUNCTION(7, "B1_SCP_JTAG0_TMS_VLP") + ), + MTK_PIN( + 84, "GPIO84", + MTK_EINT_FUNCTION(0, 84), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO84"), + MTK_FUNCTION(1, "O_SPI0_CSB1"), + MTK_FUNCTION(4, "I0_SSPM_JTAG_TRSTN_VLP"), + MTK_FUNCTION(5, "I1_VADSP_JTAG0_TRSTN"), + MTK_FUNCTION(6, "I0_SPM_JTAG_TRSTN_VLP"), + MTK_FUNCTION(7, "I0_SCP_JTAG0_TRSTN_VLP") + ), + MTK_PIN( + 85, "GPIO85", + MTK_EINT_FUNCTION(0, 85), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO85"), + MTK_FUNCTION(1, "O_DISP_PWM"), + MTK_FUNCTION(4, "B1_U4CP_JTAG_TMS"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_1"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_1"), + MTK_FUNCTION(7, "O_DBG_MON_A21") + ), + MTK_PIN( + 86, "GPIO86", + MTK_EINT_FUNCTION(0, 86), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO86"), + MTK_FUNCTION(1, "O_DISP_BL_EN"), + MTK_FUNCTION(4, "I1_U4CP_JTAG_TDI"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_1"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_1"), + MTK_FUNCTION(7, "O_DBG_MON_A22") + ), + MTK_PIN( + 87, "GPIO87", + MTK_EINT_FUNCTION(0, 87), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO87"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N_1"), + MTK_FUNCTION(4, "O_U4CP_JTAG_TDO"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_2"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_2"), + MTK_FUNCTION(7, "O_DBG_MON_A23") + ), + MTK_PIN( + 88, "GPIO88", + MTK_EINT_FUNCTION(0, 88), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO88"), + MTK_FUNCTION(1, "B1_DISP_SCL1"), + MTK_FUNCTION(4, "I1_U4CP_JTAG_TCK"), + MTK_FUNCTION(5, "I1_EDP0_SCL"), + MTK_FUNCTION(6, "I1_HDMITX_DBG_I2C_SCL") + ), + MTK_PIN( + 89, "GPIO89", + MTK_EINT_FUNCTION(0, 89), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO89"), + MTK_FUNCTION(1, "B1_DISP_SDA1"), + MTK_FUNCTION(4, "I0_U4CP_JTAG_TRSTN"), + MTK_FUNCTION(5, "B1_EDP0_SDA"), + MTK_FUNCTION(6, "B1_HDMITX_DBG_I2C_SDA") + ), + MTK_PIN( + 90, "GPIO90", + MTK_EINT_FUNCTION(0, 90), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO90"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N2"), + MTK_FUNCTION(4, "O_CLKM0_C"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_2"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_2"), + MTK_FUNCTION(7, "O_DBG_MON_A24") + ), + MTK_PIN( + 91, "GPIO91", + MTK_EINT_FUNCTION(0, 91), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO91"), + MTK_FUNCTION(1, "I0_DPAUX_HPD_IN_4"), + MTK_FUNCTION(4, "O_CLKM1_C"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_3"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_3"), + MTK_FUNCTION(7, "O_DBG_MON_A25") + ), + MTK_PIN( + 92, "GPIO92", + MTK_EINT_FUNCTION(0, 92), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO92"), + MTK_FUNCTION(1, "B1_I2C_SCL5"), + MTK_FUNCTION(2, "O_TP_UTXD0_VLP"), + MTK_FUNCTION(3, "O_SSPM_UTXD_AO_VLP"), + MTK_FUNCTION(4, "O_TSFDC_FOUT"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_3"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_3"), + MTK_FUNCTION(7, "O_DBG_MON_A26") + ), + MTK_PIN( + 93, "GPIO93", + MTK_EINT_FUNCTION(0, 93), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO93"), + MTK_FUNCTION(1, "B1_I2C_SDA5"), + MTK_FUNCTION(2, "I1_TP_URXD0_VLP"), + MTK_FUNCTION(3, "I1_SSPM_URXD_AO_VLP"), + MTK_FUNCTION(4, "O_TSFDC_SDO"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_4"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_4"), + MTK_FUNCTION(7, "O_DBG_MON_A27") + ), + MTK_PIN( + 94, "GPIO94", + MTK_EINT_FUNCTION(0, 94), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO94"), + MTK_FUNCTION(1, "O_CMFLASH0"), + MTK_FUNCTION(4, "I0_TSFDC_26M"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_4"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_4"), + MTK_FUNCTION(7, "O_DBG_MON_A28") + ), + MTK_PIN( + 95, "GPIO95", + MTK_EINT_FUNCTION(0, 95), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO95"), + MTK_FUNCTION(1, "O_CMFLASH1"), + MTK_FUNCTION(4, "I0_TSFDC_SCF"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_5"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_5"), + MTK_FUNCTION(7, "O_DBG_MON_A29") + ), + MTK_PIN( + 96, "GPIO96", + MTK_EINT_FUNCTION(0, 96), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO96"), + MTK_FUNCTION(1, "O_CMVREF0"), + MTK_FUNCTION(2, "O_CMFLASH1"), + MTK_FUNCTION(4, "I0_TSFDC_SCK"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_5"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_5"), + MTK_FUNCTION(7, "O_DBG_MON_A30") + ), + MTK_PIN( + 97, "GPIO97", + MTK_EINT_FUNCTION(0, 97), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO97"), + MTK_FUNCTION(1, "O_CMVREF1"), + MTK_FUNCTION(2, "O_CMFLASH0"), + MTK_FUNCTION(4, "I0_TSFDC_SDI"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_6"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_6"), + MTK_FUNCTION(7, "O_U4CP_UTXD") + ), + MTK_PIN( + 98, "GPIO98", + MTK_EINT_FUNCTION(0, 98), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO98"), + MTK_FUNCTION(2, "O_CMFLASH2"), + MTK_FUNCTION(4, "I0_RG_TSFDC_LDO_EN"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_6"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_6"), + MTK_FUNCTION(7, "I1_U4CP_URXD") + ), + MTK_PIN( + 99, "GPIO99", + MTK_EINT_FUNCTION(0, 99), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO99"), + MTK_FUNCTION(1, "I0_MCU_M_PMIC_POC_I"), + MTK_FUNCTION(4, "I0_DA_TSFDC_LDO_MODE"), + MTK_FUNCTION(5, "I0_CLUSTER0_UDI_TDI_7"), + MTK_FUNCTION(6, "I0_CLUSTER1_UDI_TDI_7"), + MTK_FUNCTION(7, "O_U4CP_URTS") + ), + MTK_PIN( + 100, "GPIO100", + MTK_EINT_FUNCTION(0, 100), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO100"), + MTK_FUNCTION(1, "I0_MCU_B_PMIC_POC_I"), + MTK_FUNCTION(4, "I0_RG_TSFDC_LDO_REFSEL1"), + MTK_FUNCTION(5, "O_CLUSTER0_UDI_TDO_7"), + MTK_FUNCTION(6, "O_CLUSTER1_UDI_TDO_7"), + MTK_FUNCTION(7, "I1_U4CP_UCTS") + ), + MTK_PIN( + 101, "GPIO101", + MTK_EINT_FUNCTION(0, 101), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO101"), + MTK_FUNCTION(1, "O_CMFLASH2"), + MTK_FUNCTION(2, "O_CMVREF1"), + MTK_FUNCTION(3, "I1_UCTS0"), + MTK_FUNCTION(4, "I0_RG_TSFDC_LDO_REFSEL0"), + MTK_FUNCTION(5, "I1_U4CP_JTAG_TCK"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TCK_VCORE"), + MTK_FUNCTION(7, "O_CLKM0_A") + ), + MTK_PIN( + 102, "GPIO102", + MTK_EINT_FUNCTION(0, 102), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO102"), + MTK_FUNCTION(1, "O_CMFLASH3"), + MTK_FUNCTION(2, "O_CMVREF0"), + MTK_FUNCTION(3, "O_URTS0"), + MTK_FUNCTION(4, "O_TSFDC_BG_COMP"), + MTK_FUNCTION(5, "B1_U4CP_JTAG_TMS"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TMS_VCORE"), + MTK_FUNCTION(7, "O_CLKM1_A") + ), + MTK_PIN( + 103, "GPIO103", + MTK_EINT_FUNCTION(0, 103), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO103"), + MTK_FUNCTION(1, "O_CMVREF2"), + MTK_FUNCTION(2, "O_UTXD0"), + MTK_FUNCTION(4, "O_CLKM2_B"), + MTK_FUNCTION(5, "I1_U4CP_JTAG_TDI"), + MTK_FUNCTION(6, "I1_PBUD_CTRL_JTAG_TDI_VCORE"), + MTK_FUNCTION(7, "O_CLKM2_A") + ), + MTK_PIN( + 104, "GPIO104", + MTK_EINT_FUNCTION(0, 104), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO104"), + MTK_FUNCTION(1, "O_CMVREF3"), + MTK_FUNCTION(2, "I1_URXD0"), + MTK_FUNCTION(4, "O_CLKM3_B"), + MTK_FUNCTION(5, "O_U4CP_JTAG_TDO"), + MTK_FUNCTION(6, "O_PBUD_CTRL_JTAG_TDO_VCORE"), + MTK_FUNCTION(7, "O_CLKM3_A") + ), + MTK_PIN( + 105, "GPIO105", + MTK_EINT_FUNCTION(0, 105), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO105"), + MTK_FUNCTION(2, "O_CMFLASH3"), + MTK_FUNCTION(4, "O_CLKM0_B"), + MTK_FUNCTION(5, "I0_U4CP_JTAG_TRSTN"), + MTK_FUNCTION(6, "I0_PBUD_CTRL_JTAG_TRSTN_VCORE"), + MTK_FUNCTION(7, "O_PMSR_SMAP") + ), + MTK_PIN( + 106, "GPIO106", + MTK_EINT_FUNCTION(0, 106), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO106"), + MTK_FUNCTION(1, "B0_SPINOR_CK") + ), + MTK_PIN( + 107, "GPIO107", + MTK_EINT_FUNCTION(0, 107), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO107"), + MTK_FUNCTION(1, "B0_SPINOR_IO0") + ), + MTK_PIN( + 108, "GPIO108", + MTK_EINT_FUNCTION(0, 108), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO108"), + MTK_FUNCTION(1, "B0_SPINOR_IO1") + ), + MTK_PIN( + 109, "GPIO109", + MTK_EINT_FUNCTION(0, 109), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO109"), + MTK_FUNCTION(1, "B0_SPINOR_IO2") + ), + MTK_PIN( + 110, "GPIO110", + MTK_EINT_FUNCTION(0, 110), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO110"), + MTK_FUNCTION(1, "B0_SPINOR_IO3") + ), + MTK_PIN( + 111, "GPIO111", + MTK_EINT_FUNCTION(0, 111), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO111"), + MTK_FUNCTION(1, "B1_SPINOR_CS") + ), + MTK_PIN( + 112, "GPIO112", + MTK_EINT_FUNCTION(0, 112), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO112"), + MTK_FUNCTION(1, "O_SPI1_CLK"), + MTK_FUNCTION(5, "I1_HFRP_JTAG1_TCK") + ), + MTK_PIN( + 113, "GPIO113", + MTK_EINT_FUNCTION(0, 113), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO113"), + MTK_FUNCTION(1, "B0_SPI1_MI"), + MTK_FUNCTION(5, "I1_HFRP_JTAG1_TMS") + ), + MTK_PIN( + 114, "GPIO114", + MTK_EINT_FUNCTION(0, 114), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO114"), + MTK_FUNCTION(1, "B0_SPI1_MO"), + MTK_FUNCTION(5, "I1_HFRP_JTAG1_TDI") + ), + MTK_PIN( + 115, "GPIO115", + MTK_EINT_FUNCTION(0, 115), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO115"), + MTK_FUNCTION(1, "O_SPI1_CSB0"), + MTK_FUNCTION(5, "O_HFRP_JTAG1_TDO") + ), + MTK_PIN( + 116, "GPIO116", + MTK_EINT_FUNCTION(0, 116), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO116"), + MTK_FUNCTION(1, "O_SPI1_CSB1"), + MTK_FUNCTION(5, "I0_HFRP_JTAG1_TRSTN") + ), + MTK_PIN( + 117, "GPIO117", + MTK_EINT_FUNCTION(0, 117), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO117"), + MTK_FUNCTION(1, "B1_I2C_SCL1"), + MTK_FUNCTION(2, "B1_OSROOT_SCL"), + MTK_FUNCTION(4, "O_SPI_CS_S_MON0"), + MTK_FUNCTION(5, "I1_USB4_L_PAR_SCL"), + MTK_FUNCTION(6, "I1_CKM_SCL"), + MTK_FUNCTION(7, "I1_USB4_R_PAR_SCL") + ), + MTK_PIN( + 118, "GPIO118", + MTK_EINT_FUNCTION(0, 118), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO118"), + MTK_FUNCTION(1, "B1_I2C_SDA1"), + MTK_FUNCTION(2, "B1_OSROOT_SDA"), + MTK_FUNCTION(4, "O_SPI_SCL_S_MON0"), + MTK_FUNCTION(5, "B1_USB4_L_PAR_SDA"), + MTK_FUNCTION(6, "B1_CKM_SDA"), + MTK_FUNCTION(7, "B1_USB4_R_PAR_SDA") + ), + MTK_PIN( + 119, "GPIO119", + MTK_EINT_FUNCTION(0, 119), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO119"), + MTK_FUNCTION(1, "B1_I2C_SCL2"), + MTK_FUNCTION(4, "I1_HDMITX_DBG_I2C_SCL"), + MTK_FUNCTION(5, "O_CLUSTER0_MBISTREADEN_TRIGGER"), + MTK_FUNCTION(6, "O_CLUSTER1_MBISTREADEN_TRIGGER"), + MTK_FUNCTION(7, "O_PMSR_SMAP_MAX") + ), + MTK_PIN( + 120, "GPIO120", + MTK_EINT_FUNCTION(0, 120), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO120"), + MTK_FUNCTION(1, "B1_I2C_SDA2+J130_S133"), + MTK_FUNCTION(4, "B1_HDMITX_DBG_I2C_SDA"), + MTK_FUNCTION(5, "O_CLUSTER0_MBISTWRITEEN_TRIGGER"), + MTK_FUNCTION(6, "O_CLUSTER1_MBISTWRITEEN_TRIGGER"), + MTK_FUNCTION(7, "O_PMSR_SMAP_MAX_W") + ), + MTK_PIN( + 121, "GPIO121", + MTK_EINT_FUNCTION(0, 121), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO121"), + MTK_FUNCTION(1, "B1_I3C_SCL0"), + MTK_FUNCTION(2, "B1_I3C_HCI_0_AO_SCL"), + MTK_FUNCTION(4, "I1_PCIE5_PAR_SCL"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST0"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST0"), + MTK_FUNCTION(7, "I1_USB4_R_PAR_SCL") + ), + MTK_PIN( + 122, "GPIO122", + MTK_EINT_FUNCTION(0, 122), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO122"), + MTK_FUNCTION(1, "B1_I3C_SDA0"), + MTK_FUNCTION(2, "B1_I3C_HCI_0_AO_SDA"), + MTK_FUNCTION(4, "B1_PCIE5_PAR_SDA"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST1"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST1"), + MTK_FUNCTION(7, "B1_USB4_R_PAR_SDA") + ), + MTK_PIN( + 123, "GPIO123", + MTK_EINT_FUNCTION(0, 123), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO123"), + MTK_FUNCTION(1, "B1_I3C_SCL1"), + MTK_FUNCTION(2, "B1_I3C_HCI_1_AO_SCL"), + MTK_FUNCTION(4, "I1_PCIE4_USB3_PAR_SCL"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST2"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST2"), + MTK_FUNCTION(7, "O_VADSP_UTXD0") + ), + MTK_PIN( + 124, "GPIO124", + MTK_EINT_FUNCTION(0, 124), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO124"), + MTK_FUNCTION(1, "B1_I3C_SDA1"), + MTK_FUNCTION(2, "B1_I3C_HCI_1_AO_SDA"), + MTK_FUNCTION(4, "B1_PCIE4_USB3_PAR_SDA"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST3"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST3"), + MTK_FUNCTION(7, "I1_VADSP_URXD0") + ), + MTK_PIN( + 125, "GPIO125", + MTK_EINT_FUNCTION(0, 125), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO125"), + MTK_FUNCTION(1, "O_I2SIN_1_MCK"), + MTK_FUNCTION(4, "I0_TSFDC_SCK"), + MTK_FUNCTION(6, "O_CLKM2_C") + ), + MTK_PIN( + 126, "GPIO126", + MTK_EINT_FUNCTION(0, 126), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO126"), + MTK_FUNCTION(1, "O_SOUNDWIRE1_CK"), + MTK_FUNCTION(2, "O_I2SIN_1_BCK"), + MTK_FUNCTION(4, "O_CLUSTER0_AD_ILDO_DTEST4"), + MTK_FUNCTION(5, "O_CLUSTER1_AD_ILDO_DTEST4"), + MTK_FUNCTION(6, "I0_ADSP_JTAG0_TCK"), + MTK_FUNCTION(7, "I1_HFRP_JTAG0_TCK") + ), + MTK_PIN( + 127, "GPIO127", + MTK_EINT_FUNCTION(0, 127), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO127"), + MTK_FUNCTION(1, "B0_SOUNDWIRE1_D0"), + MTK_FUNCTION(2, "O_I2SOUT1_DO"), + MTK_FUNCTION(4, "O_CLUSTER0_AD_ILDO_DTEST5"), + MTK_FUNCTION(5, "O_CLUSTER1_AD_ILDO_DTEST5"), + MTK_FUNCTION(6, "I1_ADSP_JTAG0_TMS"), + MTK_FUNCTION(7, "B1_HFRP_JTAG0_TMS") + ), + MTK_PIN( + 128, "GPIO128", + MTK_EINT_FUNCTION(0, 128), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO128"), + MTK_FUNCTION(1, "B0_SOUNDWIRE1_D1"), + MTK_FUNCTION(2, "I0_I2SIN_1_DI"), + MTK_FUNCTION(4, "O_CLUSTER0_AD_ILDO_DTEST6"), + MTK_FUNCTION(5, "O_CLUSTER1_AD_ILDO_DTEST6"), + MTK_FUNCTION(6, "I1_ADSP_JTAG0_TDI"), + MTK_FUNCTION(7, "I1_HFRP_JTAG0_TDI") + ), + MTK_PIN( + 129, "GPIO129", + MTK_EINT_FUNCTION(0, 129), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO129"), + MTK_FUNCTION(1, "B0_SOUNDWIRE1_D2"), + MTK_FUNCTION(2, "O_I2SIN_1_LRCK"), + MTK_FUNCTION(4, "O_CLUSTER0_AD_ILDO_DTEST7"), + MTK_FUNCTION(5, "O_CLUSTER1_AD_ILDO_DTEST7"), + MTK_FUNCTION(6, "O_ADSP_JTAG0_TDO"), + MTK_FUNCTION(7, "O_HFRP_JTAG0_TDO") + ), + MTK_PIN( + 130, "GPIO130", + MTK_EINT_FUNCTION(0, 130), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO130"), + MTK_FUNCTION(1, "B0_I2SIN0_BCK"), + MTK_FUNCTION(3, "O_DISP_CLKM0"), + MTK_FUNCTION(4, "I0_TSFDC_26M"), + MTK_FUNCTION(5, "I0_RG_TSFDC_LDO_EN"), + MTK_FUNCTION(6, "O_CCU0_URTS"), + MTK_FUNCTION(7, "O_DBG_MON_B4") + ), + MTK_PIN( + 131, "GPIO131", + MTK_EINT_FUNCTION(0, 131), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO131"), + MTK_FUNCTION(1, "I0_I2SIN0_DI"), + MTK_FUNCTION(3, "O_DISP_CLKM1"), + MTK_FUNCTION(4, "O_TSFDC_FOUT"), + MTK_FUNCTION(5, "I0_DA_TSFDC_LDO_MODE"), + MTK_FUNCTION(6, "I1_CCU0_UCTS"), + MTK_FUNCTION(7, "O_DBG_MON_B5") + ), + MTK_PIN( + 132, "GPIO132", + MTK_EINT_FUNCTION(0, 132), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO132"), + MTK_FUNCTION(1, "O_I2SOUT0_DO"), + MTK_FUNCTION(3, "O_DISP_CLKM2"), + MTK_FUNCTION(4, "O_TSFDC_SDO"), + MTK_FUNCTION(5, "I0_RG_TSFDC_LDO_REFSEL1"), + MTK_FUNCTION(6, "O_CCU1_URTS"), + MTK_FUNCTION(7, "O_DBG_MON_B6") + ), + MTK_PIN( + 133, "GPIO133", + MTK_EINT_FUNCTION(0, 133), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO133"), + MTK_FUNCTION(1, "B0_I2SIN0_LRCK"), + MTK_FUNCTION(3, "O_DISP_CLKM3"), + MTK_FUNCTION(4, "I0_TSFDC_SCF"), + MTK_FUNCTION(5, "I0_RG_TSFDC_LDO_REFSEL0"), + MTK_FUNCTION(6, "I1_CCU1_UCTS"), + MTK_FUNCTION(7, "O_DBG_MON_B7") + ), + MTK_PIN( + 134, "GPIO134", + MTK_EINT_FUNCTION(0, 134), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO134"), + MTK_FUNCTION(1, "B1_SCP_SCL1"), + MTK_FUNCTION(2, "B1_VADSP_SCL0"), + MTK_FUNCTION(3, "B1_SROOT_SCL"), + MTK_FUNCTION(4, "O_SSPM_UTXD_AO_VLP"), + MTK_FUNCTION(5, "O_SPI_HID_IRQ_S_MON0"), + MTK_FUNCTION(6, "I1_ADSP_JTAG0_TRSTN"), + MTK_FUNCTION(7, "O_DBG_MON_B8") + ), + MTK_PIN( + 135, "GPIO135", + MTK_EINT_FUNCTION(0, 135), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO135"), + MTK_FUNCTION(1, "B1_SCP_SDA1"), + MTK_FUNCTION(2, "B1_VADSP_SDA0"), + MTK_FUNCTION(3, "B1_SROOT_SDA"), + MTK_FUNCTION(4, "I1_SSPM_URXD_AO_VLP"), + MTK_FUNCTION(7, "O_DBG_MON_B9") + ), + MTK_PIN( + 136, "GPIO136", + MTK_EINT_FUNCTION(0, 136), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO136"), + MTK_FUNCTION(1, "O_CMVREF2"), + MTK_FUNCTION(2, "B1_SCP_SCL3"), + MTK_FUNCTION(3, "B1_VADSP_SCL0"), + MTK_FUNCTION(4, "B1_SCP_SCL1"), + MTK_FUNCTION(5, "I1_MD32_8_RXD"), + MTK_FUNCTION(6, "I1_MD32_9_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_B10") + ), + MTK_PIN( + 137, "GPIO137", + MTK_EINT_FUNCTION(0, 137), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO137"), + MTK_FUNCTION(1, "O_CMVREF3"), + MTK_FUNCTION(2, "B1_SCP_SDA3"), + MTK_FUNCTION(3, "B1_VADSP_SDA0"), + MTK_FUNCTION(4, "B1_SCP_SDA1"), + MTK_FUNCTION(5, "O_MD32_8_GPIO0"), + MTK_FUNCTION(6, "O_MD32_9_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_B11") + ), + MTK_PIN( + 138, "GPIO138", + MTK_EINT_FUNCTION(0, 138), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO138"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N3"), + MTK_FUNCTION(2, "I0_DISP_LSPII"), + MTK_FUNCTION(4, "O_CLKM1_B"), + MTK_FUNCTION(5, "O_MD32_8_TXD"), + MTK_FUNCTION(6, "O_MD32_9_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_B12") + ), + MTK_PIN( + 139, "GPIO139", + MTK_EINT_FUNCTION(0, 139), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO139"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N4"), + MTK_FUNCTION(2, "O_DISP_HSYNC0"), + MTK_FUNCTION(3, "O_DISP_HSYNC1"), + MTK_FUNCTION(4, "O_CLKM0_A"), + MTK_FUNCTION(5, "O_CLKM0_B"), + MTK_FUNCTION(6, "O_CLKM0_C"), + MTK_FUNCTION(7, "O_DBG_MON_B13") + ), + MTK_PIN( + 140, "GPIO140", + MTK_EINT_FUNCTION(0, 140), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO140"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N5"), + MTK_FUNCTION(2, "O_DISP_VSYNC0"), + MTK_FUNCTION(3, "O_DISP_VSYNC1"), + MTK_FUNCTION(4, "O_CLKM1_A"), + MTK_FUNCTION(5, "O_CLKM1_B"), + MTK_FUNCTION(6, "O_CLKM1_C"), + MTK_FUNCTION(7, "O_DBG_MON_A14") + ), + MTK_PIN( + 141, "GPIO141", + MTK_EINT_FUNCTION(0, 141), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO141"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N6"), + MTK_FUNCTION(2, "O_DISP_HSYNC2"), + MTK_FUNCTION(3, "O_DISP_HSYNC3"), + MTK_FUNCTION(4, "O_CLKM2_A"), + MTK_FUNCTION(5, "O_CLKM2_B"), + MTK_FUNCTION(6, "O_CLKM2_C"), + MTK_FUNCTION(7, "O_DBG_MON_A15") + ), + MTK_PIN( + 142, "GPIO142", + MTK_EINT_FUNCTION(0, 142), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO142"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N7"), + MTK_FUNCTION(2, "O_DISP_VSYNC2"), + MTK_FUNCTION(3, "O_DISP_VSYNC3"), + MTK_FUNCTION(4, "O_CLKM3_A"), + MTK_FUNCTION(5, "O_CLKM3_B"), + MTK_FUNCTION(6, "O_CLKM3_C"), + MTK_FUNCTION(7, "O_DBG_MON_A16") + ), + MTK_PIN( + 143, "GPIO143", + MTK_EINT_FUNCTION(0, 143), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO143"), + MTK_FUNCTION(1, "I0_MCU_M_PMIC_POC_I"), + MTK_FUNCTION(2, "I0_JTCK_SEL1"), + MTK_FUNCTION(3, "O_JTAGAP_JTCK"), + MTK_FUNCTION(4, "I0_ADSP_JTAG1_TCK"), + MTK_FUNCTION(5, "I0_ADSP_JTAG0_TCK"), + MTK_FUNCTION(6, "I0_CLUSTER0_UDI_TCK"), + MTK_FUNCTION(7, "I0_CLUSTER1_UDI_TCK") + ), + MTK_PIN( + 144, "GPIO144", + MTK_EINT_FUNCTION(0, 144), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO144"), + MTK_FUNCTION(1, "I0_MCU_B_PMIC_POC_I"), + MTK_FUNCTION(2, "B1_JTMS_SEL1"), + MTK_FUNCTION(3, "O_JTAGAP_JTMS"), + MTK_FUNCTION(4, "I1_ADSP_JTAG1_TMS"), + MTK_FUNCTION(5, "I1_ADSP_JTAG0_TMS"), + MTK_FUNCTION(6, "I0_CLUSTER0_UDI_TMS"), + MTK_FUNCTION(7, "I0_CLUSTER1_UDI_TMS") + ), + MTK_PIN( + 145, "GPIO145", + MTK_EINT_FUNCTION(0, 145), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO145"), + MTK_FUNCTION(1, "I1_UCTS0"), + MTK_FUNCTION(2, "I1_JTDI_SEL1"), + MTK_FUNCTION(3, "O_JTAGAP_JTDI"), + MTK_FUNCTION(4, "I1_ADSP_JTAG1_TDI"), + MTK_FUNCTION(5, "I1_ADSP_JTAG0_TDI"), + MTK_FUNCTION(6, "I0_CLUSTER0_UDI_TDI_0"), + MTK_FUNCTION(7, "I0_CLUSTER1_UDI_TDI_0") + ), + MTK_PIN( + 146, "GPIO146", + MTK_EINT_FUNCTION(0, 146), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO146"), + MTK_FUNCTION(1, "O_URTS0"), + MTK_FUNCTION(2, "O_JTDO_SEL1"), + MTK_FUNCTION(3, "I0_JTAGAP_JTDO"), + MTK_FUNCTION(4, "O_ADSP_JTAG1_TDO"), + MTK_FUNCTION(5, "O_ADSP_JTAG0_TDO"), + MTK_FUNCTION(6, "O_CLUSTER0_UDI_TDO_0"), + MTK_FUNCTION(7, "O_CLUSTER1_UDI_TDO_0") + ), + MTK_PIN( + 147, "GPIO147", + MTK_EINT_FUNCTION(0, 147), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO147"), + MTK_FUNCTION(2, "I1_JTRSTn_SEL1"), + MTK_FUNCTION(3, "O_JTAGAP_JTRSTn"), + MTK_FUNCTION(4, "I1_ADSP_JTAG1_TRSTN"), + MTK_FUNCTION(5, "I1_ADSP_JTAG0_TRSTN"), + MTK_FUNCTION(6, "I0_CLUSTER0_UDI_NTRST"), + MTK_FUNCTION(7, "I0_CLUSTER1_UDI_NTRST") + ), + MTK_PIN( + 148, "GPIO148", + MTK_EINT_FUNCTION(0, 148), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO148"), + MTK_FUNCTION(1, "O_SRCLKENA0") + ), + MTK_PIN( + 149, "GPIO149", + MTK_EINT_FUNCTION(0, 149), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO149"), + MTK_FUNCTION(1, "O_SRCLKENA1") + ), + MTK_PIN( + 150, "GPIO150", + MTK_EINT_FUNCTION(0, 150), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO150"), + MTK_FUNCTION(2, "O_NVJTAG_SEL"), + MTK_FUNCTION(4, "I0_TSFDC_SDI"), + MTK_FUNCTION(5, "O_TSFDC_BG_COMP"), + MTK_FUNCTION(6, "O_CLKM3_C"), + MTK_FUNCTION(7, "I0_HFRP_JTAG0_TRSTN") + ), + MTK_PIN( + 151, "GPIO151", + MTK_EINT_FUNCTION(0, 151), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO151"), + MTK_FUNCTION(1, "I1_JTRSTn_SEL1"), + MTK_FUNCTION(2, "O_JTAGAP_JTRSTn"), + MTK_FUNCTION(6, "I0_HFRP_JTAG1_TRSTN"), + MTK_FUNCTION(7, "I1_ADSP_JTAG1_TRSTN") + ), + MTK_PIN( + 152, "GPIO152", + MTK_EINT_FUNCTION(0, 152), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO152"), + MTK_FUNCTION(1, "B1_JTMS_SEL1"), + MTK_FUNCTION(2, "O_JTAGAP_JTMS"), + MTK_FUNCTION(6, "I1_HFRP_JTAG1_TMS"), + MTK_FUNCTION(7, "I1_ADSP_JTAG1_TMS") + ), + MTK_PIN( + 153, "GPIO153", + MTK_EINT_FUNCTION(0, 153), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO153"), + MTK_FUNCTION(1, "O_JTDO_SEL1"), + MTK_FUNCTION(2, "I0_JTAGAP_JTDO"), + MTK_FUNCTION(6, "O_HFRP_JTAG1_TDO"), + MTK_FUNCTION(7, "O_ADSP_JTAG1_TDO") + ), + MTK_PIN( + 154, "GPIO154", + MTK_EINT_FUNCTION(0, 154), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO154"), + MTK_FUNCTION(1, "I0_JTCK_SEL1"), + MTK_FUNCTION(2, "O_JTAGAP_JTCK"), + MTK_FUNCTION(6, "I1_HFRP_JTAG1_TCK"), + MTK_FUNCTION(7, "I0_ADSP_JTAG1_TCK") + ), + MTK_PIN( + 155, "GPIO155", + MTK_EINT_FUNCTION(0, 155), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO155"), + MTK_FUNCTION(1, "I1_JTDI_SEL1"), + MTK_FUNCTION(2, "O_JTAGAP_JTDI"), + MTK_FUNCTION(6, "I1_HFRP_JTAG1_TDI"), + MTK_FUNCTION(7, "I1_ADSP_JTAG1_TDI") + ), + MTK_PIN( + 156, "GPIO156", + MTK_EINT_FUNCTION(0, 156), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO156"), + MTK_FUNCTION(1, "O_TP_UTXD0_VLP"), + MTK_FUNCTION(2, "O_UTXD3"), + MTK_FUNCTION(3, "O_VADSP_UTXD1"), + MTK_FUNCTION(4, "O_SPI_SIO0_S_MON0"), + MTK_FUNCTION(5, "O_ADSP_UTXD1"), + MTK_FUNCTION(6, "I0_SROOT_TCK"), + MTK_FUNCTION(7, "O_OSROOT_UTX") + ), + MTK_PIN( + 157, "GPIO157", + MTK_EINT_FUNCTION(0, 157), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO157"), + MTK_FUNCTION(1, "I1_TP_URXD0_VLP"), + MTK_FUNCTION(2, "I1_URXD3"), + MTK_FUNCTION(3, "I1_VADSP_URXD1"), + MTK_FUNCTION(4, "O_SPI_SIO1_S_MON0"), + MTK_FUNCTION(5, "I1_ADSP_URXD1"), + MTK_FUNCTION(6, "I0_SROOT_TDI"), + MTK_FUNCTION(7, "I1_OSROOT_URX") + ), + MTK_PIN( + 158, "GPIO158", + MTK_EINT_FUNCTION(0, 158), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO158"), + MTK_FUNCTION(1, "O_TP_URTS0_VLP"), + MTK_FUNCTION(2, "O_URTS3"), + MTK_FUNCTION(3, "O_VADSP_URTS1"), + MTK_FUNCTION(4, "O_SPI_SIO2_S_MON0"), + MTK_FUNCTION(5, "O_ADSP_URTX1"), + MTK_FUNCTION(6, "O_SROOT_TDO"), + MTK_FUNCTION(7, "O_PBUD_CTRL_UTXD_AO_VLP") + ), + MTK_PIN( + 159, "GPIO159", + MTK_EINT_FUNCTION(0, 159), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO159"), + MTK_FUNCTION(1, "I1_TP_UCTS0_VLP"), + MTK_FUNCTION(2, "I1_UCTS3"), + MTK_FUNCTION(3, "I1_VADSP_UCTS1"), + MTK_FUNCTION(4, "O_SPI_SIO3_S_MON0"), + MTK_FUNCTION(5, "I1_ADSP_UCTS1"), + MTK_FUNCTION(6, "I0_SROOT_TMS"), + MTK_FUNCTION(7, "I1_PBUD_CTRL_URXD_AO_VLP") + ), + MTK_PIN( + 160, "GPIO160", + MTK_EINT_FUNCTION(0, 160), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO160"), + MTK_FUNCTION(1, "O_UTXD1"), + MTK_FUNCTION(2, "O_ADSP_UTXD1"), + MTK_FUNCTION(3, "O_HFRP_UTXD1"), + MTK_FUNCTION(4, "O_CCU1_UTXD"), + MTK_FUNCTION(5, "O_PBUD_CTRL_UTXD_AO_VCORE"), + MTK_FUNCTION(6, "O_U4CP_UTXD"), + MTK_FUNCTION(7, "O_DBG_MON_B14") + ), + MTK_PIN( + 161, "GPIO161", + MTK_EINT_FUNCTION(0, 161), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO161"), + MTK_FUNCTION(1, "I1_URXD1"), + MTK_FUNCTION(2, "I1_ADSP_URXD1"), + MTK_FUNCTION(3, "I1_HFRP_URXD1"), + MTK_FUNCTION(4, "I1_CCU1_URXD"), + MTK_FUNCTION(5, "I1_PBUD_CTRL_URXD_AO_VCORE"), + MTK_FUNCTION(6, "I1_U4CP_URXD"), + MTK_FUNCTION(7, "O_DBG_MON_B15") + ), + MTK_PIN( + 162, "GPIO162", + MTK_EINT_FUNCTION(0, 162), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO162"), + MTK_FUNCTION(1, "O_URTS1"), + MTK_FUNCTION(2, "O_ADSP_URTX1"), + MTK_FUNCTION(3, "O_HFRP_URTS1"), + MTK_FUNCTION(4, "O_CCU0_UTXD"), + MTK_FUNCTION(5, "O_PBUD_CTRL_UTXD_AO_VCORE"), + MTK_FUNCTION(6, "O_U4CP_URTS"), + MTK_FUNCTION(7, "O_DBG_MON_B16") + ), + MTK_PIN( + 163, "GPIO163", + MTK_EINT_FUNCTION(0, 163), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO163"), + MTK_FUNCTION(1, "I1_UCTS1"), + MTK_FUNCTION(2, "I1_ADSP_UCTS1"), + MTK_FUNCTION(3, "I1_HFRP_UCTS1"), + MTK_FUNCTION(4, "I1_CCU0_URXD"), + MTK_FUNCTION(5, "I1_PBUD_CTRL_URXD_AO_VCORE"), + MTK_FUNCTION(6, "I1_U4CP_UCTS"), + MTK_FUNCTION(7, "O_DBG_MON_B17") + ), + MTK_PIN( + 164, "GPIO164", + MTK_EINT_FUNCTION(0, 164), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO164"), + MTK_FUNCTION(1, "O_HDMITX_DC_CTRL"), + MTK_FUNCTION(6, "I0_SROOT_NTRST"), + MTK_FUNCTION(7, "O_DBG_MON_B18") + ), + MTK_PIN( + 165, "GPIO165", + MTK_EINT_FUNCTION(0, 165), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO165"), + MTK_FUNCTION(1, "B1_DISP_SCL0"), + MTK_FUNCTION(4, "I1_EDP0_SCL"), + MTK_FUNCTION(5, "O_CCU0_URTS"), + MTK_FUNCTION(6, "O_CCU1_URTS"), + MTK_FUNCTION(7, "I1_MD32_6_RXD") + ), + MTK_PIN( + 166, "GPIO166", + MTK_EINT_FUNCTION(0, 166), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO166"), + MTK_FUNCTION(1, "B1_DISP_SDA0"), + MTK_FUNCTION(4, "B1_EDP0_SDA"), + MTK_FUNCTION(5, "I1_CCU0_UCTS"), + MTK_FUNCTION(6, "I1_CCU1_UCTS"), + MTK_FUNCTION(7, "O_MD32_6_GPIO0") + ), + MTK_PIN( + 167, "GPIO167", + MTK_EINT_FUNCTION(0, 167), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO167"), + MTK_FUNCTION(1, "B0_DISP_GPIO_N0"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST0"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST0"), + MTK_FUNCTION(7, "O_DBG_MON_B19") + ), + MTK_PIN( + 168, "GPIO168", + MTK_EINT_FUNCTION(0, 168), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO168"), + MTK_FUNCTION(1, "I0_DPAUX_HPD_IN_0"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST1"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST1"), + MTK_FUNCTION(7, "O_DBG_MON_B20") + ), + MTK_PIN( + 169, "GPIO169", + MTK_EINT_FUNCTION(0, 169), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO169"), + MTK_FUNCTION(1, "I0_DPAUX_HPD_IN_1"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST3"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST3"), + MTK_FUNCTION(7, "O_DBG_MON_B21") + ), + MTK_PIN( + 170, "GPIO170", + MTK_EINT_FUNCTION(0, 170), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO170"), + MTK_FUNCTION(1, "O_USB4_R_TCPC_RESET"), + MTK_FUNCTION(5, "O_CLUSTER0_AD_ILDO_DTEST2"), + MTK_FUNCTION(6, "O_CLUSTER1_AD_ILDO_DTEST2"), + MTK_FUNCTION(7, "O_DBG_MON_A17") + ), + MTK_PIN( + 171, "GPIO171", + MTK_EINT_FUNCTION(0, 171), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO171"), + MTK_FUNCTION(1, "O_PCIE_PERSTN"), + MTK_FUNCTION(4, "O_MD32_14_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_B22") + ), + MTK_PIN( + 172, "GPIO172", + MTK_EINT_FUNCTION(0, 172), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO172"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_0P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_0P"), + MTK_FUNCTION(5, "O_HFRP_UTXD1"), + MTK_FUNCTION(6, "O_CCU0_UTXD"), + MTK_FUNCTION(7, "O_DBG_MON_B23") + ), + MTK_PIN( + 173, "GPIO173", + MTK_EINT_FUNCTION(0, 173), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO173"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_1P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_1P"), + MTK_FUNCTION(5, "I1_HFRP_URXD1"), + MTK_FUNCTION(6, "I1_CCU0_URXD"), + MTK_FUNCTION(7, "O_DBG_MON_B24") + ), + MTK_PIN( + 174, "GPIO174", + MTK_EINT_FUNCTION(0, 174), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO174"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_2P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_2P"), + MTK_FUNCTION(4, "O_MD32PCM_UTXD_AO_VLP"), + MTK_FUNCTION(5, "O_HFRP_URTS1"), + MTK_FUNCTION(6, "O_CCU1_UTXD"), + MTK_FUNCTION(7, "O_DBG_MON_B25") + ), + MTK_PIN( + 175, "GPIO175", + MTK_EINT_FUNCTION(0, 175), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO175"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_3P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_3P"), + MTK_FUNCTION(4, "I1_MD32PCM_URXD_AO_VLP"), + MTK_FUNCTION(5, "I1_HFRP_UCTS1"), + MTK_FUNCTION(6, "I1_CCU1_URXD"), + MTK_FUNCTION(7, "O_DBG_MON_B26") + ), + MTK_PIN( + 176, "GPIO176", + MTK_EINT_FUNCTION(0, 176), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO176"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_4P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_4P"), + MTK_FUNCTION(4, "O_MD32_14_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_B27") + ), + MTK_PIN( + 177, "GPIO177", + MTK_EINT_FUNCTION(0, 177), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO177"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_5P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_5P"), + MTK_FUNCTION(4, "I1_MD32_14_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_B28") + ), + MTK_PIN( + 178, "GPIO178", + MTK_EINT_FUNCTION(0, 178), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO178"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_6P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_6P"), + MTK_FUNCTION(4, "O_MD32_15_TXD"), + MTK_FUNCTION(7, "O_DBG_MON_B29") + ), + MTK_PIN( + 179, "GPIO179", + MTK_EINT_FUNCTION(0, 179), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO179"), + MTK_FUNCTION(1, "B1_PCIE_CLKREQN_7P"), + MTK_FUNCTION(2, "I1_PCIE_PRSNT_7P"), + MTK_FUNCTION(4, "I1_MD32_15_RXD"), + MTK_FUNCTION(7, "O_DBG_MON_B30") + ), + MTK_PIN( + 180, "GPIO180", + MTK_EINT_FUNCTION(0, 180), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO180"), + MTK_FUNCTION(1, "I1_PCIE_WAKEN"), + MTK_FUNCTION(4, "O_MD32_15_GPIO0"), + MTK_FUNCTION(7, "O_DBG_MON_B31") + ), + MTK_PIN( + 181, "GPIO181", + MTK_EINT_FUNCTION(0, 181), + DRV_GRP4, + MTK_FUNCTION(0, "B_GPIO181"), + MTK_FUNCTION(1, "O_GPU_PWRGOOD") + ), +}; + +static struct mtk_eint_pin eint_pins_mt8901[] = { + MTK_EINT_PIN(0, 0, 16, 0), + MTK_EINT_PIN(1, 0, 17, 0), + MTK_EINT_PIN(2, 0, 18, 0), + MTK_EINT_PIN(3, 0, 19, 0), + MTK_EINT_PIN(4, 0, 20, 0), + MTK_EINT_PIN(5, 0, 21, 0), + MTK_EINT_PIN(6, 0, 22, 0), + MTK_EINT_PIN(7, 0, 23, 0), + MTK_EINT_PIN(8, 0, 24, 0), + MTK_EINT_PIN(9, 0, 25, 0), + MTK_EINT_PIN(10, 0, 26, 0), + MTK_EINT_PIN(11, 0, 27, 0), + MTK_EINT_PIN(12, INVALID_BASE, 0, 0), + MTK_EINT_PIN(13, INVALID_BASE, 0, 0), + MTK_EINT_PIN(14, 0, 28, 0), + MTK_EINT_PIN(15, 0, 29, 0), + MTK_EINT_PIN(16, 0, 30, 0), + MTK_EINT_PIN(17, INVALID_BASE, 0, 0), + MTK_EINT_PIN(18, INVALID_BASE, 0, 0), + MTK_EINT_PIN(19, 0, 31, 0), + MTK_EINT_PIN(20, 0, 0, 1), + MTK_EINT_PIN(21, 0, 1, 1), + MTK_EINT_PIN(22, INVALID_BASE, 0, 0), + MTK_EINT_PIN(23, 0, 2, 1), + MTK_EINT_PIN(24, 0, 3, 1), + MTK_EINT_PIN(25, 0, 4, 1), + MTK_EINT_PIN(26, 0, 5, 1), + MTK_EINT_PIN(27, 0, 6, 1), + MTK_EINT_PIN(28, 0, 7, 1), + MTK_EINT_PIN(29, 0, 8, 1), + MTK_EINT_PIN(30, 0, 9, 1), + MTK_EINT_PIN(31, 0, 10, 1), + MTK_EINT_PIN(32, 1, 0, 1), + MTK_EINT_PIN(33, 0, 32, 0), + MTK_EINT_PIN(34, 0, 33, 0), + MTK_EINT_PIN(35, 0, 34, 0), + MTK_EINT_PIN(36, 1, 1, 1), + MTK_EINT_PIN(37, 0, 11, 1), + MTK_EINT_PIN(38, 0, 12, 1), + MTK_EINT_PIN(39, 0, 35, 0), + MTK_EINT_PIN(40, 0, 36, 0), + MTK_EINT_PIN(41, 0, 37, 0), + MTK_EINT_PIN(42, 0, 13, 1), + MTK_EINT_PIN(43, 0, 14, 1), + MTK_EINT_PIN(44, 0, 38, 0), + MTK_EINT_PIN(45, 0, 39, 0), + MTK_EINT_PIN(46, 0, 40, 0), + MTK_EINT_PIN(47, 0, 15, 1), + MTK_EINT_PIN(48, 0, 41, 0), + MTK_EINT_PIN(49, 0, 42, 0), + MTK_EINT_PIN(50, 0, 43, 0), + MTK_EINT_PIN(51, 0, 44, 0), + MTK_EINT_PIN(52, 0, 45, 0), + MTK_EINT_PIN(53, 0, 46, 0), + MTK_EINT_PIN(54, 2, 13, 0), + MTK_EINT_PIN(55, 2, 14, 0), + MTK_EINT_PIN(56, INVALID_BASE, 0, 0), + MTK_EINT_PIN(57, INVALID_BASE, 0, 0), + MTK_EINT_PIN(58, INVALID_BASE, 0, 0), + MTK_EINT_PIN(59, INVALID_BASE, 0, 0), + MTK_EINT_PIN(60, 1, 3, 0), + MTK_EINT_PIN(61, 1, 4, 0), + MTK_EINT_PIN(62, 1, 5, 0), + MTK_EINT_PIN(63, 1, 6, 0), + MTK_EINT_PIN(64, 1, 7, 0), + MTK_EINT_PIN(65, 1, 8, 0), + MTK_EINT_PIN(66, 2, 15, 0), + MTK_EINT_PIN(67, 2, 16, 0), + MTK_EINT_PIN(68, 1, 2, 1), + MTK_EINT_PIN(69, INVALID_BASE, 0, 0), + MTK_EINT_PIN(70, 2, 17, 0), + MTK_EINT_PIN(71, 2, 18, 0), + MTK_EINT_PIN(72, 2, 19, 0), + MTK_EINT_PIN(73, 2, 20, 0), + MTK_EINT_PIN(74, 2, 21, 0), + MTK_EINT_PIN(75, 2, 22, 0), + MTK_EINT_PIN(76, 2, 23, 0), + MTK_EINT_PIN(77, 2, 24, 0), + MTK_EINT_PIN(78, 2, 25, 0), + MTK_EINT_PIN(79, 2, 26, 0), + MTK_EINT_PIN(80, 2, 27, 0), + MTK_EINT_PIN(81, 2, 28, 0), + MTK_EINT_PIN(82, 2, 29, 1), + MTK_EINT_PIN(83, 2, 30, 1), + MTK_EINT_PIN(84, 2, 31, 1), + MTK_EINT_PIN(85, 2, 32, 1), + MTK_EINT_PIN(86, 2, 33, 0), + MTK_EINT_PIN(87, 2, 34, 0), + MTK_EINT_PIN(88, 2, 35, 0), + MTK_EINT_PIN(89, 2, 36, 0), + MTK_EINT_PIN(90, 2, 37, 0), + MTK_EINT_PIN(91, 2, 38, 0), + MTK_EINT_PIN(92, 2, 39, 0), + MTK_EINT_PIN(93, 2, 40, 0), + MTK_EINT_PIN(94, 2, 0, 1), + MTK_EINT_PIN(95, 2, 1, 1), + MTK_EINT_PIN(96, 2, 2, 1), + MTK_EINT_PIN(97, 2, 3, 1), + MTK_EINT_PIN(98, 2, 41, 0), + MTK_EINT_PIN(99, 2, 4, 1), + MTK_EINT_PIN(100, 2, 5, 1), + MTK_EINT_PIN(101, 2, 6, 1), + MTK_EINT_PIN(102, 2, 7, 1), + MTK_EINT_PIN(103, 2, 8, 1), + MTK_EINT_PIN(104, 2, 9, 1), + MTK_EINT_PIN(105, 2, 10, 1), + MTK_EINT_PIN(106, 0, 47, 0), + MTK_EINT_PIN(107, 0, 48, 0), + MTK_EINT_PIN(108, 0, 49, 0), + MTK_EINT_PIN(109, 0, 50, 0), + MTK_EINT_PIN(110, 0, 51, 0), + MTK_EINT_PIN(111, 0, 52, 0), + MTK_EINT_PIN(112, 2, 42, 0), + MTK_EINT_PIN(113, 2, 43, 0), + MTK_EINT_PIN(114, 2, 44, 0), + MTK_EINT_PIN(115, 2, 45, 0), + MTK_EINT_PIN(116, 2, 46, 0), + MTK_EINT_PIN(117, 2, 47, 0), + MTK_EINT_PIN(118, 2, 48, 0), + MTK_EINT_PIN(119, 2, 49, 0), + MTK_EINT_PIN(120, 2, 50, 0), + MTK_EINT_PIN(121, 2, 51, 0), + MTK_EINT_PIN(122, 2, 52, 0), + MTK_EINT_PIN(123, 2, 53, 0), + MTK_EINT_PIN(124, 2, 54, 0), + MTK_EINT_PIN(125, 2, 55, 0), + MTK_EINT_PIN(126, 2, 56, 0), + MTK_EINT_PIN(127, 2, 57, 0), + MTK_EINT_PIN(128, 2, 58, 0), + MTK_EINT_PIN(129, 2, 59, 0), + MTK_EINT_PIN(130, 2, 60, 0), + MTK_EINT_PIN(131, 2, 61, 0), + MTK_EINT_PIN(132, 2, 62, 0), + MTK_EINT_PIN(133, 2, 63, 0), + MTK_EINT_PIN(134, 2, 64, 0), + MTK_EINT_PIN(135, 2, 65, 0), + MTK_EINT_PIN(136, 2, 66, 0), + MTK_EINT_PIN(137, 2, 67, 0), + MTK_EINT_PIN(138, 2, 11, 1), + MTK_EINT_PIN(139, 2, 12, 1), + MTK_EINT_PIN(140, 2, 68, 0), + MTK_EINT_PIN(141, 2, 69, 0), + MTK_EINT_PIN(142, 2, 70, 0), + MTK_EINT_PIN(143, 2, 71, 0), + MTK_EINT_PIN(144, 2, 72, 0), + MTK_EINT_PIN(145, 2, 73, 0), + MTK_EINT_PIN(146, 2, 74, 0), + MTK_EINT_PIN(147, 2, 75, 0), + MTK_EINT_PIN(148, INVALID_BASE, 0, 0), + MTK_EINT_PIN(149, INVALID_BASE, 0, 0), + MTK_EINT_PIN(150, 2, 76, 0), + MTK_EINT_PIN(151, 2, 77, 0), + MTK_EINT_PIN(152, 2, 78, 0), + MTK_EINT_PIN(153, 2, 79, 0), + MTK_EINT_PIN(154, 2, 80, 0), + MTK_EINT_PIN(155, 2, 81, 0), + MTK_EINT_PIN(156, 2, 82, 0), + MTK_EINT_PIN(157, 2, 83, 0), + MTK_EINT_PIN(158, 2, 84, 0), + MTK_EINT_PIN(159, 2, 85, 0), + MTK_EINT_PIN(160, 2, 86, 0), + MTK_EINT_PIN(161, 2, 87, 0), + MTK_EINT_PIN(162, 2, 88, 0), + MTK_EINT_PIN(163, 2, 89, 0), + MTK_EINT_PIN(164, 2, 90, 0), + MTK_EINT_PIN(165, 2, 91, 0), + MTK_EINT_PIN(166, 2, 92, 0), + MTK_EINT_PIN(167, 2, 93, 0), + MTK_EINT_PIN(168, 2, 94, 0), + MTK_EINT_PIN(169, 2, 95, 0), + MTK_EINT_PIN(170, 2, 96, 0), + MTK_EINT_PIN(171, 2, 97, 0), + MTK_EINT_PIN(172, 2, 98, 0), + MTK_EINT_PIN(173, 2, 99, 0), + MTK_EINT_PIN(174, 2, 100, 0), + MTK_EINT_PIN(175, 2, 101, 0), + MTK_EINT_PIN(176, 2, 102, 0), + MTK_EINT_PIN(177, 2, 103, 0), + MTK_EINT_PIN(178, 2, 104, 0), + MTK_EINT_PIN(179, 2, 105, 0), + MTK_EINT_PIN(180, 2, 106, 0), + MTK_EINT_PIN(181, 3, 0, 0), + MTK_EINT_PIN(182, 3, 1, 0), + MTK_EINT_PIN(183, 3, 2, 0), + MTK_EINT_PIN(184, 3, 3, 0), + MTK_EINT_PIN(185, 3, 4, 0), + MTK_EINT_PIN(186, 3, 5, 0), + MTK_EINT_PIN(187, 3, 6, 0), + MTK_EINT_PIN(188, 3, 7, 0), + MTK_EINT_PIN(189, 3, 8, 0), + MTK_EINT_PIN(190, 3, 9, 0), + MTK_EINT_PIN(191, 3, 10, 0), + MTK_EINT_PIN(192, 3, 11, 0), + MTK_EINT_PIN(193, 3, 12, 0), + MTK_EINT_PIN(194, 3, 13, 0), + MTK_EINT_PIN(195, 3, 14, 0), + MTK_EINT_PIN(196, 3, 15, 0), + MTK_EINT_PIN(197, 3, 16, 0), + MTK_EINT_PIN(198, 3, 17, 0), + MTK_EINT_PIN(199, 3, 18, 0), + MTK_EINT_PIN(200, 3, 19, 0), + MTK_EINT_PIN(201, 3, 20, 0), + MTK_EINT_PIN(202, 3, 21, 0), + MTK_EINT_PIN(203, 3, 22, 0), + MTK_EINT_PIN(204, 3, 23, 0), + MTK_EINT_PIN(205, 3, 24, 0), + MTK_EINT_PIN(206, 3, 25, 0), + MTK_EINT_PIN(207, 3, 26, 0), + MTK_EINT_PIN(208, 3, 27, 0), +}; + +#endif /* __PINCTRL__MTK_MT8901_H */ From 5e493213d7b243fb9d4be4b7349b3ec4307f9944 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Tue, 22 Jul 2025 14:08:34 +0000 Subject: [PATCH 056/464] UBUNTU: [Config] nvidia: Update annotations to enable CONFIG_PINCTRL_MT8901 BugLink: https://bugs.launchpad.net/bugs/2117784 Signed-off-by: Abhishek Sahu Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Acked-by: nvmochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 0bd85d02570017780577755049df70668266ad78) (cherry picked from commit 0bd85d025700 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 94b2089f6ff5607413bf98b24c5bd62a70d87290 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 8404149f7f758..d6b264f2030fe 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -162,6 +162,9 @@ CONFIG_NVIDIA_FFA_EC note<'LP: #2114230'> CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> +CONFIG_PINCTRL_MT8901 policy<{'arm64': 'y'}> +CONFIG_PINCTRL_MT8901 note<'LP: #2117784'> + CONFIG_R8127 policy<{'amd64': 'n', 'arm64': 'm'}> CONFIG_R8127 note<'LP: #2109730'> From 19212ddee45e62f58eb9fd2e2b46dc491c8097aa Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Tue, 22 Jul 2025 13:45:33 +0000 Subject: [PATCH 057/464] NVIDIA: SAUCE: Fix FFH data response length BugLink: https://bugs.launchpad.net/bugs/2118357 commit d0038ee1df2b ("NVIDIA: SAUCE: Add support for EC secure service communication") added nvidia_ffh_handler() function. While copying the data back into ACPI FFH packet, it uses the request length. The response data can be larger than request length. The response length can't be fetched in the linux FFH handler function. We can copy all the bytes from ffa_data.data. The ACPI AML code will only use the required number bytes from this. Normally we don't need response length to be known. The ACPI table are not using that. It is parsing response data directly. In the latest revision of spec, the length field itself has been removed https://github.com/OpenDevicePartnership/documentation/blob/b23acb09f7cf03a5c3167509533f396d547e6291/guide_book/src/specs/ec_interface/secure-ec-services-overview.md#operation-region-definition For DIGITS GB10, it is using older revision of spec and the launch is planned with older revision of spec. When we move to latest revision, then we need to copy all data bytes for both request and response. The info->length is corresponding to FFH buffer length in ACPI table. Following is the code in ACPI table Name (_HID, "MSFT000C") // _HID: Hardware ID OperationRegion (AFFH, FFixedHW, 0x04, 0x90) info->length will be 0x90 (144) bytes. ffa_packet->length in the older revision is valid data bytes (https://github.com/OpenDevicePartnership/documentation/blob/45ad9b30be0f40e229deed2fef7a60d0b0b591f5/bookshelf/Shelf%204%20Specifications/EC%20Interface/src/secure-ec-services-overview.md) struct nvidia_ec_ffa_packet *ffa_packet = (struct nvidia_ec_ffa_packet *)value; This value buffer length should be info->length. We are taking minimum of sizeof(ffa_data.data) = 112 and (info->length = 144) - (offsetof(struct nvidia_ec_ffa_packet, rawdata) = 18) = 126, so ffh_copy_len will be 112 for the current DIGITS ACPI implementation. In the latest revision, this length mismatch is also fixed. Raw data will start at offset 32, so there both will come as 112. Fixes: d0038ee1df2b ("NVIDIA: SAUCE: Add support for EC secure service communication") Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 141bd5652ecc noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 0477ce56148b65bbaa946135a4791fe7d89b7e8d noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index 78068f1237b57..d9e8b7fdda30c 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -530,6 +530,7 @@ static int nvidia_ffh_handler(struct acpi_ffh_info *info, acpi_integer *value, v struct nvidia_ec_ffa_packet *ffa_packet = (struct nvidia_ec_ffa_packet *)value; struct nvidia_ec_ffa_device *cur, *ec_dev = NULL; int ret; + unsigned int ffh_copy_len; uuid_t uuid; /* Only offset 4 is supported */ @@ -592,9 +593,16 @@ static int nvidia_ffh_handler(struct acpi_ffh_info *info, acpi_integer *value, v /* Set the status as success */ ffa_packet->status = 0; - /* Copy the ACPI FFA data back into ACPI FFH packet */ - memcpy(ffa_packet->rawdata, ffa_data.data, ffa_packet->length); + /* + * Copy the ACPI FFA data back into ACPI FFH packet. + * + * ACPI FFH packet raw data length can't be fetched here, so copy + * all bytes from ffa_data.data + */ + ffh_copy_len = min(sizeof(ffa_data.data), + info->length - offsetof(struct nvidia_ec_ffa_packet, rawdata)); + memcpy(ffa_packet->rawdata, ffa_data.data, ffh_copy_len); return 0; } From 0112d2c3f922823f59ed3c8189d774e15546b7f8 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Mon, 4 Dec 2023 22:38:25 +0000 Subject: [PATCH 058/464] NVIDIA: SAUCE: arm64: configs: Build NVGRACE_GPU_VFIO_PCI as LKM BugLink: https://bugs.launchpad.net/bugs/2119656 Signed-off-by: Nicolin Chen Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 9433fd4ac5f0d1a63feed968a8b16261fcd7d808 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit a1bdf88a26695bd4a255bdad7c263fe9d6d2ab58 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 60f9b04529b6 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 13f98cc45ca813feca952ff9c61bd3129338a115 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- arch/arm64/configs/defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index b67d5b1fc45b0..4ea813d63f65f 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1957,3 +1957,4 @@ CONFIG_CORESIGHT_STM=m CONFIG_CORESIGHT_CPU_DEBUG=m CONFIG_CORESIGHT_CTI=m CONFIG_MEMTEST=y +CONFIG_NVGRACE_GPU_VFIO_PCI=m From e44408f94ca77c8698599e8f354feb13305d174f Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Tue, 7 Nov 2023 04:07:47 -0800 Subject: [PATCH 059/464] NVIDIA: SAUCE: arm64: configs: Enable IOMMUFD and VFIO_DEVICE_CDEV BugLink: https://bugs.launchpad.net/bugs/2119656 Signed-off-by: Nicolin Chen Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 3eff6df2e892f9ea4a564ac27ae8fc005ad054be https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 6c6e8936e0f502f9225ad38070f90d259266ffb8 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit a6a3ccc38385 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit ecc87b5069293ce3a37fa15c4d870cb7b4abaa8e noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- arch/arm64/configs/defconfig | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 4ea813d63f65f..678b993e77828 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1958,3 +1958,8 @@ CONFIG_CORESIGHT_CPU_DEBUG=m CONFIG_CORESIGHT_CTI=m CONFIG_MEMTEST=y CONFIG_NVGRACE_GPU_VFIO_PCI=m +CONFIG_VFIO_DEVICE_CDEV=y +CONFIG_FAULT_INJECTION=y +CONFIG_IOMMUFD_DRIVER=y +CONFIG_IOMMUFD=y +CONFIG_IOMMUFD_TEST=y From 22443b20b4cbcfb7bc2c147c85cf7ad3a3b1da85 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Thu, 29 Aug 2024 08:15:40 +0000 Subject: [PATCH 060/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Introduce module to manage EGM BugLink: https://bugs.launchpad.net/bugs/2119656 The Extended GPU Memory (EGM) feature enables the GPU access to the system memory across sockets and nodes. In this mode, the physical memory can be allocated for GPU usage from anywhere in a multi-node system. The feature is being extended to virtualization. EGM when enabled in the virtualization stack, the host memory is partitioned into 2: One partition for the Host OS usage, and a second EGM region. The EGM region essentially becomes the system memory of the VM. The following figure shows the memory map in the virtualization environment. |---- Sysmem ----| |--- GPU mem ---| VM Memory Map | | | | | | | | |------ EGM -----|--Host Mem----| |--- GPU mem ---| Host Memory Map The EGM region is not available to the host memory for its usage as it is not added to the kernel. Its base HPA and the length is communicated through the DSDT entries. A linear mapping between the VM IPA and system HPA is a requirement for EGM support. The EGM region is thus assigned to a VM by mapping the QEMU VMA to a linearly increasing HPA of the EGM region using remap_pfn_range(). Introduce a new nvgrace-egm helper module to nvgrace-gpu to manage the EGM/VM region for the VM. nvgrace-egm module handles the following: 1. Fetch the EGM memory properties (base HPA, length, proximity domain). 2. Create a char device that can be used as memory-backend-file by Qemu for the VM and implement file operations. The char device is /dev/egmX, where X is the PXM node ID of the EGM being mapped fetched in 1. 3. Zero the EGM memory on first device open(). 4. Map the QEMU VMA to the EGM region using remap_pfn_range. 5. Cleaning up state and destroying the chardev on device unbind. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 892ac2417c614969ff215ad75c0249af6073ffb9 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 3a1b8196060afeaec7b37a1300706d59642e8212 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 8807f4b90409 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (backported from commit fa304984adc4ac07d2fd10d68c1cea99e2b7c12f noble:linux-nvidia-6.17) [jacobmartin: adjust patch context to align with upstream commit e5f19b619fa0 ("vfio/nvgrace-gpu: register device memory for poison handling"), as opposed to the original SAUCE version of the same patch.] Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/Kconfig | 11 ++ drivers/vfio/pci/nvgrace-gpu/Makefile | 3 + drivers/vfio/pci/nvgrace-gpu/egm.c | 235 ++++++++++++++++++++++++++ drivers/vfio/pci/nvgrace-gpu/egm.h | 12 ++ drivers/vfio/pci/nvgrace-gpu/main.c | 31 +++- 5 files changed, 291 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm.c create mode 100644 drivers/vfio/pci/nvgrace-gpu/egm.h diff --git a/drivers/vfio/pci/nvgrace-gpu/Kconfig b/drivers/vfio/pci/nvgrace-gpu/Kconfig index a7f624b37e410..d5773bbd22f5e 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Kconfig +++ b/drivers/vfio/pci/nvgrace-gpu/Kconfig @@ -1,8 +1,19 @@ # SPDX-License-Identifier: GPL-2.0-only +config NVGRACE_EGM + tristate "EGM driver for NVIDIA Grace Hopper and Blackwell Superchip" + depends on ARM64 || (COMPILE_TEST && 64BIT) + help + Extended GPU Memory (EGM) support for the GPU in the NVIDIA Grace + based chips required to avail the CPU memory as additional + cross-node/cross-socket memory for GPU using KVM/qemu. + + If you don't know what to do here, say N. + config NVGRACE_GPU_VFIO_PCI tristate "VFIO support for the GPU in the NVIDIA Grace Hopper Superchip" depends on ARM64 || (COMPILE_TEST && 64BIT) select VFIO_PCI_CORE + select NVGRACE_EGM help VFIO support for the GPU in the NVIDIA Grace Hopper Superchip is required to assign the GPU device to userspace using KVM/qemu/etc. diff --git a/drivers/vfio/pci/nvgrace-gpu/Makefile b/drivers/vfio/pci/nvgrace-gpu/Makefile index 3ca8c187897a9..c99b04a94e770 100644 --- a/drivers/vfio/pci/nvgrace-gpu/Makefile +++ b/drivers/vfio/pci/nvgrace-gpu/Makefile @@ -1,3 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu-vfio-pci.o nvgrace-gpu-vfio-pci-y := main.o + +obj-$(CONFIG_NVGRACE_EGM) += nvgrace-egm.o +nvgrace-egm-y := egm.o diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c new file mode 100644 index 0000000000000..f3c22a9dfecb9 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include "egm.h" + +#define MAX_EGM_NODES 256 + +struct egm_region { + struct list_head list; + int egmpxm; + atomic_t open_count; + phys_addr_t egmphys; + size_t egmlength; + struct device device; + struct cdev cdev; +}; + +static dev_t dev; +static struct class *class; +static struct list_head egm_list; + +static int nvgrace_egm_open(struct inode *inode, struct file *file) +{ + void *memaddr; + struct egm_region *region = container_of(inode->i_cdev, + struct egm_region, cdev); + + if (!region) + return -EINVAL; + + if (atomic_inc_return(®ion->open_count) > 1) + return 0; + + memaddr = memremap(region->egmphys, region->egmlength, MEMREMAP_WB); + if (!memaddr) { + atomic_dec(®ion->open_count); + return -EINVAL; + } + + memset((u8 *)memaddr, 0, region->egmlength); + memunmap(memaddr); + file->private_data = region; + + return 0; +} + +static int nvgrace_egm_release(struct inode *inode, struct file *file) +{ + struct egm_region *region = container_of(inode->i_cdev, + struct egm_region, cdev); + + if (!region) + return -EINVAL; + + if (atomic_dec_and_test(®ion->open_count)) + file->private_data = NULL; + + return 0; +} + +static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) +{ + int ret = 0; + struct egm_region *region = file->private_data; + + if (!region) + return -EINVAL; + + ret = remap_pfn_range(vma, vma->vm_start, + PHYS_PFN(region->egmphys), + (vma->vm_end - vma->vm_start), + vma->vm_page_prot); + return ret; +} + +static const struct file_operations file_ops = { + .owner = THIS_MODULE, + .open = nvgrace_egm_open, + .release = nvgrace_egm_release, + .mmap = nvgrace_egm_mmap, +}; + +static int setup_egm_chardev(struct egm_region *region) +{ + int ret = 0; + + device_initialize(®ion->device); + + /* + * Use the proximity domain number as the device minor + * number. So the EGM corresponding to node X would be + * /dev/egmX. + */ + region->device.devt = MKDEV(MAJOR(dev), region->egmpxm); + region->device.class = class; + cdev_init(®ion->cdev, &file_ops); + region->cdev.owner = THIS_MODULE; + + ret = dev_set_name(®ion->device, "egm%d", region->egmpxm); + if (ret) + return ret; + + ret = cdev_device_add(®ion->cdev, ®ion->device); + + return ret; +} + +static int +nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, + u64 *pegmlength, u64 *pegmpxm) +{ + int ret; + + /* + * The memory information is present in the system ACPI tables as DSD + * properties nvidia,egm-base-pa and nvidia,egmm-size. + */ + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-size", + pegmlength); + if (ret) + return ret; + + if (*pegmlength > type_max(size_t)) + return -EOVERFLOW; + + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", + pegmphys); + if (ret) + return ret; + + if (*pegmphys > type_max(phys_addr_t)) + return -EOVERFLOW; + + ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); + + if (*pegmpxm > type_max(phys_addr_t)) + return -EOVERFLOW; + + return ret; +} + +int register_egm_node(struct pci_dev *pdev) +{ + struct egm_region *region = NULL; + u64 egmphys, egmlength, egmpxm; + int ret; + + ret = nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength, &egmpxm); + if (ret) + return ret; + + list_for_each_entry(region, &egm_list, list) { + if (region->egmphys == egmphys) + return 0; + } + + region = kvzalloc(sizeof(*region), GFP_KERNEL); + region->egmphys = egmphys; + region->egmlength = egmlength; + region->egmpxm = egmpxm; + + atomic_set(®ion->open_count, 0); + + list_add_tail(®ion->list, &egm_list); + + setup_egm_chardev(region); + + return 0; +} +EXPORT_SYMBOL_GPL(register_egm_node); + +static void destroy_egm_chardev(struct egm_region *region) +{ + cdev_device_del(®ion->cdev, ®ion->device); +} + +void unregister_egm_node(int egm_node) +{ + struct egm_region *region, *temp_region; + + list_for_each_entry_safe(region, temp_region, &egm_list, list) { + if (egm_node == region->egmpxm) { + destroy_egm_chardev(region); + list_del(®ion->list); + } + } +} +EXPORT_SYMBOL_GPL(unregister_egm_node); + +static char *egm_devnode(const struct device *device, umode_t *mode) +{ + if (mode) + *mode = 0600; + + return NULL; +} + +static int __init nvgrace_egm_init(void) +{ + int ret; + + ret = alloc_chrdev_region(&dev, + 0, MAX_EGM_NODES, "egm"); + if (ret < 0) + return ret; + + class = class_create("egm"); + if (IS_ERR(class)) { + unregister_chrdev_region(dev, MAX_EGM_NODES); + return PTR_ERR(class); + } + + class->devnode = egm_devnode; + + INIT_LIST_HEAD(&egm_list); + + return 0; +} + +static void __exit nvgrace_egm_cleanup(void) +{ + class_destroy(class); + unregister_chrdev_region(dev, MAX_EGM_NODES); +} + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ankit Agrawal "); +MODULE_DESCRIPTION("NVGRACE EGM - Helper module of NVGRACE GPU to support Extended GPU Memory"); + +module_init(nvgrace_egm_init); +module_exit(nvgrace_egm_cleanup); diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.h b/drivers/vfio/pci/nvgrace-gpu/egm.h new file mode 100644 index 0000000000000..28cc59e04a0b0 --- /dev/null +++ b/drivers/vfio/pci/nvgrace-gpu/egm.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef NVGRACE_EGM_H +#define NVGRACE_EGM_H + +int register_egm_node(struct pci_dev *pdev); +void unregister_egm_node(int egm_node); + +#endif /* NVGRACE_EGM_H */ diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index fa056b69f899a..fe2ab87a1aed8 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -10,6 +10,7 @@ #include #include #include +#include "egm.h" /* * The device memory usable to the workloads running in the VM is cached @@ -64,8 +65,11 @@ struct nvgrace_gpu_pci_core_device { bool has_mig_hw_bug; /* GPU has just been reset */ bool reset_done; + int egm_node; }; +static bool egm_enabled; + static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev) { struct nvgrace_gpu_pci_core_device *nvdev = @@ -1012,6 +1016,13 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev, return ret; } +static int +nvgrace_gpu_has_egm_property(struct pci_dev *pdev, u64 *pegmpxm) +{ + return device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", + pegmpxm); +} + static int nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev, struct nvgrace_gpu_pci_core_device *nvdev, @@ -1181,6 +1192,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, const struct vfio_device_ops *ops = &nvgrace_gpu_pci_core_ops; struct nvgrace_gpu_pci_core_device *nvdev; u64 memphys, memlength; + u64 egmpxm; int ret; ret = nvgrace_gpu_probe_check_device_ready(pdev); @@ -1188,9 +1200,14 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, return ret; ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength); - if (!ret) + if (!ret) { ops = &nvgrace_gpu_pci_ops; + ret = nvgrace_gpu_has_egm_property(pdev, &egmpxm); + if (!ret) + egm_enabled = true; + } + nvdev = vfio_alloc_device(nvgrace_gpu_pci_core_device, core_device.vdev, &pdev->dev, ops); if (IS_ERR(nvdev)) @@ -1210,6 +1227,12 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, if (ret) goto out_put_vdev; nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops; + + if (egm_enabled) { + register_egm_node(pdev); + nvdev->egm_node = egmpxm; + } + } else { nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_core_ops; } @@ -1228,6 +1251,12 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, static void nvgrace_gpu_remove(struct pci_dev *pdev) { struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); + struct nvgrace_gpu_pci_core_device *nvdev = + container_of(core_device, struct nvgrace_gpu_pci_core_device, + core_device); + + if (egm_enabled) + unregister_egm_node(nvdev->egm_node); vfio_pci_core_unregister_device(core_device); vfio_put_device(&core_device->vdev); From ef799b1bb5d9698bead528e03109fce11f7bbd47 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Thu, 29 Aug 2024 08:15:41 +0000 Subject: [PATCH 061/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Handle pages with ECC errors on the EGM BugLink: https://bugs.launchpad.net/bugs/2119656 It is possible for some system memory pages on the EGM to have uncorrectable ECC errors. A list of pages known with such errors (referred as retired pages) are maintained by the Host UEFI. The Host UEFI populates such list in a reserved region. It communicates the SPA of this region through a ACPI DSDT property. nvgrace-egm module is responsible to store the list of retired page offsets to be made available for usermode processes. The module: 1. Get the reserved memory region SPA and maps to it to fetch the list of bad pages. 2. Calculate the retired page offsets in the EGM and stores it. 3. Expose an ioctl to allow querying of the offsets. The ioctl is called by usermode apps such as QEMU to get the retired page offsets. The usermode apps are expected to take appropriate action to communicate the list to the VM. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit be54641b9f3e52a471e9d02aa12723bfb47a7060 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit c4cb1930d93ac2c7bb4f0cfba0a9e3e4ff180879 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 6b0a6d6644e3 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 64942451faf3b08f9f3014d70c871fe0ffcf041f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 126 +++++++++++++++++++++++++++++ include/uapi/linux/egm.h | 26 ++++++ 2 files changed, 152 insertions(+) create mode 100644 include/uapi/linux/egm.h diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index f3c22a9dfecb9..8c9ff6313e9f4 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -4,6 +4,8 @@ */ #include +#include +#include #include "egm.h" #define MAX_EGM_NODES 256 @@ -16,6 +18,12 @@ struct egm_region { size_t egmlength; struct device device; struct cdev cdev; + DECLARE_HASHTABLE(htbl, 0x10); +}; + +struct h_node { + unsigned long mem_offset; + struct hlist_node node; }; static dev_t dev; @@ -76,11 +84,80 @@ static int nvgrace_egm_mmap(struct file *file, struct vm_area_struct *vma) return ret; } +static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + unsigned long minsz = offsetofend(struct egm_bad_pages_list, count); + struct egm_bad_pages_list info; + void __user *uarg = (void __user *)arg; + struct egm_region *region = file->private_data; + + if (copy_from_user(&info, uarg, minsz)) + return -EFAULT; + + if (info.argsz < minsz) + return -EINVAL; + + if (!region) + return -EINVAL; + + switch (cmd) { + case EGM_BAD_PAGES_LIST: + int ret; + unsigned long bad_page_struct_size = sizeof(struct egm_bad_pages_info); + struct egm_bad_pages_info tmp; + struct h_node *cur_page; + struct hlist_node *tmp_node; + unsigned long bkt; + int count = 0, index = 0; + + hash_for_each_safe(region->htbl, bkt, tmp_node, cur_page, node) + count++; + + if (info.argsz < (minsz + count * bad_page_struct_size)) { + info.argsz = minsz + count * bad_page_struct_size; + info.count = 0; + goto done; + } else { + hash_for_each_safe(region->htbl, bkt, tmp_node, cur_page, node) { + /* + * This check fails if there was an ECC error + * after the usermode app read the count of + * bad pages through this ioctl. + */ + if (minsz + index * bad_page_struct_size >= info.argsz) { + info.argsz = minsz + index * bad_page_struct_size; + info.count = index; + goto done; + } + + tmp.offset = cur_page->mem_offset; + tmp.size = PAGE_SIZE; + + ret = copy_to_user(uarg + minsz + + index * bad_page_struct_size, + &tmp, bad_page_struct_size); + if (ret) + return ret; + index++; + } + + info.count = index; + } + break; + default: + return -EINVAL; + } + +done: + return copy_to_user(uarg, &info, minsz) ? -EFAULT : 0; +} + static const struct file_operations file_ops = { .owner = THIS_MODULE, .open = nvgrace_egm_open, .release = nvgrace_egm_release, .mmap = nvgrace_egm_mmap, + .unlocked_ioctl = nvgrace_egm_ioctl, }; static int setup_egm_chardev(struct egm_region *region) @@ -143,6 +220,45 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, return ret; } +static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, + struct egm_region *region) +{ + u64 retiredpagesphys, count; + void *memaddr; + int index; + + if (device_property_read_u64(&pdev->dev, + "nvidia,egm-retired-pages-data-base", + &retiredpagesphys)) + return; + + memaddr = memremap(retiredpagesphys, PAGE_SIZE, MEMREMAP_WB); + if (!memaddr) + return; + + count = *(u64 *)memaddr; + + hash_init(region->htbl); + + for (index = 0; index < count; index++) { + struct h_node *retired_page; + + /* + * Since the EGM is linearly mapped, the offset in the + * carveout is the same offset in the VM system memory. + * + * Calculate the offset to communicate to the usermode + * apps. + */ + retired_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + retired_page->mem_offset = *((u64 *)memaddr + index + 1) - + region->egmphys; + hash_add(region->htbl, &retired_page->node, retired_page->mem_offset); + } + + memunmap(memaddr); +} + int register_egm_node(struct pci_dev *pdev) { struct egm_region *region = NULL; @@ -165,6 +281,8 @@ int register_egm_node(struct pci_dev *pdev) atomic_set(®ion->open_count, 0); + nvgrace_egm_fetch_bad_pages(pdev, region); + list_add_tail(®ion->list, &egm_list); setup_egm_chardev(region); @@ -181,9 +299,17 @@ static void destroy_egm_chardev(struct egm_region *region) void unregister_egm_node(int egm_node) { struct egm_region *region, *temp_region; + struct h_node *cur_page; + unsigned long bkt; + struct hlist_node *temp_node; list_for_each_entry_safe(region, temp_region, &egm_list, list) { if (egm_node == region->egmpxm) { + hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { + hash_del(&cur_page->node); + vfree(cur_page); + } + destroy_egm_chardev(region); list_del(®ion->list); } diff --git a/include/uapi/linux/egm.h b/include/uapi/linux/egm.h new file mode 100644 index 0000000000000..8a808e45c2052 --- /dev/null +++ b/include/uapi/linux/egm.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef _UAPIEGM_H +#define _UAPIEGM_H + +#define EGM_TYPE ('E') + +struct egm_bad_pages_info { + __aligned_u64 offset; + __aligned_u64 size; +}; + +struct egm_bad_pages_list { + __u32 argsz; + /* out */ + __u32 count; + /* out */ + struct egm_bad_pages_info bad_pages[]; +}; + +#define EGM_BAD_PAGES_LIST _IO(EGM_TYPE, 100) + +#endif /* _UAPIEGM_H */ From e4834678537a3afc58fa9f8a40b7499c77042f2e Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 29 Aug 2024 18:49:03 -0700 Subject: [PATCH 062/464] NVIDIA: SAUCE: arm64: configs: Build CONFIG_NVGRACE_EGM as LKM BugLink: https://bugs.launchpad.net/bugs/2119656 Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 5bb23c179220ec77ac9fb2ed610618ce1a902bd4 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 7d2ea5531c96fb9acd5704b6bec20aa29ca1fd39 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 077c8340953f noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit f5a03d00aed1fb83a77abd723dee5a8a79392f3b noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- arch/arm64/configs/defconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 678b993e77828..269e811132f9b 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1958,6 +1958,7 @@ CONFIG_CORESIGHT_CPU_DEBUG=m CONFIG_CORESIGHT_CTI=m CONFIG_MEMTEST=y CONFIG_NVGRACE_GPU_VFIO_PCI=m +CONFIG_NVGRACE_EGM=m CONFIG_VFIO_DEVICE_CDEV=y CONFIG_FAULT_INJECTION=y CONFIG_IOMMUFD_DRIVER=y From 076d06cc1501a7777f70c3e13b6a9b3d844cf4ca Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Sun, 13 Oct 2024 04:53:38 +0000 Subject: [PATCH 063/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Move the egm header file to include BugLink: https://bugs.launchpad.net/bugs/2119656 nvgrace-egm exposes the API register_egm_node & unregister_egm_node to manage EGM (Extended GPU Memory) present on the system. To allow out-of-tree driver such as nvidia-vgpu-vfio make use of them, move the declaration to a new nvgrace-egm.h in include. Signed-off-by: Ankit Agrawal Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit bed340f2023f22192893e9121834ee3ce252edd1 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit a9616639ce81799f5b3133c47c25f8d875728f4f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 020c46c87e7a noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 739457a5ff1b0b30f904d2c1ab28eebcd670a628 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 +- drivers/vfio/pci/nvgrace-gpu/main.c | 2 +- .../pci/nvgrace-gpu/egm.h => include/linux/nvgrace-egm.h | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) rename drivers/vfio/pci/nvgrace-gpu/egm.h => include/linux/nvgrace-egm.h (55%) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 8c9ff6313e9f4..598a1d07d00b7 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -6,7 +6,7 @@ #include #include #include -#include "egm.h" +#include #define MAX_EGM_NODES 256 diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index fe2ab87a1aed8..cb3eeaa3f560f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -10,7 +10,7 @@ #include #include #include -#include "egm.h" +#include /* * The device memory usable to the workloads running in the VM is cached diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.h b/include/linux/nvgrace-egm.h similarity index 55% rename from drivers/vfio/pci/nvgrace-gpu/egm.h rename to include/linux/nvgrace-egm.h index 28cc59e04a0b0..48add892aa5bf 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.h +++ b/include/linux/nvgrace-egm.h @@ -1,12 +1,12 @@ -// SPDX-License-Identifier: GPL-2.0-only +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ /* * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved */ -#ifndef NVGRACE_EGM_H -#define NVGRACE_EGM_H +#ifndef _NVGRACE_EGM_H +#define _NVGRACE_EGM_H int register_egm_node(struct pci_dev *pdev); void unregister_egm_node(int egm_node); -#endif /* NVGRACE_EGM_H */ +#endif /* _NVGRACE_EGM_H */ From 5fe45983b6e4ccb22792fc5edc6786f24c42fed1 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:06:57 -0800 Subject: [PATCH 064/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Free region memory during unregistration BugLink: https://bugs.launchpad.net/bugs/2119656 Free the kmalloc'd region when the EGM is unregistered. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit fc592b9b4f8b455205abd2b2395671a831bb942e https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit f24760ccecb8c5517fca6791082ab89cf94b9f9f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 374b166787e0 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 8f781d07d28638ab3c31c46ed79c0fdc9711a9c4 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 598a1d07d00b7..06f41049275bd 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -312,6 +312,7 @@ void unregister_egm_node(int egm_node) destroy_egm_chardev(region); list_del(®ion->list); + kfree(region); } } } From 34c9c4577528857007705884bba5304e5c38d497 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:38:11 -0800 Subject: [PATCH 065/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Move region hash initialization BugLink: https://bugs.launchpad.net/bugs/2119656 Move region hash initiaization alongside the other region initialization statements to avoid situations where the hash table was not properly initialized. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 8021c1d2b1c73015102bc69eda0029114989dd1f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit e1264a62e8841fd5332f7f02a921242ff1b51dfa https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 0f8a09890f67 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 22f790add5eb4d1b17cd3e056d8f783af25b3f72 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 06f41049275bd..621d046084a18 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -238,8 +238,6 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, count = *(u64 *)memaddr; - hash_init(region->htbl); - for (index = 0; index < count; index++) { struct h_node *retired_page; @@ -279,6 +277,7 @@ int register_egm_node(struct pci_dev *pdev) region->egmlength = egmlength; region->egmpxm = egmpxm; + hash_init(region->htbl); atomic_set(®ion->open_count, 0); nvgrace_egm_fetch_bad_pages(pdev, region); From 47213f7ccab38a160cc0e6802d85ae9c3c834bb0 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:48:47 -0800 Subject: [PATCH 066/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Handle and convey EGM registration errors BugLink: https://bugs.launchpad.net/bugs/2119656 Update error handling within EGM regiration routine to catch and return errors to the caller. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit a57210c88c1c3693a24684c967c0858d75cabd32 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit a706ff8c445abed002e0b9493dfc9c664b1ffd57 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit edc0ac06e8e9 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e7a177e2e55f20b7508e3017be2a5259a91e38d5 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 621d046084a18..140f0f10f2c3e 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -273,6 +273,9 @@ int register_egm_node(struct pci_dev *pdev) } region = kvzalloc(sizeof(*region), GFP_KERNEL); + if (!region) + return -ENOMEM; + region->egmphys = egmphys; region->egmlength = egmlength; region->egmpxm = egmpxm; @@ -282,11 +285,16 @@ int register_egm_node(struct pci_dev *pdev) nvgrace_egm_fetch_bad_pages(pdev, region); - list_add_tail(®ion->list, &egm_list); + ret = setup_egm_chardev(region); + if (ret) + goto err; - setup_egm_chardev(region); + list_add_tail(®ion->list, &egm_list); return 0; +err: + kfree(region); + return ret; } EXPORT_SYMBOL_GPL(register_egm_node); From 4fc45a83c53c41ae307e172c21ce20dc09f4bede Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 15:55:58 -0800 Subject: [PATCH 067/464] NVIDIA: SAUCE: vfio/nvgrace-gpu: Handle EGM registration failure BugLink: https://bugs.launchpad.net/bugs/2119656 Detect and handle a failure from the EGM registration service. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit f18eee3bbdea77a9b525c0665d7ebe1992bb00b2 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 8371b68c33cc03a7ea6dfd7bdfc0fe9d47ec64fb https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit be5ae8ffa6ef noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 2dd59038740de38b4b2400930b05a49fd6051446 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/main.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index cb3eeaa3f560f..5b26235e32cad 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1229,7 +1229,10 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, nvdev->core_device.pci_ops = &nvgrace_gpu_pci_dev_ops; if (egm_enabled) { - register_egm_node(pdev); + ret = register_egm_node(pdev); + if (ret) + goto out_put_vdev; + nvdev->egm_node = egmpxm; } @@ -1239,10 +1242,13 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, ret = vfio_pci_core_register_device(&nvdev->core_device); if (ret) - goto out_put_vdev; + goto out_egm_unreg; return ret; +out_egm_unreg: + if (egm_enabled) + unregister_egm_node(nvdev->egm_node); out_put_vdev: vfio_put_device(&nvdev->core_device.vdev); return ret; From 64ea21ded75dfe19e2966b94b5a5fdfa01ccb009 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 16:07:26 -0800 Subject: [PATCH 068/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Address sparse errors BugLink: https://bugs.launchpad.net/bugs/2119656 Fix minor syntax errors from sparse. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit bbb64e63a0b5e8c8eeec52b1e901745ba64b96d3 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit fe7819421a04be2b2405376da3550beb03986b6c https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit b19296004d0d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c1d3f2196850eef4c2eb77326d9475252e7a4087 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 140f0f10f2c3e..33ed9a1f1a03f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -102,6 +102,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long switch (cmd) { case EGM_BAD_PAGES_LIST: + { int ret; unsigned long bad_page_struct_size = sizeof(struct egm_bad_pages_info); struct egm_bad_pages_info tmp; @@ -144,6 +145,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long info.count = index; } break; + } default: return -EINVAL; } From 3c6b192d9d2f6df1bc1135856730e007a300b7e0 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 20:09:38 -0800 Subject: [PATCH 069/464] NVIDIA: SAUCE: vfio/nvgrace-gpu: Address smatch errors BugLink: https://bugs.launchpad.net/bugs/2119656 Use the correct macro and types for overflow checking. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit afa8f63898cf65cf0d9cf3209ef486daa11a42c7 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit d110330e4b93894a6234b6c4036f8422883fab90 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit a6c050804fa4 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 8ec6da274a04af027be4532917932380e09d7d97 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 5b26235e32cad..d812cbbf84512 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -995,7 +995,7 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev, if (ret) return ret; - if (*pmemphys > type_max(phys_addr_t)) + if (overflows_type(*pmemphys, phys_addr_t)) return -EOVERFLOW; ret = device_property_read_u64(&pdev->dev, "nvidia,gpu-mem-size", @@ -1003,7 +1003,7 @@ nvgrace_gpu_fetch_memory_property(struct pci_dev *pdev, if (ret) return ret; - if (*pmemlength > type_max(size_t)) + if (overflows_type(*pmemlength, size_t)) return -EOVERFLOW; /* From 57199082d230f973cabda40cbc082ec1f979f631 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Fri, 22 Nov 2024 15:48:10 -0800 Subject: [PATCH 070/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Ensure ACPI value reads are successful BugLink: https://bugs.launchpad.net/bugs/2119656 Ensure ACPI table reads are successful prior to using the value. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit b2947b075de6c887660cc8bc23ab5f0b6e7bfd17 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 92583550c3b22d1d00bfc6f59f3fc943cbd3a29e https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 2c5b472932c1 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 3f72f24ead3380c02907ee5d145f1b7a8e053d96 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 33ed9a1f1a03f..9388bdefe09aa 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -215,11 +215,13 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", pegmpxm); + if (ret) + return ret; if (*pegmpxm > type_max(phys_addr_t)) return -EOVERFLOW; - return ret; + return 0; } static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, From 185001b7de7a2d45e107522eb2eb0ccee2b5170a Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 14 Nov 2024 08:12:22 -0800 Subject: [PATCH 071/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Avoid invalid retired pages base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2119656 Some environments may provide a "nvidia,egm-retired-pages-data-base” but fail to populate it with a base address, leaving it NULL. Mapping this invalid value results in a synchronous exception when the region is first touched. Detect a NULL value, generate a warning to draw attention to the firmware bug, and return without mapping. INFO: th500_ras_intr_handler: External Abort reason=1 syndrome=0x92000410 flags=0x1 [ 82.104493] Internal error: synchronous external abort: 0000000096000410 [#1] SMP [ 82.114898] Modules linked in: nvgrace_gpu_vfio_pci(E) nvgrace_egm(E) [ 82.257218] CPU: 0 PID: 10 Comm: kworker/0:1 Tainted: G OE 6.8.12+ #5 [ 82.265135] Hardware name: NVIDIA GH200 P5042, BIOS 24103110 20241031 [ 82.271720] Workqueue: events work_for_cpu_fn [ 82.276180] pstate: 03400009 (nzcv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) [ 82.283298] pc : register_egm_node+0x2cc/0x440 [nvgrace_egm] [ 82.289087] lr : register_egm_node+0x2c4/0x440 [nvgrace_egm] [ 82.294872] sp : ffff8000802ebc30 [ 82.298254] x29: ffff8000802ebc60 x28: 00000000000000ff x27: 0000000000000000 [ 82.305550] x26: ffff000087a320c8 x25: ffff0000a5700000 x24: ffff000087a32000 [ 82.312846] x23: ffffa77cd758e368 x22: 0000000000000000 x21: ffffa77cd758c640 [ 82.320141] x20: ffffa77cd758e170 x19: ffff800081e7d000 x18: ffff800080293038 [ 82.327437] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 [ 82.334732] x14: 0000000000000000 x13: 65203a65646f6e5f x12: 0000000000000000 [ 82.342027] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000 [ 82.349322] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000 [ 82.356618] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 82.363913] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff800081e7d000 [ 82.371210] Call trace: [ 82.373705] register_egm_node+0x2cc/0x440 [nvgrace_egm] [ 82.379135] nvgrace_gpu_probe+0x2ac/0x528 [nvgrace_gpu_vfio_pci] [ 82.385366] local_pci_probe+0x4c/0xe0 [ 82.389198] work_for_cpu_fn+0x28/0x58 [ 82.393026] process_one_work+0x168/0x3f0 [ 82.397123] worker_thread+0x360/0x480 [ 82.400952] kthread+0x11c/0x128 [ 82.404248] ret_from_fork+0x10/0x20 [ 82.407906] Code: d2820001 940002b3 aa0003f3 b4fffac0 (f9400017) [ 82.414134] ---[ end trace 0000000000000000 ]--- Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 7ba29302925c6f2e1b9825d06f7468acc175ab85 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 349fb1c23faef926f3bdbc479b088c7b6b66853f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 6e9c94a06e83 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c5992d5bdc0402f5c327a5fe65791cb5df278de9 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9388bdefe09aa..2ffae71f7f458 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -236,6 +236,10 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, &retiredpagesphys)) return; + /* Catch firmware bug and avoid a crash */ + if (WARN_ON_ONCE(retiredpagesphys == 0)) + return; + memaddr = memremap(retiredpagesphys, PAGE_SIZE, MEMREMAP_WB); if (!memaddr) return; From e9adfd3c2f08cec582e647a645262316b9e80a13 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 23 Jan 2025 12:07:12 -0800 Subject: [PATCH 072/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Update EGM unregistration API BugLink: https://bugs.launchpad.net/bugs/2119656 In an effort to simplify the programming model, use a symmetrical model for the the EGM regsiration APIs. This avoids the caller needing to keep a cookie or even have knowlege of if EGM is supported. Update the EGM unregisration API to use the PCI device as its parameter. Signed-off-by: Matthew R. Ochs (cherry picked from commit d8903ecbf6ae94cbf67b8492996021cd2488033c https://github.com/nvmochs/NV-Kernels/tree/vegm_01232025) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 5839fc506349c858a90a19e713c46fce025b2ec6 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit f6fb40e917fd noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 0e607bc7591e5b9e161ae98dfe9e42c02864b402 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 10 ++++++++-- drivers/vfio/pci/nvgrace-gpu/main.c | 4 ++-- include/linux/nvgrace-egm.h | 2 +- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 2ffae71f7f458..1545ac695ad77 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -311,15 +311,21 @@ static void destroy_egm_chardev(struct egm_region *region) cdev_device_del(®ion->cdev, ®ion->device); } -void unregister_egm_node(int egm_node) +void unregister_egm_node(struct pci_dev *pdev) { struct egm_region *region, *temp_region; struct h_node *cur_page; unsigned long bkt; struct hlist_node *temp_node; + u64 egmphys, egmlength, egmpxm; + int ret; + + ret = nvgrace_gpu_fetch_egm_property(pdev, &egmphys, &egmlength, &egmpxm); + if (ret) + return; list_for_each_entry_safe(region, temp_region, &egm_list, list) { - if (egm_node == region->egmpxm) { + if (egmpxm == region->egmpxm) { hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { hash_del(&cur_page->node); vfree(cur_page); diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index d812cbbf84512..967313b9e0029 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1248,7 +1248,7 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, out_egm_unreg: if (egm_enabled) - unregister_egm_node(nvdev->egm_node); + unregister_egm_node(pdev); out_put_vdev: vfio_put_device(&nvdev->core_device.vdev); return ret; @@ -1262,7 +1262,7 @@ static void nvgrace_gpu_remove(struct pci_dev *pdev) core_device); if (egm_enabled) - unregister_egm_node(nvdev->egm_node); + unregister_egm_node(pdev); vfio_pci_core_unregister_device(core_device); vfio_put_device(&core_device->vdev); diff --git a/include/linux/nvgrace-egm.h b/include/linux/nvgrace-egm.h index 48add892aa5bf..4bbd383a02732 100644 --- a/include/linux/nvgrace-egm.h +++ b/include/linux/nvgrace-egm.h @@ -7,6 +7,6 @@ #define _NVGRACE_EGM_H int register_egm_node(struct pci_dev *pdev); -void unregister_egm_node(int egm_node); +void unregister_egm_node(struct pci_dev *pdev); #endif /* _NVGRACE_EGM_H */ From 5ef69c66445d7a9ac54e49973080385dadc9e5dd Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:38:38 -0500 Subject: [PATCH 073/464] NVIDIA: SAUCE: vfio/nvgrace-egm: track GPUs associated with the EGM regions BugLink: https://bugs.launchpad.net/bugs/2119656 GB200 systems could have multiple GPUs associated with an EGM region. For proper EGM functionality the host topology in terms of GPU affinity has to be replicated in the VM. Hence the EGM region structure must track the GPU devices belonging to the same socket. On the device probe, the device pci_dev struct is added to a linked list of the appropriate EGM region. Similarly on device remove, the pci_dev struct for the GPU is removed from the EGM region. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0001_vfio_nvgrace-egm_track_GPUs_associated_with_the_EGM_regions.patch (koba: Enhance error handling, Remove egm_node from unregister_egm_node and move destroy_egm_chardev a little forward) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 0222c35fb26285ee1a6185ef50414093850ea352 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 5ba1a1f84f9d noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c167095eba967b7ae64023927a015df0643c7460 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 68 ++++++++++++++++++++++++++---- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 1545ac695ad77..67cc5254f681b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -10,6 +10,11 @@ #define MAX_EGM_NODES 256 +struct gpu_node { + struct list_head list; + struct pci_dev *pdev; +}; + struct egm_region { struct list_head list; int egmpxm; @@ -18,6 +23,7 @@ struct egm_region { size_t egmlength; struct device device; struct cdev cdev; + struct list_head gpus; DECLARE_HASHTABLE(htbl, 0x10); }; @@ -187,6 +193,11 @@ static int setup_egm_chardev(struct egm_region *region) return ret; } +static void destroy_egm_chardev(struct egm_region *region) +{ + cdev_device_del(®ion->cdev, ®ion->device); +} + static int nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, u64 *pegmlength, u64 *pegmpxm) @@ -265,6 +276,32 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, memunmap(memaddr); } +static int add_gpu(struct egm_region *region, struct pci_dev *pdev) +{ + struct gpu_node *node; + + node = kvzalloc(sizeof(*node), GFP_KERNEL); + if (!node) + return -ENOMEM; + + node->pdev = pdev; + + list_add_tail(&node->list, ®ion->gpus); + return 0; +} + +static void remove_gpu(struct egm_region *region, struct pci_dev *pdev) +{ + struct gpu_node *node, *tmp; + + list_for_each_entry_safe(node, tmp, ®ion->gpus, list) { + if (node->pdev == pdev) { + list_del(&node->list); + kvfree(node); + } + } +} + int register_egm_node(struct pci_dev *pdev) { struct egm_region *region = NULL; @@ -275,11 +312,15 @@ int register_egm_node(struct pci_dev *pdev) if (ret) return ret; + /* Check if region already exists */ list_for_each_entry(region, &egm_list, list) { - if (region->egmphys == egmphys) - return 0; + if (region->egmphys == egmphys) { + /* Add GPU to existing region */ + return add_gpu(region, pdev); + } } + /* Create new region */ region = kvzalloc(sizeof(*region), GFP_KERNEL); if (!region) return -ENOMEM; @@ -289,28 +330,33 @@ int register_egm_node(struct pci_dev *pdev) region->egmpxm = egmpxm; hash_init(region->htbl); + INIT_LIST_HEAD(®ion->gpus); + atomic_set(®ion->open_count, 0); nvgrace_egm_fetch_bad_pages(pdev, region); ret = setup_egm_chardev(region); if (ret) - goto err; + goto err_free_region; list_add_tail(®ion->list, &egm_list); + ret = add_gpu(region, pdev); + if (ret) + goto err_remove_from_list; + return 0; -err: + +err_remove_from_list: + list_del(®ion->list); + destroy_egm_chardev(region); +err_free_region: kfree(region); return ret; } EXPORT_SYMBOL_GPL(register_egm_node); -static void destroy_egm_chardev(struct egm_region *region) -{ - cdev_device_del(®ion->cdev, ®ion->device); -} - void unregister_egm_node(struct pci_dev *pdev) { struct egm_region *region, *temp_region; @@ -326,6 +372,10 @@ void unregister_egm_node(struct pci_dev *pdev) list_for_each_entry_safe(region, temp_region, &egm_list, list) { if (egmpxm == region->egmpxm) { + remove_gpu(region, pdev); + if (!list_empty(®ion->gpus)) + break; + hash_for_each_safe(region->htbl, bkt, temp_node, cur_page, node) { hash_del(&cur_page->node); vfree(cur_page); From 5b59ac83555104613c4ec93c371f2310baf6a63a Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:39:33 -0500 Subject: [PATCH 074/464] NVIDIA: SAUCE: vfio/nvgrace-egm: list gpus through sysfs BugLink: https://bugs.launchpad.net/bugs/2119656 To replicate the host EGM topology in the VM in terms of the GPU affinity, the userspace need to be aware of which GPUs belong to the same socket as the EGM region. Expose the list of GPUs associated with an EGM region through sysfs. The list can be queried from the location /sys/devices/virtual/egm/egmX/gpu_devices. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0002_vfio_nvgrace-egm_list_gpus_through_sysfs.patch (koba: Enchance error handling for sysfs_create_group) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit fec2356d20f7054c0c89b1d32e7862bba34bda54 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 5dde2f0e0bb5 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit a5284ca673f5a3732431c7e1cb49a48c347a0087 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 41 +++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 67cc5254f681b..2988d55208bf4 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -276,6 +276,38 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, memunmap(memaddr); } +static ssize_t gpu_devices_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct egm_region *region = + container_of(dev, struct egm_region, device); + struct gpu_node *node, *temp_node; + int len = 0; + + list_for_each_entry_safe(node, temp_node, ®ion->gpus, list) { + struct pci_dev *pdev = node->pdev; + + len += sysfs_emit_at(buf, len, "%04x:%02x:%02x.%x\n", + pci_domain_nr(pdev->bus), + pdev->bus->number, + PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn)); + } + + return len; +} + +static DEVICE_ATTR_RO(gpu_devices); + +static struct attribute *attrs[] = { + &dev_attr_gpu_devices.attr, + NULL, +}; + +static struct attribute_group attr_group = { + .attrs = attrs, +}; + static int add_gpu(struct egm_region *region, struct pci_dev *pdev) { struct gpu_node *node; @@ -342,12 +374,18 @@ int register_egm_node(struct pci_dev *pdev) list_add_tail(®ion->list, &egm_list); - ret = add_gpu(region, pdev); + ret = sysfs_create_group(®ion->device.kobj, &attr_group); if (ret) goto err_remove_from_list; + ret = add_gpu(region, pdev); + if (ret) + goto err_remove_sysfs; + return 0; +err_remove_sysfs: + sysfs_remove_group(®ion->device.kobj, &attr_group); err_remove_from_list: list_del(®ion->list); destroy_egm_chardev(region); @@ -381,6 +419,7 @@ void unregister_egm_node(struct pci_dev *pdev) vfree(cur_page); } + sysfs_remove_group(®ion->device.kobj, &attr_group); destroy_egm_chardev(region); list_del(®ion->list); kfree(region); From b80680e88056943671ff12f0e2fee931cb93ac0c Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Tue, 6 May 2025 09:40:16 -0500 Subject: [PATCH 075/464] NVIDIA: SAUCE: vfio/nvgrace-egm: expose the egm size through sysfs BugLink: https://bugs.launchpad.net/bugs/2119656 To allocate the EGM, the userspace need to know it's size. Currently, there is no easy way for the userspace to determine that. Make nvgrace-egm expose the size through sysfs that can be queried by the userspace from /sys/devices/virtual/egm/egmX/egm_size. Signed-off-by: Ankit Agrawal Ref: sj24: /home/nvidia/ankita/kernel_patches/0003_vfio_nvgrace-egm_expose_the_egm_size_through_sysfs.patch Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit dcdcef245e8d648d38ef75f1023c7437b5639ddf https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 994015745197 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit e025c29ebeb237a2e85b86e5c6b1f83602e7f8bd noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 2988d55208bf4..1e8f2f10b06f9 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -299,8 +299,19 @@ static ssize_t gpu_devices_show(struct device *dev, struct device_attribute *att static DEVICE_ATTR_RO(gpu_devices); +static ssize_t egm_size_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct egm_region *region = + container_of(dev, struct egm_region, device); + return sysfs_emit(buf, "0x%lx\n", region->egmlength); +} + +static DEVICE_ATTR_RO(egm_size); + static struct attribute *attrs[] = { &dev_attr_gpu_devices.attr, + &dev_attr_egm_size.attr, NULL, }; From 853d41e4b38095b0595823ba2c829d0c62e68797 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Sun, 18 Jan 2026 02:03:13 +0000 Subject: [PATCH 076/464] NVIDIA: SAUCE: vfio/nvgrace-egm: register EGM PFNMAP range with memory_failure BugLink: https://bugs.launchpad.net/bugs/2138892 EGM carveout memory is mapped directly into userspace (QEMU) and is not added to the kernel. It is not managed by the kernel page allocator and has no struct pages. The module can thus utilize the Linux memory manager's memory_failure mechanism for regions with no struct pages. The Linux MM code exposes register/unregister APIs allowing modules to register such memory regions for memory_failure handling. Register the EGM PFN range with the MM memory_failure infrastructure on open, and unregister it on the last close. Provide a PFN-to-VMA offset callback that validates the PFN is within the EGM region and the VMA, then converts it to a file offset and records the poisoned offset in the existing hashtable for reporting to userspace. Signed-off-by: Ankit Agrawal Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 3fde504ffb6ab8def2971607833069daa29835c4 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 100 ++++++++++++++++++++++++++++- 1 file changed, 99 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 1e8f2f10b06f9..9e47813f4ecbe 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #define MAX_EGM_NODES 256 @@ -25,6 +27,7 @@ struct egm_region { struct cdev cdev; struct list_head gpus; DECLARE_HASHTABLE(htbl, 0x10); + struct pfn_address_space pfn_address_space; }; struct h_node { @@ -36,11 +39,97 @@ static dev_t dev; static struct class *class; static struct list_head egm_list; +static int pfn_memregion_offset(struct egm_region *region, + unsigned long pfn, + pgoff_t *pfn_offset_in_region) +{ + unsigned long start_pfn, num_pages; + + start_pfn = PHYS_PFN(region->egmphys); + num_pages = region->egmlength >> PAGE_SHIFT; + + if (pfn < start_pfn || pfn >= start_pfn + num_pages) + return -EFAULT; + + *pfn_offset_in_region = pfn - start_pfn; + + return 0; +} + +static int track_ecc_offset(struct egm_region *region, + unsigned long mem_offset) +{ + struct h_node *cur_page, *ecc_page; + unsigned long bkt; + + hash_for_each(region->htbl, bkt, cur_page, node) { + if (cur_page->mem_offset == mem_offset) + return 0; + } + + ecc_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + if (!ecc_page) + return -ENOMEM; + + ecc_page->mem_offset = mem_offset; + + hash_add(region->htbl, &ecc_page->node, ecc_page->mem_offset); + + return 0; +} + +static int nvgrace_egm_pfn_to_vma_pgoff(struct vm_area_struct *vma, + unsigned long pfn, + pgoff_t *pgoff) +{ + struct egm_region *region = vma->vm_file->private_data; + pgoff_t vma_offset_in_region = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + pgoff_t pfn_offset_in_region; + int ret; + + ret = pfn_memregion_offset(region, pfn, &pfn_offset_in_region); + if (ret) + return ret; + + /* Ensure PFN is not before VMA's start within the region */ + if (pfn_offset_in_region < vma_offset_in_region) + return -EFAULT; + + /* Calculate offset from VMA start */ + *pgoff = vma->vm_pgoff + + (pfn_offset_in_region - vma_offset_in_region); + + /* Track and save the poisoned offset */ + return track_ecc_offset(region, *pgoff << PAGE_SHIFT); +} + +static int +nvgrace_egm_vfio_pci_register_pfn_range(struct inode *inode, + struct egm_region *region) +{ + int ret; + unsigned long pfn, nr_pages; + + pfn = PHYS_PFN(region->egmphys); + nr_pages = region->egmlength >> PAGE_SHIFT; + + region->pfn_address_space.node.start = pfn; + region->pfn_address_space.node.last = pfn + nr_pages - 1; + region->pfn_address_space.mapping = inode->i_mapping; + region->pfn_address_space.pfn_to_vma_pgoff = nvgrace_egm_pfn_to_vma_pgoff; + + ret = register_pfn_address_space(®ion->pfn_address_space); + + return ret; +} + static int nvgrace_egm_open(struct inode *inode, struct file *file) { void *memaddr; struct egm_region *region = container_of(inode->i_cdev, struct egm_region, cdev); + int ret; if (!region) return -EINVAL; @@ -58,6 +147,12 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) memunmap(memaddr); file->private_data = region; + ret = nvgrace_egm_vfio_pci_register_pfn_range(inode, region); + if (ret && ret != -EOPNOTSUPP) { + file->private_data = NULL; + return ret; + } + return 0; } @@ -69,8 +164,11 @@ static int nvgrace_egm_release(struct inode *inode, struct file *file) if (!region) return -EINVAL; - if (atomic_dec_and_test(®ion->open_count)) + if (atomic_dec_and_test(®ion->open_count)) { + unregister_pfn_address_space(®ion->pfn_address_space); + file->private_data = NULL; + } return 0; } From 672a2af83c0d1ecbc234e60abc731b717de66190 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 7 Nov 2024 20:03:50 -0800 Subject: [PATCH 077/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Address smatch errors BugLink: https://bugs.launchpad.net/bugs/2119656 Return the intended errno upon a copyout fault, remove unnecessary checks following container_of pointer derivation, and use the correct macro and types for overflow checking. Signed-off-by: Matthew R. Ochs Acked-by: Kai-Heng Feng Acked-by: Carol L. Soto Acked-by: Koba Ko Signed-off-by: Matthew R. Ochs (cherry picked from commit 429910b6fba450a9831f590d9622d16b79006311 https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.8-next) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit bda63f340176a3a610a64512176f0e133b9efd9f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit 942bf3b26275 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 63dd05905701cd6f68b3194afafca2bedb344222 noble:linux-nvidia-6.17) Signed-off-by: Matthew R. Ochs --- drivers/vfio/pci/nvgrace-gpu/egm.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9e47813f4ecbe..aa9e796b6fb4f 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -131,9 +131,6 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) struct egm_region, cdev); int ret; - if (!region) - return -EINVAL; - if (atomic_inc_return(®ion->open_count) > 1) return 0; @@ -161,9 +158,6 @@ static int nvgrace_egm_release(struct inode *inode, struct file *file) struct egm_region *region = container_of(inode->i_cdev, struct egm_region, cdev); - if (!region) - return -EINVAL; - if (atomic_dec_and_test(®ion->open_count)) { unregister_pfn_address_space(®ion->pfn_address_space); @@ -242,7 +236,7 @@ static long nvgrace_egm_ioctl(struct file *file, unsigned int cmd, unsigned long index * bad_page_struct_size, &tmp, bad_page_struct_size); if (ret) - return ret; + return -EFAULT; index++; } @@ -311,7 +305,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmlength > type_max(size_t)) + if (overflows_type(*pegmlength, size_t)) return -EOVERFLOW; ret = device_property_read_u64(&pdev->dev, "nvidia,egm-base-pa", @@ -319,7 +313,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmphys > type_max(phys_addr_t)) + if (overflows_type(*pegmphys, phys_addr_t)) return -EOVERFLOW; ret = device_property_read_u64(&pdev->dev, "nvidia,egm-pxm", @@ -327,7 +321,7 @@ nvgrace_gpu_fetch_egm_property(struct pci_dev *pdev, u64 *pegmphys, if (ret) return ret; - if (*pegmpxm > type_max(phys_addr_t)) + if (overflows_type(*pegmpxm, int)) return -EOVERFLOW; return 0; From c4c5721aed8a2c1d87e4f3d140ea7e43b503674a Mon Sep 17 00:00:00 2001 From: kobakonvidia Date: Mon, 26 May 2025 16:48:35 +0000 Subject: [PATCH 078/464] NVIDIA: SAUCE: vfio/nvgrace-egm: Add null pointer checks after memory allocations BugLink: https://bugs.launchpad.net/bugs/2119656 Add missing null pointer checks after vzalloc() calls in the NVIDIA Grace GPU driver's EGM (External GPU Memory) handling code. This prevents potential null pointer dereferences in the memory failure handling and bad page fetching functions, providing proper error handling for allocation failures. Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L. Soto Signed-off-by: Matthew R. Ochs (cherry picked from commit 63127e2996a244841309ee86b4535f41e2b0de1f https://github.com/NVIDIA/NV-Kernels/tree/24.04_linux-nvidia-adv-6.11-next) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit e5f0c8d1ba27 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (backported from commit 862ed5a2b5c58c1a1bc5d6a9d012ea82ee177389 noble:linux-nvidia-6.17) [mochs: Addressed collission for a null pointer check that is no longer needed] Signed-off-by: Matthew R. Ochs --- drivers/vfio/pci/nvgrace-gpu/egm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index aa9e796b6fb4f..9ad08c9cb59d0 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -360,6 +360,8 @@ static void nvgrace_egm_fetch_bad_pages(struct pci_dev *pdev, * apps. */ retired_page = (struct h_node *)(vzalloc(sizeof(struct h_node))); + if (!retired_page) + continue; /* Skip this entry on allocation failure */ retired_page->mem_offset = *((u64 *)memaddr + index + 1) - region->egmphys; hash_add(region->htbl, &retired_page->node, retired_page->mem_offset); From 7ae7737d5ee93f121ce8deb329c550bd6bb2d9e8 Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Fri, 13 Feb 2026 04:11:25 +0000 Subject: [PATCH 079/464] NVIDIA: SAUCE: vfio/nvgrace-egm: split zapping EGM into 1GB chunks BugLink: https://bugs.launchpad.net/bugs/2142160 When initializing EGM (Extended GPU Memory) regions, the current implementation performs a single memset operation over the entire memory region. For very large regions, this can result in long-running uninterruptible operations that may cause system responsiveness issues or trigger watchdog timeouts. Split the memset operation into 1GB chunks. Signed-off-by: Ankit Agrawal Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 355474478031a909b2588458efb9d2c48e571735 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/vfio/pci/nvgrace-gpu/egm.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/egm.c b/drivers/vfio/pci/nvgrace-gpu/egm.c index 9ad08c9cb59d0..a2e4c05a83c34 100644 --- a/drivers/vfio/pci/nvgrace-gpu/egm.c +++ b/drivers/vfio/pci/nvgrace-gpu/egm.c @@ -140,7 +140,20 @@ static int nvgrace_egm_open(struct inode *inode, struct file *file) return -EINVAL; } - memset((u8 *)memaddr, 0, region->egmlength); + { + size_t remaining = region->egmlength; + u8 *chunk_addr = (u8 *)memaddr; + size_t chunk_size; + + while (remaining > 0) { + chunk_size = min(remaining, SZ_1G); + memset(chunk_addr, 0, chunk_size); + cond_resched(); + chunk_addr += chunk_size; + remaining -= chunk_size; + } + } + memunmap(memaddr); file->private_data = region; From e04f9f4cdada4db371b1b03d0fba91bc2b49826a Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 14 Jul 2025 06:56:53 -0700 Subject: [PATCH 080/464] NVIDIA: SAUCE: arm64: configs: enable NVGRACE_EGM as module BugLink: https://bugs.launchpad.net/bugs/2119656 Add CONFIG_NVGRACE_EGM with policy 'm' for arm64 architecture. Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matt Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off--by: Brad Figg (cherry picked from commit ddf68d0e4cc7 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 9ef26d933e9a9a1e1468393b0d59c0b758b1301a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index d6b264f2030fe..76b45d4e6b595 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -156,6 +156,9 @@ CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> +CONFIG_NVGRACE_EGM policy<{'arm64': 'm'}> +CONFIG_NVGRACE_EGM note<'LP: #2119656'> + CONFIG_NVIDIA_FFA_EC policy<{'arm64': 'y'}> CONFIG_NVIDIA_FFA_EC note<'LP: #2114230'> From 5b89b49fb2f3aad2d4c9e485590dd341fcc68bdc Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 26 Aug 2025 16:27:29 -0500 Subject: [PATCH 081/464] UBUNTU: [Config] nvidia: Use performance CPU frequency governor on amd64 BugLink: https://bugs.launchpad.net/bugs/2028576 The bug indicates NVIDIA wanted to enable the performance governor by default on all arches for the NVIDIA kernels. However, this was mistakenly only configured for arm64 systems. Fix this by also using the performance CPU frequency governor as the default on amd64 systems. Signed-off-by: Jacob Martin (cherry picked from commit a5304114539a272f61f5e9085c6c9ab4f1e1c783) (cherry picked from commit a5304114539a noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 93d3ca3c2d83d06cb771abf0c177e88060f02f76 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 76b45d4e6b595..20b2f23c27369 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -97,10 +97,13 @@ CONFIG_CORESIGHT_TRBE policy<{'arm64': 'm'}> CONFIG_CORESIGHT_TRBE note<'Required for Grace enablement'> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND policy<{'arm64': 'n'}> -CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND note<'required for NVIDIA workloads'> +CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND note<'LP: #2028576: Perf governor required for NVIDIA workloads'> -CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE policy<{'amd64': 'n', 'arm64': 'y'}> -CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'required for NVIDIA workloads'> +CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'LP: #2028576: Perf governor required for NVIDIA workloads'> + +CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL note<'LP: #2028576: Perf governor required for NVIDIA workloads'> CONFIG_DRM_NOUVEAU policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_DRM_NOUVEAU note<'Disable nouveau for NVIDIA kernels'> From 776fd695fbb4044a5a36d3b1169ec6108e8c56a2 Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Mon, 15 Sep 2025 17:23:32 +0530 Subject: [PATCH 082/464] NVIDIA: SAUCE: Fix FFA notification count initialization BugLink: https://bugs.launchpad.net/bugs/2123861 In nvidia_ffa_create_notifications(), it invokes nvidia_ffa_fill_notification_map(), which fills the virtual notification IDs array supported by the current FFA device. This function updates notification_count after traversing the notification array in the _DSD method. For FFA devices without an entry in the _DSD method, notification_count is assumed to be zero initialized. However, nvidia_ffa_ec_service_probe() uses kmalloc() instead of kzalloc(), so notification_count may contain random values. This causes FFA device probe failures. This patch fixes this by using kzalloc() to zero initialize the nvidia_ec_ffa_device structure. Fixes: ae8718738ca8 ("NVIDIA: SAUCE: Add support for notifications from secure EC services") Signed-off-by: Abhishek Sahu Acked-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off--by: Brad Figg (cherry picked from commit 518a89b25556 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit bf0587bed38496ce0affa41f55a51c140a4ef3e0 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/platform/arm64/nvidia-ffa-ec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/platform/arm64/nvidia-ffa-ec.c b/drivers/platform/arm64/nvidia-ffa-ec.c index d9e8b7fdda30c..7a259ba64956c 100644 --- a/drivers/platform/arm64/nvidia-ffa-ec.c +++ b/drivers/platform/arm64/nvidia-ffa-ec.c @@ -617,7 +617,7 @@ static int nvidia_ffa_ec_service_probe(struct ffa_device *ffa_dev) return -ENODEV; } - nvidia_ec_ffa_dev = devm_kmalloc(&ffa_dev->dev, + nvidia_ec_ffa_dev = devm_kzalloc(&ffa_dev->dev, sizeof(*nvidia_ec_ffa_dev), GFP_KERNEL); if (!nvidia_ec_ffa_dev) { From 8c83451a8b7d023244dac1c23a073e10fc923357 Mon Sep 17 00:00:00 2001 From: Us Chien Date: Mon, 26 May 2025 17:46:21 +0800 Subject: [PATCH 083/464] NVIDIA: SAUCE: MEDIATEK: usb: host: xhci-hub: fix MT89xx SoCs return PORTLI value BugLink: https://bugs.launchpad.net/bugs/2125126 For DIGITS GB10, USB SuperSpeed Plus Gen2x1 device enumeration speed is being shown as Gen2x2 in the dmesg log. usb 4-1: new SuperSpeed Plus Gen 2x2 USB device number 2 using xhci-hcd The USB3 link lanes count is recorded in the read-only TLC and RLC fields of the PORTLI register. In MT89xx SoCs (used by GB10), the lane count can be wrongly set to 2 instead of 1 for USB Gen2x1 devices due to HW Bug. As per Table 7-13 in USB xHCI revision 1.2 specification, the value 0x5 is for SuperSpeedPlus Gen2x1. This patch adds a SW WAR to read the port speed in PORTSC register. If port speed value is 0x5, then the lane count can be updated to 1. Signed-off-by: Us Chien Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Kai-Heng Feng Acked-by: Matthew R. Ochs Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off--by: Brad Figg (cherry picked from commit b5b3a58605e6 noble:linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit c65499578d0c4af75076eea7d9a44ba8476d34a7 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/usb/host/xhci-hub.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/drivers/usb/host/xhci-hub.c b/drivers/usb/host/xhci-hub.c index 04cc3d681495f..09a92ab1537b2 100644 --- a/drivers/usb/host/xhci-hub.c +++ b/drivers/usb/host/xhci-hub.c @@ -1290,6 +1290,17 @@ int xhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue, } port_li = readl(&port->port_reg->portli); status = xhci_get_ext_port_status(temp, port_li); + + /* + * In MT8901 USB host controller, the lane count can be wrongly set + * to 2 instead of 1 for USB Gen2x1 devices due to a HW Bug. As a SW + * WAR, check if port speed is 0x5 (SuperSpeedPlus Gen2x1) in + * PORTSC register and update the lane count as 1. + */ + if ((xhci->quirks & XHCI_NVIDIA_MT8901_HOST) && + DEV_SUPERSPEEDPLUS(temp)) + status &= ~0xff00; + put_unaligned_le32(status, &buf[4]); } break; From 2d4179081a183c6f6a27be900fb47a46a4a4fa26 Mon Sep 17 00:00:00 2001 From: Andy Ritger Date: Tue, 20 May 2025 17:11:58 -0700 Subject: [PATCH 084/464] NVIDIA: SAUCE: iommu/io-pgtable-arm: backport contiguous bit support BugLink: https://bugs.launchpad.net/bugs/2112600 iommu/io-pgtable-arm: Support contiguous bit in translation tables The contiguous bit in translation table entries can be used as a hint to SMMU that a group of adjacent translation table entries have consistent attributes and point to a contiguous and properly aligned output address range. This enables SMMU to predict the properties of the remaining translation table entries in the same group without accessing them. It also allows an SMMU implementation to make more efficient use of its TLB by using a single TLB entry to cover all translation table entries in the same group. In the case of 4KB granule size, there are 16 translation table entries in one group. This change sets the contiguous bit for such groups of entries that are completely covered by a single call to map_pages. As it stands, the code wouldn't set the contiguous bit if a group of adjacent descriptors is completed by separate calls to map_pages. Signed-off-by: Daniel Mentz Link: https://lore.kernel.org/linux-iommu/20250430231924.1481493-1-danielmentz@google.com/ [aritger: For the backport: updated the __arm_lpae_init_pte() callsite in arm_lpae_split_blk_unmap() (arm_lpae_split_blk_unmap() is no longer present at top of tree where the original commit was written).] Signed-off-by: Andy Ritger Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off--by: Brad Figg (backported from commit 990fa55beac8efffb7a94b26be553d52c232b9f1 linux-nvidia-6.11) [kobak: arm_lpae_split_blk_unmap is removed since https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=33729a5fc0caf7a97d20507acbeee6b012e7e519 iommu/io-pgtable-arm: Remove split on unmap behavior so modify as per functions] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Ian May (cherry picked from commit c601e605745ea9f16834da6e24049b92b759b78f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/iommu/io-pgtable-arm.c | 52 ++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 0208e5897c299..1ebf9631c1825 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -75,6 +75,7 @@ #define ARM_LPAE_PTE_NSTABLE (((arm_lpae_iopte)1) << 63) #define ARM_LPAE_PTE_XN (((arm_lpae_iopte)3) << 53) +#define ARM_LPAE_PTE_CONT (((arm_lpae_iopte)1) << 52) #define ARM_LPAE_PTE_DBM (((arm_lpae_iopte)1) << 51) #define ARM_LPAE_PTE_AF (((arm_lpae_iopte)1) << 10) #define ARM_LPAE_PTE_SH_NS (((arm_lpae_iopte)0) << 8) @@ -320,6 +321,27 @@ static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, sizeof(*ptep) * num_entries, DMA_TO_DEVICE); } +static int arm_lpae_cont_ptes(int lvl, struct arm_lpae_io_pgtable *data) +{ + switch (ARM_LPAE_GRANULE(data)) { + case SZ_4K: + if (lvl >= 1) + return 16; + break; + case SZ_16K: + if (lvl == 2) + return 32; + else if (lvl == 3) + return 128; + break; + case SZ_64K: + if (lvl >= 2) + return 32; + break; + } + return 1; +} + static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_cfg *cfg, int num_entries) { for (int i = 0; i < num_entries; i++) @@ -329,13 +351,35 @@ static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_cfg *cf __arm_lpae_sync_pte(ptep, num_entries, cfg); } +static bool arm_lpae_use_contpte(struct arm_lpae_io_pgtable *data, + unsigned long iova, phys_addr_t paddr, + int lvl, int num_entries, int i) +{ + size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data); + int cont_ptes = arm_lpae_cont_ptes(lvl, data); + int contmask = cont_ptes - 1; + int contpte_addr_mask = sz * cont_ptes - 1; + int map_idx_start, tbl_idx; + + if ((paddr & contpte_addr_mask) != (iova & contpte_addr_mask)) + return false; + + map_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data); + tbl_idx = map_idx_start + i; + if (((tbl_idx & contmask) <= i) && + (tbl_idx < ((map_idx_start + num_entries) & ~contmask))) + return true; + + return false; +} + static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, struct iommu_iotlb_gather *gather, unsigned long iova, size_t size, size_t pgcount, int lvl, arm_lpae_iopte *ptep); static void __arm_lpae_init_pte(struct arm_lpae_io_pgtable *data, - phys_addr_t paddr, arm_lpae_iopte prot, + unsigned long iova, phys_addr_t paddr, arm_lpae_iopte prot, int lvl, int num_entries, arm_lpae_iopte *ptep) { arm_lpae_iopte pte = prot; @@ -349,7 +393,9 @@ static void __arm_lpae_init_pte(struct arm_lpae_io_pgtable *data, pte |= ARM_LPAE_PTE_TYPE_BLOCK; for (i = 0; i < num_entries; i++) - ptep[i] = pte | paddr_to_iopte(paddr + i * sz, data); + ptep[i] = pte | paddr_to_iopte(paddr + i * sz, data) | + (arm_lpae_use_contpte(data, iova, paddr, lvl, num_entries, i) ? + ARM_LPAE_PTE_CONT : 0); if (!cfg->coherent_walk) __arm_lpae_sync_pte(ptep, num_entries, cfg); @@ -383,7 +429,7 @@ static int arm_lpae_init_pte(struct arm_lpae_io_pgtable *data, } } - __arm_lpae_init_pte(data, paddr, prot, lvl, num_entries, ptep); + __arm_lpae_init_pte(data, iova, paddr, prot, lvl, num_entries, ptep); return 0; } From 086c374d26d5aadc0f2e4901ea05b8ceaf3a9155 Mon Sep 17 00:00:00 2001 From: Leon Yen Date: Fri, 26 Sep 2025 13:34:47 +0800 Subject: [PATCH 085/464] NVIDIA: SAUCE: wifi: mt76: mt7925: introduce CSA support in non-MLO mode BugLink: https://bugs.launchpad.net/bugs/2129209 Add CSA (Channel Switch Announcement) related implementation in collaboration with mac80211 to deal with dynamic channel switching. Signed-off-by: Leon Yen Signed-off-by: Ming Yen Hsieh Conflicts: - drivers/net/wireless/mediatek/mt76/mt7925/main.c Code is different in mt7925_add_interface() - drivers/net/wireless/mediatek/mt76/mt792x_core.c Code organization is different Signed-off-by: Abhishek Sahu [Backported from https://patchwork.kernel.org/project/linux-wireless/patch/20250926053447.4036650-1-mingyen.hsieh@mediatek.com/] Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Jacob Martin Signed-off-by: Ian May (cherry picked from commit 060dc92285ea8a49e7ea0a87f028066cd105b62f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../net/wireless/mediatek/mt76/mt7925/main.c | 138 ++++++++++++++++++ .../wireless/mediatek/mt76/mt7925/mt7925.h | 1 + .../net/wireless/mediatek/mt76/mt792x_core.c | 4 +- 3 files changed, 140 insertions(+), 3 deletions(-) diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/main.c b/drivers/net/wireless/mediatek/mt76/mt7925/main.c index fec54d5f4eaf1..28de962a99a0e 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7925/main.c @@ -245,6 +245,7 @@ int mt7925_init_mlo_caps(struct mt792x_phy *phy) { struct wiphy *wiphy = phy->mt76->hw->wiphy; static const u8 ext_capa_sta[] = { + [0] = WLAN_EXT_CAPA1_EXT_CHANNEL_SWITCHING, [2] = WLAN_EXT_CAPA3_MULTI_BSSID_SUPPORT, [7] = WLAN_EXT_CAPA8_OPMODE_NOTIF, }; @@ -438,6 +439,8 @@ mt7925_add_interface(struct ieee80211_hw *hw, struct ieee80211_vif *vif) if (phy->chip_cap & MT792x_CHIP_CAP_RSSI_NOTIFY_EVT_EN) vif->driver_flags |= IEEE80211_VIF_SUPPORTS_CQM_RSSI; + INIT_WORK(&mvif->csa_work, mt7925_csa_work); + timer_setup(&mvif->csa_timer, mt792x_csa_timer, 0); out: mt792x_mutex_release(dev); @@ -1753,6 +1756,10 @@ static int mt7925_add_chanctx(struct ieee80211_hw *hw, struct ieee80211_chanctx_conf *ctx) { + struct mt792x_dev *dev = mt792x_hw_dev(hw); + + dev->new_ctx = ctx; + return 0; } @@ -1760,6 +1767,11 @@ static void mt7925_remove_chanctx(struct ieee80211_hw *hw, struct ieee80211_chanctx_conf *ctx) { + struct mt792x_dev *dev = mt792x_hw_dev(hw); + + if (dev->new_ctx == ctx) + dev->new_ctx = NULL; + } static void @@ -2142,6 +2154,11 @@ static void mt7925_unassign_vif_chanctx(struct ieee80211_hw *hw, mctx->bss_conf = NULL; mconf->mt76.ctx = NULL; mutex_unlock(&dev->mt76.mutex); + + if (link_conf->csa_active) { + timer_delete_sync(&mvif->csa_timer); + cancel_work_sync(&mvif->csa_work); + } } static void mt7925_rfkill_poll(struct ieee80211_hw *hw) @@ -2156,6 +2173,121 @@ static void mt7925_rfkill_poll(struct ieee80211_hw *hw) wiphy_rfkill_set_hw_state(hw->wiphy, ret == 0); } +static int mt7925_switch_vif_chanctx(struct ieee80211_hw *hw, + struct ieee80211_vif_chanctx_switch *vifs, + int n_vifs, + enum ieee80211_chanctx_switch_mode mode) +{ + return mt7925_assign_vif_chanctx(hw, vifs->vif, vifs->link_conf, + vifs->new_ctx); +} + +void mt7925_csa_work(struct work_struct *work) +{ + struct mt792x_vif *mvif; + struct mt792x_dev *dev; + struct ieee80211_vif *vif; + struct ieee80211_bss_conf *link_conf; + struct mt792x_bss_conf *mconf; + u8 link_id, roc_rtype; + int ret = 0; + + mvif = (struct mt792x_vif *)container_of(work, struct mt792x_vif, + csa_work); + dev = mvif->phy->dev; + vif = container_of((void *)mvif, struct ieee80211_vif, drv_priv); + + if (ieee80211_vif_is_mld(vif)) + return; + + if (!dev->new_ctx) + return; + + link_id = 0; + mconf = &mvif->bss_conf; + link_conf = &vif->bss_conf; + roc_rtype = MT7925_ROC_REQ_JOIN; + + mt792x_mutex_acquire(dev); + ret = mt7925_set_roc(mvif->phy, mconf, dev->new_ctx->def.chan, + 4000, roc_rtype); + mt792x_mutex_release(dev); + if (!ret) { + mt792x_mutex_acquire(dev); + ret = mt7925_mcu_set_chctx(mvif->phy->mt76, &mconf->mt76, link_conf, + dev->new_ctx); + mt792x_mutex_release(dev); + + mt7925_abort_roc(mvif->phy, mconf); + } + + ieee80211_chswitch_done(vif, !ret, link_id); +} + +static int mt7925_pre_channel_switch(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_channel_switch *chsw) +{ + if (ieee80211_vif_is_mld(vif)) + return -EOPNOTSUPP; + + if (vif->type != NL80211_IFTYPE_STATION || !vif->cfg.assoc) + return -EOPNOTSUPP; + + if (!cfg80211_chandef_usable(hw->wiphy, &chsw->chandef, + IEEE80211_CHAN_DISABLED)) + return -EOPNOTSUPP; + + return 0; +} + +static void mt7925_channel_switch(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_channel_switch *chsw) +{ + struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv; + u16 beacon_interval; + + if (ieee80211_vif_is_mld(vif)) + return; + + beacon_interval = vif->bss_conf.beacon_int; + + mvif->csa_timer.expires = TU_TO_EXP_TIME(beacon_interval * chsw->count); + add_timer(&mvif->csa_timer); +} + +static void mt7925_abort_channel_switch(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_bss_conf *link_conf) +{ + struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv; + + timer_delete_sync(&mvif->csa_timer); + cancel_work_sync(&mvif->csa_work); +} + +static void mt7925_channel_switch_rx_beacon(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct ieee80211_channel_switch *chsw) +{ + struct mt792x_dev *dev = mt792x_hw_dev(hw); + struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv; + u16 beacon_interval; + + if (ieee80211_vif_is_mld(vif)) + return; + + beacon_interval = vif->bss_conf.beacon_int; + + if (cfg80211_chandef_identical(&chsw->chandef, + &dev->new_ctx->def) && + chsw->count) { + mod_timer(&mvif->csa_timer, + TU_TO_EXP_TIME(beacon_interval * chsw->count)); + } +} + const struct ieee80211_ops mt7925_ops = { .tx = mt792x_tx, .start = mt7925_start, @@ -2219,6 +2351,12 @@ const struct ieee80211_ops mt7925_ops = { .change_vif_links = mt7925_change_vif_links, .change_sta_links = mt7925_change_sta_links, .rfkill_poll = mt7925_rfkill_poll, + + .switch_vif_chanctx = mt7925_switch_vif_chanctx, + .pre_channel_switch = mt7925_pre_channel_switch, + .channel_switch = mt7925_channel_switch, + .abort_channel_switch = mt7925_abort_channel_switch, + .channel_switch_rx_beacon = mt7925_channel_switch_rx_beacon, }; EXPORT_SYMBOL_GPL(mt7925_ops); diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/mt7925.h b/drivers/net/wireless/mediatek/mt76/mt7925/mt7925.h index a1d902ccce6d2..0f0eff748bb77 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/mt7925.h +++ b/drivers/net/wireless/mediatek/mt76/mt7925/mt7925.h @@ -298,6 +298,7 @@ int mt7925_mcu_uni_rx_ba(struct mt792x_dev *dev, void mt7925_mlo_pm_work(struct work_struct *work); void mt7925_scan_work(struct work_struct *work); void mt7925_roc_work(struct work_struct *work); +void mt7925_csa_work(struct work_struct *work); int mt7925_mcu_uni_bss_ps(struct mt792x_dev *dev, struct ieee80211_bss_conf *link_conf); void mt7925_coredump_work(struct work_struct *work); diff --git a/drivers/net/wireless/mediatek/mt76/mt792x_core.c b/drivers/net/wireless/mediatek/mt76/mt792x_core.c index f2ed16feb6c1b..1d8b8c00b7eef 100644 --- a/drivers/net/wireless/mediatek/mt76/mt792x_core.c +++ b/drivers/net/wireless/mediatek/mt76/mt792x_core.c @@ -691,9 +691,7 @@ int mt792x_init_wiphy(struct ieee80211_hw *hw) ieee80211_hw_set(hw, SUPPORTS_MULTI_BSSID); ieee80211_hw_set(hw, SUPPORTS_ONLY_HE_MULTI_BSSID); - if (is_mt7921(&dev->mt76)) { - ieee80211_hw_set(hw, CHANCTX_STA_CSA); - } + ieee80211_hw_set(hw, CHANCTX_STA_CSA); if (dev->pm.enable) ieee80211_hw_set(hw, CONNECTION_MONITOR); From 4e2658384744eb2bca6af7b6d6fa8511fb5976a0 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Wed, 10 Sep 2025 08:23:52 -0700 Subject: [PATCH 086/464] NVIDIA: SAUCE: iommu/arm-smmu-v3: Set DGX Spark iGPU default domain type to DMA BugLink: https://bugs.launchpad.net/bugs/2129776 Force DGX Spark systems to use DMA translation as current drivers require this. Suggested-by: Jason Gunthorpe Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Jacob Martin Signed-off-by: Ian May (cherry picked from commit 54575cfd0fa45463ba90a504e01fcd27d9dd020f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 606abe051e68b..eab18dbd6191a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3737,6 +3737,9 @@ static int arm_smmu_def_domain_type(struct device *dev) if (IS_HISI_PTT_DEVICE(pdev)) return IOMMU_DOMAIN_IDENTITY; + + if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && pdev->device == 0x2E12) + return IOMMU_DOMAIN_DMA; } return 0; From 4014550605a110cf5ef661d95202031e029c458d Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Fri, 24 Oct 2025 09:44:45 -0700 Subject: [PATCH 087/464] UBUNTU: [Config] nvidia: Update annotations to set CONFIG_IOMMU_DEFAULT_PASSTHROUGH BugLink: https://bugs.launchpad.net/bugs/2129776 Default to CONFIG_IOMMU_DEFAULT_PASSTHROUGH on NVIDIA CPU on kernel above 6.11 as suggested by perf team. x86 always defaults to CONFIG_IOMMU_DEFAULT_DMA_LAZY so remove redundant amd64 setting. Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Jacob Martin Signed-off-by: Ian May (backported from commit e74a7d849ad16ef5a96df087a880c42ce6985a0b noble:linux-nvidia-6.17) [mochs: Addressed minor context collission] Signed-off-by: Matthew R. Ochs --- debian.nvidia-bos/config/annotations | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 20b2f23c27369..e9093504c2ddd 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -132,6 +132,15 @@ CONFIG_GPIO_AAEON note<'Disable all Ubuntu ODM dri CONFIG_IOMMUFD_VFIO_CONTAINER policy<{'arm64': 'y'}> CONFIG_IOMMUFD_VFIO_CONTAINER note<'LP: #2095028'> +CONFIG_IOMMU_DEFAULT_DMA_LAZY policy<{'amd64': 'y', 'arm64': 'n'}> +CONFIG_IOMMU_DEFAULT_DMA_LAZY note<'On Nvidia CPU passthrough mode is recommend so set passthrough mode as default for better performance'> + +CONFIG_IOMMU_DEFAULT_DMA_STRICT policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_IOMMU_DEFAULT_DMA_STRICT note<'On Nvidia CPU passthrough mode is recommend so set passthrough mode as default for better performance'> + +CONFIG_IOMMU_DEFAULT_PASSTHROUGH policy<{'amd64': 'n', 'arm64': 'y'}> +CONFIG_IOMMU_DEFAULT_PASSTHROUGH note<'On Nvidia CPU passthrough mode is recommend so set passthrough mode as default for better performance'> + CONFIG_LEDS_AAEON policy<{'amd64': '-'}> CONFIG_LEDS_AAEON note<'Disable all Ubuntu ODM drivers'> From 1e6bf44e4e638f9c8f8d3a6b5b81fff7ebc79b23 Mon Sep 17 00:00:00 2001 From: ChunHao Lin Date: Tue, 28 Oct 2025 22:51:44 +0800 Subject: [PATCH 088/464] NVIDIA: SAUCE: r8127: fix a kernel panic when dump all registers BugLink: https://bugs.launchpad.net/bugs/2130445 The call to cat registers2 will acquire rtnl_lock and dump all mapped mmio. Due to acquire rtnl_lock too long, it will cause a kernel panic "not syncing: SBSA Generic Watchdog timeout". Fix this issue by acquiring rtnl_lock every 16 byte when dump all mapped mmio. Signed-off-by: ChunHao Lin Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Ian May Acked-by: Noah Wager Acked-by: Abdur Rahman Signed-off--by: Brad Figg (cherry picked from commit 9737b84c973cb1e6b8352d6a679051b4d4b09acc noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/r8127_n.c | 32 ++++++++++++-------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/drivers/net/ethernet/realtek/r8127/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c index 4f83e44869deb..4ec33b3d77d5f 100755 --- a/drivers/net/ethernet/realtek/r8127/r8127_n.c +++ b/drivers/net/ethernet/realtek/r8127/r8127_n.c @@ -1161,17 +1161,21 @@ static int proc_get_registers(struct seq_file *m, void *v) seq_puts(m, "\nDump MAC Registers\n"); seq_puts(m, "Offset\tValue\n------\t-----\n"); - rtnl_lock(); - for (n = 0; n < max;) { seq_printf(m, "\n0x%04x:\t", n); + rtnl_lock(); + for (i = 0; i < 16 && n < max; i++, n++) { byte_rd = readb(ioaddr + n); seq_printf(m, "%02x ", byte_rd); } + + rtnl_unlock(); } + rtnl_lock(); + max = 0xB00; for (n = 0xA00; n < max;) { seq_printf(m, "\n0x%04x:\t", n); @@ -1220,20 +1224,20 @@ static int proc_get_all_registers(struct seq_file *m, void *v) seq_puts(m, "\nDump All MAC Registers\n"); seq_puts(m, "Offset\tValue\n------\t-----\n"); - rtnl_lock(); - max = pci_resource_len(pdev, 2); for (n = 0; n < max;) { seq_printf(m, "\n0x%04x:\t", n); + rtnl_lock(); + for (i = 0; i < 16 && n < max; i++, n++) { byte_rd = readb(ioaddr + n); seq_printf(m, "%02x ", byte_rd); } - } - rtnl_unlock(); + rtnl_unlock(); + } seq_printf(m, "\nTotal length:0x%X", max); @@ -2073,21 +2077,25 @@ static int proc_get_registers(char *page, char **start, "\nDump MAC Registers\n" "Offset\tValue\n------\t-----\n"); - rtnl_lock(); - for (n = 0; n < max;) { len += snprintf(page + len, count - len, "\n0x%04x:\t", n); + rtnl_lock(); + for (i = 0; i < 16 && n < max; i++, n++) { byte_rd = readb(ioaddr + n); len += snprintf(page + len, count - len, "%02x ", byte_rd); } + + rtnl_unlock(); } + rtnl_lock(); + max = 0xB00; for (n = 0xA00; n < max;) { len += snprintf(page + len, count - len, @@ -2154,8 +2162,6 @@ static int proc_get_all_registers(char *page, char **start, "\nDump All MAC Registers\n" "Offset\tValue\n------\t-----\n"); - rtnl_lock(); - max = pci_resource_len(pdev, 2); for (n = 0; n < max;) { @@ -2163,15 +2169,17 @@ static int proc_get_all_registers(char *page, char **start, "\n0x%04x:\t", n); + rtnl_lock(); + for (i = 0; i < 16 && n < max; i++, n++) { byte_rd = readb(ioaddr + n); len += snprintf(page + len, count - len, "%02x ", byte_rd); } - } - rtnl_unlock(); + rtnl_unlock(); + } len += snprintf(page + len, count - len, "\nTotal length:0x%X", max); From 50785cb7f032da6e58f05bc36968f263289a714f Mon Sep 17 00:00:00 2001 From: ChunHao Lin Date: Tue, 28 Oct 2025 23:01:21 +0800 Subject: [PATCH 089/464] NVIDIA: SAUCE: r8127: add support for RTL8127 cable diagnostic test BugLink: https://bugs.launchpad.net/bugs/2130445 Use following command to do the test. cat /proc/net/r8127//test/cdt Signed-off-by: ChunHao Lin Signed-off-by: Abhishek Sahu Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Ian May Acked-by: Noah Wager Acked-by: Abdur Rahman Signed-off--by: Brad Figg (cherry picked from commit aaf12caf97f50f3c310db3d9a2f321e93d8ac62d noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/r8127_n.c | 45 +++++++++----------- 1 file changed, 21 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/realtek/r8127/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c index 4ec33b3d77d5f..2c9c262abe2ea 100755 --- a/drivers/net/ethernet/realtek/r8127/r8127_n.c +++ b/drivers/net/ethernet/realtek/r8127/r8127_n.c @@ -690,6 +690,12 @@ rtl8127_get_sw_tail_ptr(struct rtl8127_tx_ring *ring) } } +static u32 +rtl8127_get_phy_status(struct rtl8127_private *tp) +{ + return RTL_R32(tp, PHYstatus); +} + static bool rtl8127_sysfs_testmode_on(struct rtl8127_private *tp) { @@ -795,20 +801,19 @@ static void rtl8127_get_cp_len(struct rtl8127_private *tp, int cp_len[RTL8127_CP_NUM]) { int i; - u16 status; + u32 status; int tmp_cp_len; - status = RTL_R16(tp, PHYstatus); + status = rtl8127_get_phy_status(tp); if (status & LinkStatus) { if (status & _10bps) { tmp_cp_len = -1; } else if (status & (_100bps | _1000bpsF)) { - rtl8127_mdio_write(tp, 0x1f, 0x0a88); - tmp_cp_len = rtl8127_mdio_read(tp, 0x10); - } else if (status & _2500bpsF) { - rtl8127_mdio_write(tp, 0x1f, 0x0acb); - tmp_cp_len = rtl8127_mdio_read(tp, 0x15); - tmp_cp_len >>= 2; + tmp_cp_len = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA880);; + } else if (status & (_10000bpsF | _10000bpsL | _5000bpsF | + _5000bpsL | _2500bpsF | _2500bpsL)) { + tmp_cp_len = rtl8127_mdio_direct_read_phy_ocp(tp, 0xAC2E);; + tmp_cp_len >>= 5; } else tmp_cp_len = 0; } else @@ -834,12 +839,11 @@ static int __rtl8127_get_cp_status(u16 val) case 0x0060: return rtl8127_cp_normal; case 0x0048: + case 0x0042: return rtl8127_cp_open; case 0x0050: - return rtl8127_cp_short; - case 0x0042: case 0x0044: - return rtl8127_cp_mismatch; + return rtl8127_cp_short; default: return rtl8127_cp_normal; } @@ -853,7 +857,7 @@ static int _rtl8127_get_cp_status(struct rtl8127_private *tp, u8 pair_num) if (pair_num > 3) goto exit; - rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8027 + 4 * pair_num); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8026 + 4 * pair_num); val = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA438); cp_status = __rtl8127_get_cp_status(val); @@ -885,7 +889,7 @@ static u16 rtl8127_get_cp_pp(struct rtl8127_private *tp, u8 pair_num) if (pair_num > 3) goto exit; - rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8029 + 4 * pair_num); + rtl8127_mdio_direct_write_phy_ocp(tp, 0xA436, 0x8028 + 4 * pair_num); pp = rtl8127_mdio_direct_read_phy_ocp(tp, 0xA438); pp &= 0x3fff; @@ -899,10 +903,10 @@ static void rtl8127_get_cp_status(struct rtl8127_private *tp, int cp_status[RTL8127_CP_NUM], bool poe_mode) { - u16 status; + u32 status; int i; - status = RTL_R16(tp, PHYstatus); + status = rtl8127_get_phy_status(tp); if (status & LinkStatus && !(status & (_10bps | _100bps))) { for (i=0; iprivate; @@ -1452,12 +1456,6 @@ static int _proc_get_cable_info(struct seq_file *m, void *v, bool poe_mode) const char *pair_str[RTL8127_CP_NUM] = {"1-2", "3-6", "4-5", "7-8"}; int ret; - switch (tp->mcfg) { - default: - ret = -EOPNOTSUPP; - goto error_out; - } - rtnl_lock(); if (!rtl8127_sysfs_testmode_on(tp)) { @@ -1474,7 +1472,7 @@ static int _proc_get_cable_info(struct seq_file *m, void *v, bool poe_mode) netif_testing_on(dev); - status = RTL_R16(tp, PHYstatus); + status = rtl8127_get_phy_status(tp); if (status & LinkStatus) seq_printf(m, "\nlink speed:%d", rtl8127_convert_link_speed(status)); @@ -1511,7 +1509,6 @@ static int _proc_get_cable_info(struct seq_file *m, void *v, bool poe_mode) error_unlock: rtnl_unlock(); -error_out: return ret; } From 7e9b57354cbda51624f499eac70a5c358b771fd3 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 20 Nov 2025 08:10:15 -0800 Subject: [PATCH 090/464] NVIDIA: SAUCE: iommu/arm-smmu-v3: Add two more DGX Spark iGPU IDs for existing iommu quirk BugLink: https://bugs.launchpad.net/bugs/2132033 Add two more device IDs for the existing Spark iommu quirk. Link: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.14/+bug/2132033 Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off--by: Brad Figg (cherry picked from commit ba9315458447d2677fb83131d78e1520f51b6c8a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index eab18dbd6191a..3cf4f515753d1 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3738,7 +3738,9 @@ static int arm_smmu_def_domain_type(struct device *dev) if (IS_HISI_PTT_DEVICE(pdev)) return IOMMU_DOMAIN_IDENTITY; - if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && pdev->device == 0x2E12) + if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && + (pdev->device == 0x2E12 || pdev->device == 0x2E2A || + pdev->device == 0x2E2B)) return IOMMU_DOMAIN_DMA; } From 30c9b22fb20f43973a8ec1c3c32618e019249544 Mon Sep 17 00:00:00 2001 From: ChunHao Lin Date: Thu, 4 Dec 2025 16:51:16 +0800 Subject: [PATCH 091/464] NVIDIA: SAUCE: r8127: Remove registers2 proc entry BugLink: https://bugs.launchpad.net/bugs/2134991 Remove registers2 proc entry as it is causing system crash on running opensource LTP test suite. Change-Id: I47846bca0401d4403fba026d4a348eef3d454f80 Signed-off-by: ChunHao Lin Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Abdur Rahman Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 4ff6f9ff6979cd66f18f8c620fc966971fb48470 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/r8127_n.c | 77 -------------------- 1 file changed, 77 deletions(-) diff --git a/drivers/net/ethernet/realtek/r8127/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c index 2c9c262abe2ea..496fec1320d12 100755 --- a/drivers/net/ethernet/realtek/r8127/r8127_n.c +++ b/drivers/net/ethernet/realtek/r8127/r8127_n.c @@ -1216,39 +1216,6 @@ static int proc_get_registers(struct seq_file *m, void *v) return 0; } -static int proc_get_all_registers(struct seq_file *m, void *v) -{ - struct net_device *dev = m->private; - int i, n, max; - u8 byte_rd; - struct rtl8127_private *tp = netdev_priv(dev); - void __iomem *ioaddr = tp->mmio_addr; - struct pci_dev *pdev = tp->pci_dev; - - seq_puts(m, "\nDump All MAC Registers\n"); - seq_puts(m, "Offset\tValue\n------\t-----\n"); - - max = pci_resource_len(pdev, 2); - - for (n = 0; n < max;) { - seq_printf(m, "\n0x%04x:\t", n); - - rtnl_lock(); - - for (i = 0; i < 16 && n < max; i++, n++) { - byte_rd = readb(ioaddr + n); - seq_printf(m, "%02x ", byte_rd); - } - - rtnl_unlock(); - } - - seq_printf(m, "\nTotal length:0x%X", max); - - seq_putc(m, '\n'); - return 0; -} - static int proc_get_pcie_phy(struct seq_file *m, void *v) { struct net_device *dev = m->private; @@ -2143,49 +2110,6 @@ static int proc_get_registers(char *page, char **start, return len; } -static int proc_get_all_registers(char *page, char **start, - off_t offset, int count, - int *eof, void *data) -{ - struct net_device *dev = data; - int i, n, max; - u8 byte_rd; - struct rtl8127_private *tp = netdev_priv(dev); - void __iomem *ioaddr = tp->mmio_addr; - struct pci_dev *pdev = tp->pci_dev; - int len = 0; - - len += snprintf(page + len, count - len, - "\nDump All MAC Registers\n" - "Offset\tValue\n------\t-----\n"); - - max = pci_resource_len(pdev, 2); - - for (n = 0; n < max;) { - len += snprintf(page + len, count - len, - "\n0x%04x:\t", - n); - - rtnl_lock(); - - for (i = 0; i < 16 && n < max; i++, n++) { - byte_rd = readb(ioaddr + n); - len += snprintf(page + len, count - len, - "%02x ", - byte_rd); - } - - rtnl_unlock(); - } - - len += snprintf(page + len, count - len, "\nTotal length:0x%X", max); - - len += snprintf(page + len, count - len, "\n"); - - *eof = 1; - return len; -} - static int proc_get_pcie_phy(char *page, char **start, off_t offset, int count, int *eof, void *data) @@ -2784,7 +2708,6 @@ static const struct rtl8127_proc_file rtl8127_debug_proc_files[] = { { "driver_var", &proc_get_driver_variable }, { "tally", &proc_get_tally_counter }, { "registers", &proc_get_registers }, - { "registers2", &proc_get_all_registers }, { "pcie_phy", &proc_get_pcie_phy }, { "eth_phy", &proc_get_eth_phy }, { "ext_regs", &proc_get_extended_registers }, From 2a9f1e1a4d1c4c0c4b66e78a425485948cd72c58 Mon Sep 17 00:00:00 2001 From: Surabhi Chythanya Kumar Date: Thu, 8 Jan 2026 18:13:30 -0800 Subject: [PATCH 092/464] NVIDIA: SAUCE: MEDIATEK: platform: Add PCIe Hotplug Driver for CX7 on DGX Spark BugLink: https://bugs.launchpad.net/bugs/2138269 This driver manages PCIe link for NVIDIA ConnectX-7 (CX7) hot-plug/unplug on DGX Spark systems with GB10 SoC. It disables the PCIe link on cable removal and enables it on cable insertion. Upstream-friendly improvements over 6.14 driver: - Separated from MTK pinctrl driver into NVIDIA platform driver - Configuration via ACPI (_CRS and _DSD), no hardcoded values - Device-managed resources (devm_*) for automatic cleanup - Thread-safe state management with locking - Enhanced error handling and logging - Uses standard Linux kernel APIs The driver exposes a sysfs interface to emulate cable plug in/out: echo 1 > /sys/devices/platform/MTKP0001:00/pcie_hotplug/debug_state # plug in echo 0 > /sys/devices/platform/MTKP0001:00/pcie_hotplug/debug_state # plug out It also provides a runtime enable/disable switch via sysfs: echo 1 > /sys/devices/platform/MTKP0001:00/pcie_hotplug/hotplug_enabled # Enable echo 0 > /sys/devices/platform/MTKP0001:00/pcie_hotplug/hotplug_enabled # Disable This allows enabling/disabling hotplug functionality. Hotplug is disabled by default and must be explicitly enabled via userspace. It also implements uevent notifications for coordination with userspace: * cable plug-in: Report plug-in uevent (driver) Enable PCIe link (driver) Rescan CX7 devices (application) * cable removal: Report removal uevent (driver) Remove CX7 devices (application) Disable PCIe link (driver) Signed-off-by: Vaibhav Vyas Signed-off-by: Scott Fudally Signed-off-by: Surabhi Chythanya Kumar Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit 4894eb0dfc2743560465531f17e9070f6cf52017 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 + drivers/platform/arm64/Kconfig | 2 + drivers/platform/arm64/Makefile | 1 + drivers/platform/arm64/nvidia/Kconfig | 17 + drivers/platform/arm64/nvidia/Makefile | 9 + .../platform/arm64/nvidia/mtk-pcie-hotplug.c | 2324 +++++++++++++++++ 6 files changed, 2356 insertions(+) create mode 100644 drivers/platform/arm64/nvidia/Kconfig create mode 100644 drivers/platform/arm64/nvidia/Makefile create mode 100644 drivers/platform/arm64/nvidia/mtk-pcie-hotplug.c diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index e9093504c2ddd..4d07edf8ec02a 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -150,6 +150,9 @@ CONFIG_MFD_AAEON note<'Disable all Ubuntu ODM dri CONFIG_MTD policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_MTD note<'Essential for boot on ARM64'> +CONFIG_MTK_PCIE_HOTPLUG policy<{'arm64': 'm'}> +CONFIG_MTK_PCIE_HOTPLUG note<'CX7 PCIe hotplug driver for NVIDIA DGX Spark systems with GB10 SoC.'> + CONFIG_NOUVEAU_DEBUG policy<{'amd64': '-', 'arm64': '-'}> CONFIG_NOUVEAU_DEBUG note<'Disable nouveau for NVIDIA kernels'> diff --git a/drivers/platform/arm64/Kconfig b/drivers/platform/arm64/Kconfig index 80cefd5772cec..4bbd8eca38a87 100644 --- a/drivers/platform/arm64/Kconfig +++ b/drivers/platform/arm64/Kconfig @@ -115,4 +115,6 @@ config NVIDIA_FFA_EC Say M or Y here to include this support. +source "drivers/platform/arm64/nvidia/Kconfig" + endif # ARM64_PLATFORM_DEVICES diff --git a/drivers/platform/arm64/Makefile b/drivers/platform/arm64/Makefile index c693a0501631b..baa8d477a2cd7 100644 --- a/drivers/platform/arm64/Makefile +++ b/drivers/platform/arm64/Makefile @@ -11,3 +11,4 @@ obj-$(CONFIG_EC_LENOVO_YOGA_C630) += lenovo-yoga-c630.o obj-$(CONFIG_EC_LENOVO_THINKPAD_T14S) += lenovo-thinkpad-t14s.o obj-$(CONFIG_EC_LENOVO_YOGA_SLIM7X) += lenovo-yoga-slim7x.o obj-$(CONFIG_NVIDIA_FFA_EC) += nvidia-ffa-ec.o +obj-y += nvidia/ diff --git a/drivers/platform/arm64/nvidia/Kconfig b/drivers/platform/arm64/nvidia/Kconfig new file mode 100644 index 0000000000000..b12b290f30d4f --- /dev/null +++ b/drivers/platform/arm64/nvidia/Kconfig @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# NVIDIA ARM64 Platform-Specific Device Drivers +# + +config MTK_PCIE_HOTPLUG + tristate "CX7 PCIe Hotplug Driver" + depends on EINT_MTK + depends on PCI && ACPI + help + Say Y here to support PCIe device plug in/out detection. + It will disable PCIe link when plug out and enable + PCIe link after plug in. + + This is particularly useful for GB10 SoC. + + If unsure, say N. diff --git a/drivers/platform/arm64/nvidia/Makefile b/drivers/platform/arm64/nvidia/Makefile new file mode 100644 index 0000000000000..37cfbebb8d1af --- /dev/null +++ b/drivers/platform/arm64/nvidia/Makefile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Makefile for NVIDIA ARM64 platform-specific drivers +# +# CX7 PCIe Hotplug Driver +# Provides hotplug support for CX7 PCIe devices on GB10 SoC-based systems +# + +obj-$(CONFIG_MTK_PCIE_HOTPLUG) += mtk-pcie-hotplug.o diff --git a/drivers/platform/arm64/nvidia/mtk-pcie-hotplug.c b/drivers/platform/arm64/nvidia/mtk-pcie-hotplug.c new file mode 100644 index 0000000000000..06a84a29aa6fd --- /dev/null +++ b/drivers/platform/arm64/nvidia/mtk-pcie-hotplug.c @@ -0,0 +1,2324 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2014-2025 MediaTek Inc. + * Copyright (c) 2025-2026 NVIDIA Corporation + * + * CX7 PCIe Hotplug Driver + * + * Manages PCIe device hotplug using GPIO interrupts and ACPI resources. + * Supports cable insertion/removal detection and device power management. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define HP_PORT_MAX 3 +#define HP_POLL_CNT_MAX 200 +#define MAX_VENDOR_DATA_LEN 16 +#define CX7_HP_MMIO_REGION_COUNT 5 /* TOP, PROTECT, CKM, MAC Port 0, MAC Port 1 */ +#define CX7_HP_MIN_GPIO_COUNT 4 /* Minimum required: BOOT, PRSNT, PERST, EN */ +#define PINCTRL_MAPPING_ENTRY_SIZE 5 /* dev_name, state, ctrl_dev, group, function */ +/* Indices for pinctrl mapping entry strings */ +#define PINCTRL_IDX_DEV_NAME 0 +#define PINCTRL_IDX_STATE 1 +#define PINCTRL_IDX_CTRL_DEV 2 +#define PINCTRL_IDX_GROUP 3 +#define PINCTRL_IDX_FUNCTION 4 + +/* Hardware timing requirements (in microseconds unless noted) */ +#define CX7_HP_DELAY_SHORT_US 10 /* Short delay for register writes */ +#define CX7_HP_DELAY_STANDARD_US 10000 /* Standard delay (10ms) */ +#define CX7_HP_DELAY_BUS_PROTECT_US 5000 /* Bus protection setup delay */ +#define CX7_HP_DELAY_PHY_RESET_US 3000 /* PHY reset delay */ +#define CX7_HP_DELAY_LINK_STABLE_MS 100 /* Link stabilization delay (ms) */ +#define CX7_HP_POLL_SLEEP_US 10000 /* Polling loop sleep interval */ + +#define PLUG_IN_EVT "HOTPLUG_STATE=plugin" +#define REMOVAL_EVT "HOTPLUG_STATE=removal" + +/* Bus protection stages to prevent PCIe core reset glitches */ +#define BUS_PROTECT_INIT 0 +#define BUS_PROTECT_CABLE_REMOVAL 1 +#define BUS_PROTECT_CABLE_PLUGIN 2 +#define BUS_PROTECT_CLEANUP 3 + +enum cx7_hp_state { + STATE_READY = 0, + STATE_PLUG_OUT, /* Cable plug-out */ + STATE_DEV_POWER_OFF, /* Device is powered off */ + STATE_PLUG_IN, /* Cable plug-in detected */ + STATE_DEV_POWER_ON, /* Device is powered on */ + STATE_DEV_FW_START, /* Device firmware is running */ + STATE_RESCAN, /* Device ready, can perform bus rescan */ + STATE_UNKNOWN +}; + +enum pcie_pin_index { + PCIE_PIN_BOOT = 0, /* Device boot status pin */ + PCIE_PIN_PRSNT, /* Presence detection pin */ + PCIE_PIN_PERST, /* PCIe reset pin */ + PCIE_PIN_EN, /* Power enable pin */ + PCIE_PIN_CLQ0, /* Clock request pin 0 */ + PCIE_PIN_CLQ1, /* Clock request pin 1 */ + PCIE_PIN_MAX +}; + +struct pcie_port_info { + int domain; + int bus; + int devfn; +}; + +struct rp_bus_mmio_top { + u32 ctrl; + u32 port_bits[HP_PORT_MAX]; + u32 update_bit; +}; + +struct rp_bus_mmio_protect { + u32 mode; + u32 enable; + u32 port_bits[HP_PORT_MAX]; +}; + +struct rp_bus_mmio_mac { + u32 init_ctrl; + u32 ltssm_bit; + u32 phy_rst_bit; +}; + +struct rp_bus_mmio_ckm { + u32 ctrl; + u32 disable_bit; +}; + +struct rp_bus_mmio_info { + struct rp_bus_mmio_top top; + struct rp_bus_mmio_protect protect; + struct rp_bus_mmio_mac mac; + struct rp_bus_mmio_ckm ckm; +}; + +struct gpio_acpi_context { + struct device *dev; + unsigned int debounce_timeout_us; + int pin; + int wake_capable; + int triggering; + int polarity; + unsigned long irq_flags; + int valid; + unsigned int connection_type; + char vendor_data[MAX_VENDOR_DATA_LEN + 1]; +}; + +struct cx7_hp_dev; + +/** + * struct cx7_hp_plat_data - Platform configuration data parsed from ACPI + * + * Platform-specific configuration parsed from ACPI devices: + * - RES0 device (PNP0C02): PCIe configuration and MMIO register offsets via _DSD + * - PEDE device (MTKP0001): Pinctrl mappings via _DSD + */ +struct cx7_hp_plat_data { + int port_nums; + struct pcie_port_info ports[HP_PORT_MAX]; + u32 vendor_id; + u32 device_id; + int num_devices; + struct rp_bus_mmio_info rp_bus_mmio; + u32 ltssm_reg; + u32 ltssm_l0_state; + int pin_nums; + struct pinctrl_map *parsed_pinmap; +}; + +struct cx7_hp_gpio_ctx { + struct gpio_desc *desc; + struct gpio_acpi_context *ctx; + struct cx7_hp_dev *hp_dev; +}; + +struct acpi_gpio_parse_context { + struct gpio_acpi_context *ctx; + struct cx7_hp_dev *hp_dev; +}; + +struct acpi_gpio_walk_context { + struct device *dev; + struct gpio_info { + unsigned int pin; + unsigned int connection_type; + unsigned int triggering; + unsigned int polarity; + unsigned int debounce_timeout; + unsigned int wake_capable; + char vendor_data[MAX_VENDOR_DATA_LEN + 1]; + char resource_source[16]; + unsigned int resource_source_index; + } gpios[PCIE_PIN_MAX]; + int count; +}; + +struct cx7_hp_acpi_mmio { + struct acpi_resource_fixed_memory32 + mmio_regions[CX7_HP_MMIO_REGION_COUNT]; + int count; + struct device *dev; +}; + +enum cx7_hp_debug_val { + CX7_HP_DEBUG_PLUG_OUT = 0, + CX7_HP_DEBUG_PLUG_IN, + CX7_HP_DEBUG_MAX_VAL +}; + +struct cx7_hp_mmio_runtime { + void __iomem *top_base; + void __iomem *protect_base; + void __iomem *ckm_base; + void __iomem *mac_port_base[HP_PORT_MAX]; +}; + +/** + * cx7_hp_dev - Hotplug device structure + * + * ACPI resource sources: + * - MMIO addresses: RES0 device (PNP0C02) _CRS, stored in mmio field + * - GPIO resources: PEDE device (MTKP0001) _CRS, stored in pins field + */ +struct cx7_hp_dev { + struct cx7_hp_gpio_ctx *pins; + struct cx7_hp_plat_data *pd; + struct platform_device *pdev; + enum cx7_hp_state state; + int gpio_count; + int boot_pin; + int prsnt_pin; + enum cx7_hp_debug_val debug_state; + bool hotplug_enabled; + spinlock_t lock; + struct pci_dev *cached_root_ports[HP_PORT_MAX]; + struct cx7_hp_mmio_runtime mmio; + struct gpio_device *gdev; + struct notifier_block pci_notifier; +}; + +/* ACPI _DSD device properties GUID: daffd814-6eba-4d8c-8a91-bc9bbf4aa301 */ +static const guid_t device_properties_guid = +GUID_INIT(0xdaffd814, 0x6eba, 0x4d8c, + 0x8a, 0x91, 0xbc, 0x9b, + 0xbf, 0x4a, 0xa3, 0x01); + +/** + * cx7_hp_parse_pinctrl_config_dsd - Parse pinctrl configuration from PEDE device _DSD + * @hp_dev: hotplug device + * + * Parses pin-nums and pinctrl-mappings from _DSD. + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_parse_pinctrl_config_dsd(struct cx7_hp_dev *hp_dev) +{ + struct acpi_device *adev; + struct device *dev = &hp_dev->pdev->dev; + const union acpi_object *mappings_pkg, *mapping_entry; + struct pinctrl_map *pinmap; + u32 pin_nums = 0; + int k; + const char *strings[PINCTRL_MAPPING_ENTRY_SIZE]; + + adev = ACPI_COMPANION(dev); + if (!adev) { + dev_err(dev, "Failed to get ACPI companion device\n"); + return -ENODEV; + } + + struct acpi_buffer buffer = { ACPI_ALLOCATE_BUFFER, NULL }; + acpi_status status; + const union acpi_object *dsd_pkg, *props_pkg = NULL; + int i, j; + + status = acpi_evaluate_object_typed(adev->handle, "_DSD", NULL, &buffer, + ACPI_TYPE_PACKAGE); + if (ACPI_FAILURE(status)) { + dev_err(dev, "Failed to evaluate _DSD: %s\n", + acpi_format_exception(status)); + return -ENODEV; + } + + dsd_pkg = buffer.pointer; + if (!dsd_pkg || dsd_pkg->type != ACPI_TYPE_PACKAGE) { + dev_err(dev, "Invalid _DSD package\n"); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + /* Find Device Properties GUID package */ + for (i = 0; i + 1 < dsd_pkg->package.count; i += 2) { + const union acpi_object *guid = &dsd_pkg->package.elements[i]; + const union acpi_object *pkg = + &dsd_pkg->package.elements[i + 1]; + + /* Verify GUID matches Device Properties GUID */ + if (guid->type == ACPI_TYPE_BUFFER && guid->buffer.length == 16 && + pkg->type == ACPI_TYPE_PACKAGE && + guid_equal((guid_t *)guid->buffer.pointer, + &device_properties_guid)) { + props_pkg = pkg; + break; + } + } + + if (!props_pkg) { + dev_err(dev, + "Device Properties GUID package not found in _DSD\n"); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + + for (j = 0; j < props_pkg->package.count; j++) { + const union acpi_object *prop = &props_pkg->package.elements[j]; + + if (prop->type != ACPI_TYPE_PACKAGE || + prop->package.count != 2 || + prop->package.elements[0].type != ACPI_TYPE_STRING) + continue; + + const char *prop_name = + prop->package.elements[0].string.pointer; + const union acpi_object *prop_value = + &prop->package.elements[1]; + + if (!strcmp(prop_name, "pin-nums")) { + if (prop_value->type == ACPI_TYPE_INTEGER) { + pin_nums = prop_value->integer.value; + } + } else if (!strcmp(prop_name, "pinctrl-mappings")) { + if (prop_value->type == ACPI_TYPE_PACKAGE) + mappings_pkg = prop_value; + } + } + + if (pin_nums == 0) { + hp_dev->pd->pin_nums = 0; + ACPI_FREE(buffer.pointer); + return 0; + } + + if (!mappings_pkg) { + dev_err(dev, + "Missing required _DSD property: pinctrl-mappings\n"); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + + if (mappings_pkg->package.count != pin_nums) { + dev_err(dev, + "pinctrl-mappings count mismatch: expected %u, got %u\n", + pin_nums, mappings_pkg->package.count); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + + /* Allocate pinmap array */ + pinmap = devm_kcalloc(dev, pin_nums, sizeof(*pinmap), GFP_KERNEL); + if (!pinmap) { + ACPI_FREE(buffer.pointer); + return -ENOMEM; + } + + /* Parse each mapping entry */ + for (k = 0; k < pin_nums; k++) { + mapping_entry = &mappings_pkg->package.elements[k]; + if (mapping_entry->type != ACPI_TYPE_PACKAGE || + mapping_entry->package.count != ARRAY_SIZE(strings)) { + dev_err(dev, + "Invalid pinctrl mapping entry %d: expected Package(%zu), " + "got %s(count=%u)\n", + k, ARRAY_SIZE(strings), + mapping_entry->type == ACPI_TYPE_PACKAGE ? + "Package" : "non-Package", + mapping_entry->type == ACPI_TYPE_PACKAGE ? + mapping_entry->package.count : 0); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + + /* Extract strings: dev_name, state, ctrl_dev, group, function */ + for (int l = 0; l < ARRAY_SIZE(strings); l++) { + if (mapping_entry->package.elements[l].type != + ACPI_TYPE_STRING) { + dev_err(dev, + "Mapping entry %d element %d is not a string\n", + k, l); + ACPI_FREE(buffer.pointer); + return -EINVAL; + } + strings[l] = + mapping_entry->package.elements[l].string.pointer; + } + + /* Populate pinctrl_map structure */ + pinmap[k].dev_name = + devm_kstrdup(dev, strings[PINCTRL_IDX_DEV_NAME], + GFP_KERNEL); + pinmap[k].name = + devm_kstrdup(dev, strings[PINCTRL_IDX_STATE], GFP_KERNEL); + pinmap[k].type = PIN_MAP_TYPE_MUX_GROUP; + pinmap[k].ctrl_dev_name = + devm_kstrdup(dev, strings[PINCTRL_IDX_CTRL_DEV], + GFP_KERNEL); + pinmap[k].data.mux.group = + devm_kstrdup(dev, strings[PINCTRL_IDX_GROUP], GFP_KERNEL); + pinmap[k].data.mux.function = + devm_kstrdup(dev, strings[PINCTRL_IDX_FUNCTION], + GFP_KERNEL); + + if (!pinmap[k].dev_name || !pinmap[k].name || + !pinmap[k].ctrl_dev_name || !pinmap[k].data.mux.group || + !pinmap[k].data.mux.function) { + dev_err(dev, + "Failed to allocate memory for mapping %d\n", + k); + ACPI_FREE(buffer.pointer); + return -ENOMEM; + } + } + + hp_dev->pd->pin_nums = pin_nums; + hp_dev->pd->parsed_pinmap = pinmap; + ACPI_FREE(buffer.pointer); + dev_dbg(dev, "Successfully parsed %u pinctrl mappings from ACPI\n", + pin_nums); + return 0; +} + +/** + * cx7_hp_pinctrl_init - Register pinctrl mappings for the device + * @hp_dev: hotplug device + * + * Parses pinctrl mappings from _DSD and registers them. + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_pinctrl_init(struct cx7_hp_dev *hp_dev) +{ + int ret; + + ret = cx7_hp_parse_pinctrl_config_dsd(hp_dev); + if (ret) { + dev_err(&hp_dev->pdev->dev, + "Failed to parse pinctrl configuration from ACPI: %d\n", + ret); + return ret; + } + + if (!hp_dev->pd->pin_nums) + return 0; + + ret = + pinctrl_register_mappings(hp_dev->pd->parsed_pinmap, + hp_dev->pd->pin_nums); + if (ret) { + dev_err(&hp_dev->pdev->dev, + "Failed to register pinctrl mappings\n"); + return ret; + } + + dev_dbg(&hp_dev->pdev->dev, "Registered %u pinctrl mappings\n", + hp_dev->pd->pin_nums); + return 0; +} + +/** + * cx7_hp_pinctrl_remove - Unregister pinctrl mappings + * @hp_dev: hotplug device + */ +static void cx7_hp_pinctrl_remove(struct cx7_hp_dev *hp_dev) +{ + if (!hp_dev->pd->pin_nums) + return; + + pinctrl_unregister_mappings(hp_dev->pd->parsed_pinmap); +} + +/** + * cx7_hp_change_pinctrl_state - Change pinctrl state + * @hp_dev: hotplug device + * @new_state: new pinctrl state name + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_change_pinctrl_state(struct cx7_hp_dev *hp_dev, + const char *new_state) +{ + struct pinctrl *pinctrl; + struct pinctrl_state *state; + int ret; + + pinctrl = devm_pinctrl_get(&hp_dev->pdev->dev); + if (IS_ERR(pinctrl)) { + dev_err(&hp_dev->pdev->dev, "Failed to get pinctrl\n"); + return PTR_ERR(pinctrl); + } + + state = pinctrl_lookup_state(pinctrl, new_state); + if (IS_ERR(state)) { + dev_err(&hp_dev->pdev->dev, "Failed to lookup state:%s\n", + new_state); + return PTR_ERR(state); + } + + ret = pinctrl_select_state(pinctrl, state); + if (ret) { + dev_err(&hp_dev->pdev->dev, + "Failed to select pinctrl state:%s\n", new_state); + return ret; + } + + return 0; +} + +/** + * cx7_hp_send_uevent - Send uevent to userspace + * @hp_dev: hotplug device + * @msg: uevent message string + */ +static void cx7_hp_send_uevent(struct cx7_hp_dev *hp_dev, const char *msg) +{ + char *uevent = NULL; + char *envp[2]; + + uevent = kasprintf(GFP_KERNEL, msg); + if (!uevent) { + dev_err(&hp_dev->pdev->dev, + "Failed to allocate uevent string\n"); + return; + } + + envp[0] = uevent; + envp[1] = NULL; + + if (kobject_uevent_env(&hp_dev->pdev->dev.kobj, KOBJ_CHANGE, envp)) + dev_err(&hp_dev->pdev->dev, "Failed to send uevent\n"); + + kfree(uevent); +} + +/** + * cx7_hp_reg_update_bits - Update specific bits in a register + * @base: MMIO base address + * @offset: Register offset + * @mask: Bits to modify + * @set: true to set bits, false to clear bits + */ +static inline void cx7_hp_reg_update_bits(void __iomem *base, u32 offset, + u32 mask, bool set) +{ + u32 val = readl(base + offset); + + if (set) + val |= mask; + else + val &= ~mask; + + writel(val, base + offset); +} + +/** + * cx7_hp_toggle_update_bit - Toggle control register update bit + * @base: MMIO base address + * @ctrl_offset: Control register offset + * @bits: Bits to set/clear before toggling update + * @update_bit: Update bit mask + * @set: true to set bits, false to clear bits + * + * Performs the sequence: modify bits, clear update bit, set update bit + */ +static void cx7_hp_toggle_update_bit(void __iomem *base, u32 ctrl_offset, + u32 bits, u32 update_bit, bool set) +{ + cx7_hp_reg_update_bits(base, ctrl_offset, bits, set); + cx7_hp_reg_update_bits(base, ctrl_offset, update_bit, false); + cx7_hp_reg_update_bits(base, ctrl_offset, update_bit, true); +} + +/** + * cx7_hp_bus_protect_enable - Enable bus protection for a port + * @dev: hotplug device + * @port_idx: Port index + */ +static void cx7_hp_bus_protect_enable(struct cx7_hp_dev *dev, int port_idx) +{ + struct rp_bus_mmio_info *mmio_info = &dev->pd->rp_bus_mmio; + u32 port_bit = mmio_info->protect.port_bits[port_idx]; + + cx7_hp_reg_update_bits(dev->mmio.protect_base, + mmio_info->protect.mode, port_bit, true); + cx7_hp_reg_update_bits(dev->mmio.protect_base, + mmio_info->protect.enable, port_bit, true); +} + +/** + * cx7_hp_bus_protect_disable - Disable bus protection for a port + * @dev: hotplug device + * @port_idx: Port index + */ +static void cx7_hp_bus_protect_disable(struct cx7_hp_dev *dev, int port_idx) +{ + struct rp_bus_mmio_info *mmio_info = &dev->pd->rp_bus_mmio; + u32 port_bit = mmio_info->protect.port_bits[port_idx]; + + cx7_hp_reg_update_bits(dev->mmio.protect_base, + mmio_info->protect.enable, port_bit, false); + cx7_hp_reg_update_bits(dev->mmio.protect_base, + mmio_info->protect.mode, port_bit, false); +} + +/** + * cx7_hp_ckm_control - Control clock module + * @dev: hotplug device + * @disable: true to disable clock, false to enable + */ +static void cx7_hp_ckm_control(struct cx7_hp_dev *dev, bool disable) +{ + struct rp_bus_mmio_info *mmio_info = &dev->pd->rp_bus_mmio; + + if (!dev->mmio.ckm_base) + return; + + cx7_hp_reg_update_bits(dev->mmio.ckm_base, mmio_info->ckm.ctrl, + mmio_info->ckm.disable_bit, disable); +} + +/** + * cx7_hp_parse_mmio_resources - ACPI resource callback for parsing MMIO from _CRS + * @ares: ACPI resource being processed + * @data: pointer to cx7_hp_acpi_mmio structure + * + * Returns: AE_OK to continue iteration, AE_ERROR on error + */ +static acpi_status cx7_hp_parse_mmio_resources(struct acpi_resource *ares, + void *data) +{ + struct cx7_hp_acpi_mmio *parsed = data; + + switch (ares->type) { + case ACPI_RESOURCE_TYPE_FIXED_MEMORY32: + if (parsed->count >= CX7_HP_MMIO_REGION_COUNT) { + dev_warn(parsed->dev, + "More than %d MMIO regions found in platform configuration device, ignoring extras\n", + CX7_HP_MMIO_REGION_COUNT); + break; + } + parsed->mmio_regions[parsed->count] = ares->data.fixed_memory32; + parsed->count++; + break; + default: + break; + } + + return AE_OK; +} + +/** + * cx7_hp_find_pcie_config_device - Find PCIe configuration device by HID + * + * Finds the ACPI device that provides PCIe configuration via _DSD properties + * and MMIO resources via _CRS. + * + * Returns: acpi_device pointer on success (with reference), NULL on failure + */ +static struct acpi_device *cx7_hp_find_pcie_config_device(void) +{ + return acpi_dev_get_first_match_dev("PNP0C02", NULL, -1); +} + +/** + * cx7_hp_parse_pcie_config_dsd - Parse PCIe configuration from _DSD + * @pdev: platform device + * @pd: platform data to populate + * + * Parses PCIe MMIO register offsets, bit positions, port configuration, and PCIe device + * identification from PCIe configuration device _DSD. + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_parse_pcie_config_dsd(struct platform_device *pdev, + struct cx7_hp_plat_data *pd) +{ + struct acpi_device *config_adev; + struct device *dev = &pdev->dev; + u32 val, bit1; + + config_adev = cx7_hp_find_pcie_config_device(); + if (!config_adev) { + dev_err(dev, + "Platform configuration device (PNP0C02) not found - _DSD is required\n"); + return -ENODEV; + } + + if (!acpi_dev_has_props(config_adev)) { + dev_err(dev, + "Platform configuration device has no _DSD properties. Check DSDT.\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "mac-init-ctrl-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: mac-init-ctrl-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.mac.init_ctrl = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "mac-ltssm-bit", &val)) { + dev_err(dev, "Missing required _DSD property: mac-ltssm-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.mac.ltssm_bit = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "mac-phy-rst-bit", &val)) { + dev_err(dev, + "Missing required _DSD property: mac-phy-rst-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.mac.phy_rst_bit = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "top-ctrl-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: top-ctrl-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.top.ctrl = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "top-update-bit", &val)) { + dev_err(dev, + "Missing required _DSD property: top-update-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.top.update_bit = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "top-port0-bit", &val)) { + dev_err(dev, "Missing required _DSD property: top-port0-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.top.port_bits[0] = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "top-port1-bit", &val)) { + dev_err(dev, "Missing required _DSD property: top-port1-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.top.port_bits[1] = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "protect-mode-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: protect-mode-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.protect.mode = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "protect-enable-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: protect-enable-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.protect.enable = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "protect-port0-bit", &val)) { + dev_err(dev, + "Missing required _DSD property: protect-port0-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.protect.port_bits[0] = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "protect-port1-bit", &val)) { + dev_err(dev, + "Missing required _DSD property: protect-port1-bit\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.protect.port_bits[1] = BIT(val); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "ckm-ctrl-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: ckm-ctrl-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.ckm.ctrl = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "ckm-disable-bit0", &val)) { + dev_err(dev, + "Missing required _DSD property: ckm-disable-bit0\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "ckm-disable-bit1", &bit1)) { + dev_err(dev, + "Missing required _DSD property: ckm-disable-bit1\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->rp_bus_mmio.ckm.disable_bit = BIT(val) | BIT(bit1); + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "ltssm-reg-offset", &val)) { + dev_err(dev, + "Missing required _DSD property: ltssm-reg-offset\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ltssm_reg = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "ltssm-l0-state", &val)) { + dev_err(dev, + "Missing required _DSD property: ltssm-l0-state\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ltssm_l0_state = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port-nums", &val)) { + dev_err(dev, "Missing required _DSD property: port-nums\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + if (val == 0 || val > HP_PORT_MAX) { + dev_err(dev, + "Invalid _DSD property port-nums: %u (must be 1-%d)\n", + val, HP_PORT_MAX); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->port_nums = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port0-domain", &val)) { + dev_err(dev, "Missing required _DSD property: port0-domain\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[0].domain = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port0-bus", &val)) { + dev_err(dev, "Missing required _DSD property: port0-bus\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[0].bus = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port0-devfn", &val)) { + dev_err(dev, "Missing required _DSD property: port0-devfn\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[0].devfn = val; + + if (pd->port_nums >= 2) { + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port1-domain", &val)) { + dev_err(dev, + "Missing required _DSD property: port1-domain\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[1].domain = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port1-bus", &val)) { + dev_err(dev, + "Missing required _DSD property: port1-bus\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[1].bus = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "port1-devfn", &val)) { + dev_err(dev, + "Missing required _DSD property: port1-devfn\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->ports[1].devfn = val; + } + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "vendor-id", &val)) { + dev_err(dev, "Missing required _DSD property: vendor-id\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->vendor_id = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "device-id", &val)) { + dev_err(dev, "Missing required _DSD property: device-id\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->device_id = val; + + if (fwnode_property_read_u32 + (acpi_fwnode_handle(config_adev), "num-devices", &val)) { + dev_err(dev, "Missing required _DSD property: num-devices\n"); + acpi_dev_put(config_adev); + return -EINVAL; + } + pd->num_devices = val; + + dev_dbg(dev, "Successfully parsed all required _DSD properties\n"); + + acpi_dev_put(config_adev); + return 0; +} + +/** + * cx7_hp_parse_mmio_resources_from_acpi - Parse MMIO regions from _CRS + * @dev: hotplug device + * @parsed: pointer to parsed MMIO structure + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_parse_mmio_resources_from_acpi(struct cx7_hp_dev *dev, + struct cx7_hp_acpi_mmio + *parsed) +{ + struct acpi_device *config_adev; + acpi_status status; + int ret = 0; + + if (!dev || !dev->pdev) { + return -EINVAL; + } + + config_adev = cx7_hp_find_pcie_config_device(); + if (!config_adev) + return -ENODEV; + + parsed->count = 0; + memset(parsed->mmio_regions, 0, sizeof(parsed->mmio_regions)); + + status = + acpi_walk_resources(config_adev->handle, METHOD_NAME__CRS, + cx7_hp_parse_mmio_resources, parsed); + if (ACPI_FAILURE(status)) { + dev_err(&dev->pdev->dev, + "Failed to walk platform configuration resources: %s\n", + acpi_format_exception(status)); + ret = -ENODEV; + goto out; + } + + if (parsed->count < CX7_HP_MMIO_REGION_COUNT) { + dev_warn(&dev->pdev->dev, + "Expected %d MMIO regions from platform configuration device, found %d\n", + CX7_HP_MMIO_REGION_COUNT, parsed->count); + ret = -ENODEV; + goto out; + } + +out: + acpi_dev_put(config_adev); + return ret; +} + +/** + * cx7_hp_map_mmio_resources - Map all MMIO regions from ACPI _CRS + * @dev: hotplug device + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_map_mmio_resources(struct cx7_hp_dev *dev) +{ + struct platform_device *pdev = dev->pdev; + struct cx7_hp_acpi_mmio parsed = {.count = 0, .dev = &pdev->dev }; + int ret; + int i; + + ret = cx7_hp_parse_mmio_resources_from_acpi(dev, &parsed); + if (ret) { + dev_err(&pdev->dev, + "Failed to get MMIO regions from platform configuration device\n"); + return ret; + } + + dev_dbg(&pdev->dev, "Found %d MMIO regions in _CRS, mapping...\n", + parsed.count); + + int mapped_count = 0; + for (i = 0; i < parsed.count; i++) { + void __iomem *base = NULL; + u32 addr = parsed.mmio_regions[i].address; + u32 size = parsed.mmio_regions[i].address_length; + + switch (i) { + case 0: + if (dev->pd->port_nums >= 1) { + base = devm_ioremap(&pdev->dev, addr, size); + if (!base) { + dev_err(&pdev->dev, + "Failed to map MAC Port 0 region (0x%08x)\n", + addr); + return -ENOMEM; + } + dev->mmio.mac_port_base[0] = base; + mapped_count++; + } + break; + case 1: + if (dev->pd->port_nums >= 2) { + base = devm_ioremap(&pdev->dev, addr, size); + if (!base) { + dev_err(&pdev->dev, + "Failed to map MAC Port 1 region (0x%08x)\n", + addr); + return -ENOMEM; + } + dev->mmio.mac_port_base[1] = base; + mapped_count++; + } + break; + case 2: + base = devm_ioremap(&pdev->dev, addr, size); + if (!base) { + dev_err(&pdev->dev, + "Failed to map TOP region (0x%08x)\n", + addr); + return -ENOMEM; + } + dev->mmio.top_base = base; + mapped_count++; + break; + case 3: + base = devm_ioremap(&pdev->dev, addr, size); + if (!base) { + dev_err(&pdev->dev, + "Failed to map PROTECT region (0x%08x)\n", + addr); + return -ENOMEM; + } + dev->mmio.protect_base = base; + mapped_count++; + break; + case 4: + base = devm_ioremap(&pdev->dev, addr, size); + if (!base) { + dev_err(&pdev->dev, + "Failed to map CKM region (0x%08x)\n", + addr); + return -ENOMEM; + } + dev->mmio.ckm_base = base; + mapped_count++; + break; + default: + dev_warn(&pdev->dev, + "Unexpected MMIO region at 0x%08x (size 0x%x), skipping\n", + addr, size); + break; + } + } + + if (!dev->mmio.top_base || !dev->mmio.protect_base + || !dev->mmio.ckm_base || (dev->pd->port_nums >= 1 + && !dev->mmio.mac_port_base[0]) + || (dev->pd->port_nums >= 2 && !dev->mmio.mac_port_base[1])) { + dev_err(&pdev->dev, + "Required MMIO regions not mapped from ACPI _CRS (mapped %d)\n", + mapped_count); + if (!dev->mmio.top_base) + dev_err(&pdev->dev, " Missing: TOP\n"); + if (!dev->mmio.protect_base) + dev_err(&pdev->dev, " Missing: PROTECT\n"); + if (!dev->mmio.ckm_base) + dev_err(&pdev->dev, " Missing: CKM\n"); + if (dev->pd->port_nums >= 1 && !dev->mmio.mac_port_base[0]) + dev_err(&pdev->dev, + " Missing: MAC Port 0 (port_nums=%d)\n", + dev->pd->port_nums); + if (dev->pd->port_nums >= 2 && !dev->mmio.mac_port_base[1]) + dev_err(&pdev->dev, + " Missing: MAC Port 1 (port_nums=%d)\n", + dev->pd->port_nums); + dev->mmio.top_base = NULL; + dev->mmio.protect_base = NULL; + dev->mmio.ckm_base = NULL; + for (i = 0; i < HP_PORT_MAX; i++) + dev->mmio.mac_port_base[i] = NULL; + return -ENODEV; + } + + dev_dbg(&pdev->dev, + "Successfully mapped all MMIO regions from ACPI _CRS\n"); + return 0; +} + +/** + * cx7_hp_rp_bus_protect - Bus protection handler + * @dev: hotplug device + * @port_idx: port index (0-based) + * @stage: protection stage (BUS_PROTECT_INIT, BUS_PROTECT_CLEANUP, etc.) + */ +static void cx7_hp_rp_bus_protect(struct cx7_hp_dev *dev, int port_idx, + int stage) +{ + switch (stage) { + case BUS_PROTECT_INIT: + { + int ret; + + ret = cx7_hp_map_mmio_resources(dev); + if (ret) { + dev_err(&dev->pdev->dev, + "Failed to map MMIO resources during bus init: %d\n", + ret); + return; + } + } + return; + + case BUS_PROTECT_CLEANUP: + { + int i; + + for (i = 0; i < HP_PORT_MAX; i++) { + if (dev->mmio.mac_port_base[i]) + dev->mmio.mac_port_base[i] = NULL; + } + if (dev->mmio.top_base) + dev->mmio.top_base = NULL; + if (dev->mmio.protect_base) + dev->mmio.protect_base = NULL; + if (dev->mmio.ckm_base) + dev->mmio.ckm_base = NULL; + } + return; + + case BUS_PROTECT_CABLE_REMOVAL: + case BUS_PROTECT_CABLE_PLUGIN: + { + struct rp_bus_mmio_info *mmio_info = + &dev->pd->rp_bus_mmio; + void __iomem *mac_base; + + if (port_idx >= dev->pd->port_nums) + return; + + mac_base = dev->mmio.mac_port_base[port_idx]; + if (!mac_base) + return; + + if (stage == BUS_PROTECT_CABLE_REMOVAL) { + cx7_hp_reg_update_bits(mac_base, + mmio_info->mac.init_ctrl, + mmio_info->mac.ltssm_bit, + false); + cx7_hp_reg_update_bits(mac_base, + mmio_info->mac.init_ctrl, + mmio_info->mac. + phy_rst_bit, false); + return; + } + + cx7_hp_toggle_update_bit(dev->mmio.top_base, + mmio_info->top.ctrl, + mmio_info->top. + port_bits[port_idx], + mmio_info->top.update_bit, + false); + udelay(CX7_HP_DELAY_SHORT_US); + + cx7_hp_bus_protect_enable(dev, port_idx); + usleep_range(CX7_HP_DELAY_BUS_PROTECT_US, + CX7_HP_DELAY_BUS_PROTECT_US + 1000); + + cx7_hp_reg_update_bits(mac_base, + mmio_info->mac.init_ctrl, + mmio_info->mac.phy_rst_bit, + true); + cx7_hp_reg_update_bits(mac_base, + mmio_info->mac.init_ctrl, + mmio_info->mac.ltssm_bit, true); + usleep_range(CX7_HP_DELAY_PHY_RESET_US, + CX7_HP_DELAY_PHY_RESET_US + 1000); + + cx7_hp_bus_protect_disable(dev, port_idx); + + cx7_hp_toggle_update_bit(dev->mmio.top_base, + mmio_info->top.ctrl, + mmio_info->top. + port_bits[port_idx], + mmio_info->top.update_bit, + true); + } + break; + + default: + dev_warn(&dev->pdev->dev, "Unknown bus protect stage: %d\n", + stage); + break; + } +} + +/** + * retrain_pcie_link - Retrain PCIe link + * @dev: PCI device + */ +static void retrain_pcie_link(struct pci_dev *dev) +{ + u16 link_control, lnksta; + int pos, i = 0; + + pos = pci_find_capability(dev, PCI_CAP_ID_EXP); + if (!pos) { + dev_err(&dev->dev, "PCIe capability not found\n"); + return; + } + + pci_read_config_word(dev, pos + PCI_EXP_LNKCTL, &link_control); + link_control |= PCI_EXP_LNKCTL_RL; + + pci_write_config_word(dev, pos + PCI_EXP_LNKCTL, link_control); + + while (i < HP_POLL_CNT_MAX) { + i++; + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + if (lnksta & PCI_EXP_LNKSTA_DLLLA) + break; + usleep_range(CX7_HP_POLL_SLEEP_US, CX7_HP_POLL_SLEEP_US + 1000); + } + + pcie_capability_write_word(dev, PCI_EXP_LNKSTA, PCI_EXP_LNKSTA_LBMS); +} + +/** + * get_port_root_port - Get PCI root port device for a port + * @hp_dev: hotplug device + * @port_idx: port index + * + * Returns cached or newly found root port, or NULL if not found. + */ +static struct pci_dev *get_port_root_port(struct cx7_hp_dev *hp_dev, + int port_idx) +{ + struct pcie_port_info *port; + + if (!hp_dev->pd || port_idx >= hp_dev->pd->port_nums) + return NULL; + + port = &hp_dev->pd->ports[port_idx]; + + if (!hp_dev->cached_root_ports[port_idx]) { + hp_dev->cached_root_ports[port_idx] = + pci_get_domain_bus_and_slot(port->domain, + port->bus, port->devfn); + if (!hp_dev->cached_root_ports[port_idx]) { + dev_warn(&hp_dev->pdev->dev, + "Root port not found for domain %d bus %d\n", + port->domain, port->bus); + return NULL; + } + } + + return hp_dev->cached_root_ports[port_idx]; +} + +/** + * remove_device - Remove PCIe devices and power down hardware + * @dev: hotplug device + */ +static void remove_device(struct cx7_hp_dev *dev) +{ + int i; + + dev_info(&dev->pdev->dev, "Cable removal\n"); + + for (i = 0; i < dev->pd->port_nums; i++) + cx7_hp_rp_bus_protect(dev, i, BUS_PROTECT_CABLE_REMOVAL); + + gpiod_set_value(dev->pins[PCIE_PIN_PERST].desc, 0); + cx7_hp_change_pinctrl_state(dev, "default"); + cx7_hp_ckm_control(dev, true); + gpiod_set_value(dev->pins[PCIE_PIN_EN].desc, 0); +} + +/** + * polling_link_to_l0 - Poll until all PCIe ports reach L0 state + * @dev: hotplug device + * + * Returns: 0 on success, negative error code on failure + */ +static int polling_link_to_l0(struct cx7_hp_dev *dev) +{ + struct pci_dev *pci_dev; + u32 ltssm_reg; + u32 l0_state; + u32 ltssm_vals[HP_PORT_MAX] = { 0 }; + int count = 0; + int i; + bool all_l0; + + ltssm_reg = dev->pd->ltssm_reg; + l0_state = dev->pd->ltssm_l0_state; + + if (!ltssm_reg || !l0_state) + return 0; /* Skip if not configured */ + + /* Poll until all ports reach L0 state */ + all_l0 = false; + while (!all_l0) { + all_l0 = true; + + for (i = 0; i < dev->pd->port_nums; i++) { + pci_dev = get_port_root_port(dev, i); + if (!pci_dev) { + all_l0 = false; + continue; + } + + pci_read_config_dword(pci_dev, ltssm_reg, + <ssm_vals[i]); + if ((ltssm_vals[i] & l0_state) != l0_state) + all_l0 = false; + } + + if (all_l0) + break; + + usleep_range(CX7_HP_POLL_SLEEP_US, CX7_HP_POLL_SLEEP_US + 1000); + count++; + + if (count > HP_POLL_CNT_MAX) { + dev_err(&dev->pdev->dev, + "Timeout waiting for link to reach L0 (reached max count)\n"); + break; + } + } + + if (count > HP_POLL_CNT_MAX) { + return -ETIMEDOUT; + } + + return 0; +} + +/** + * rescan_device - Rescan PCIe bus to discover devices + * @dev: hotplug device + * + * Returns: 0 on success, negative error code on failure + */ +static int rescan_device(struct cx7_hp_dev *dev) +{ + struct pci_dev *pci_dev; + int i, err; + + err = cx7_hp_change_pinctrl_state(dev, "clkreqn"); + if (err) + return err; + + cx7_hp_ckm_control(dev, false); + usleep_range(CX7_HP_DELAY_STANDARD_US, CX7_HP_DELAY_STANDARD_US + 1000); + + for (i = 0; i < dev->pd->port_nums; i++) { + pci_dev = get_port_root_port(dev, i); + if (!pci_dev) + continue; + + err = pm_runtime_resume_and_get(&pci_dev->dev); + if (err < 0) { + dev_err(&dev->pdev->dev, + "Runtime resume failed for %s: %d\n", + pci_name(pci_dev), err); + } + } + + gpiod_set_value(dev->pins[PCIE_PIN_PERST].desc, 1); + + for (i = 0; i < dev->pd->port_nums; i++) + cx7_hp_rp_bus_protect(dev, i, BUS_PROTECT_CABLE_PLUGIN); + + err = polling_link_to_l0(dev); + if (err) + return err; + + for (i = 0; i < dev->pd->port_nums; i++) { + pci_dev = get_port_root_port(dev, i); + if (pci_dev) + retrain_pcie_link(pci_dev); + } + + msleep(CX7_HP_DELAY_LINK_STABLE_MS); + + return 0; +} + +/** + * cx7_hp_work - Work queue handler for hotplug state machine + * @irq: interrupt number + * @dev_id: GPIO context pointer + * + * Processes hotplug state transitions based on current state. + */ +static irqreturn_t cx7_hp_work(int irq, void *dev_id) +{ + struct cx7_hp_gpio_ctx *app_ctx = dev_id; + struct cx7_hp_dev *hp_dev; + enum cx7_hp_state state; + unsigned long flags; + int ret; + + if (!app_ctx || !app_ctx->hp_dev) + return IRQ_NONE; + + hp_dev = app_ctx->hp_dev; + + spin_lock_irqsave(&hp_dev->lock, flags); + if (!hp_dev->hotplug_enabled) { + spin_unlock_irqrestore(&hp_dev->lock, flags); + return IRQ_HANDLED; + } + state = hp_dev->state; + spin_unlock_irqrestore(&hp_dev->lock, flags); + + switch (state) { + case STATE_PLUG_OUT: + remove_device(hp_dev); + break; + case STATE_PLUG_IN: + dev_info(&hp_dev->pdev->dev, "Cable plugin\n"); + gpiod_set_value(hp_dev->pins[PCIE_PIN_EN].desc, 1); + break; + case STATE_DEV_POWER_OFF: + case STATE_DEV_POWER_ON: + case STATE_DEV_FW_START: + break; + case STATE_RESCAN: + ret = rescan_device(hp_dev); + spin_lock_irqsave(&hp_dev->lock, flags); + if (ret) + dev_err(app_ctx->ctx->dev, "Rescan failed: %d\n", ret); + else + hp_dev->state = STATE_READY; + spin_unlock_irqrestore(&hp_dev->lock, flags); + break; + default: + dev_err(app_ctx->ctx->dev, "Unknown state: %d\n", state); + break; + } + + return IRQ_HANDLED; +} + +/** + * hotplug_irq_handler - GPIO interrupt handler for hotplug events + * @irq: interrupt number + * @dev_id: GPIO context pointer + * + * Handles presence detection and boot status GPIO interrupts. + */ +static irqreturn_t hotplug_irq_handler(int irq, void *dev_id) +{ + struct cx7_hp_gpio_ctx *app_ctx = dev_id; + struct cx7_hp_dev *hp_dev = app_ctx->hp_dev; + struct gpio_acpi_context *gpio_ctx = app_ctx->ctx; + unsigned long flags; + int value; + enum cx7_hp_state state; + + value = gpiod_get_value(app_ctx->desc); + + if (gpio_ctx->pin == hp_dev->prsnt_pin) { + if (value) { + cx7_hp_send_uevent(hp_dev, REMOVAL_EVT); + } else { + cx7_hp_send_uevent(hp_dev, PLUG_IN_EVT); + } + return IRQ_HANDLED; + } + + spin_lock_irqsave(&hp_dev->lock, flags); + if (!hp_dev->hotplug_enabled) { + spin_unlock_irqrestore(&hp_dev->lock, flags); + return IRQ_HANDLED; + } + state = hp_dev->state; + + if (gpio_ctx->pin == hp_dev->boot_pin) { + if (value && state == STATE_PLUG_IN) { + hp_dev->state = STATE_DEV_POWER_ON; + } else if (value && state == STATE_DEV_FW_START) { + hp_dev->state = STATE_RESCAN; + } else if (!value && state == STATE_DEV_POWER_ON) { + hp_dev->state = STATE_DEV_FW_START; + } else if (!value && state == STATE_PLUG_OUT) { + hp_dev->state = STATE_DEV_POWER_OFF; + } else { + spin_unlock_irqrestore(&hp_dev->lock, flags); + return IRQ_HANDLED; + } + spin_unlock_irqrestore(&hp_dev->lock, flags); + return IRQ_WAKE_THREAD; + } + + dev_err(gpio_ctx->dev, + "Unknown GPIO pin event: pin=%d irq=%d value=%d\n", + gpio_ctx->pin, irq, value); + spin_unlock_irqrestore(&hp_dev->lock, flags); + return IRQ_HANDLED; +} + +/** + * acpi_gpio_collect_handler - ACPI resource handler to collect all GPIO resources + * @ares: ACPI resource structure + * @context: Pointer to acpi_gpio_walk_context + * + * Returns: AE_OK to continue iteration + */ +static acpi_status acpi_gpio_collect_handler(struct acpi_resource *ares, + void *context) +{ + struct acpi_gpio_walk_context *walk_ctx = context; + struct acpi_resource_gpio *agpio; + int length; + + if (ares->type != ACPI_RESOURCE_TYPE_GPIO) + return AE_OK; + + if (walk_ctx->count >= PCIE_PIN_MAX) { + dev_warn(walk_ctx->dev, + "Too many GPIO resources, truncating at %d\n", + PCIE_PIN_MAX); + return AE_OK; + } + + agpio = &ares->data.gpio; + + if (!agpio->pin_table || agpio->pin_table_length == 0) { + dev_warn(walk_ctx->dev, "GPIO resource has no pin table\n"); + return AE_OK; + } + + walk_ctx->gpios[walk_ctx->count].pin = agpio->pin_table[0]; + walk_ctx->gpios[walk_ctx->count].connection_type = + agpio->connection_type; + walk_ctx->gpios[walk_ctx->count].triggering = agpio->triggering; + walk_ctx->gpios[walk_ctx->count].polarity = agpio->polarity; + walk_ctx->gpios[walk_ctx->count].debounce_timeout = + agpio->debounce_timeout; + walk_ctx->gpios[walk_ctx->count].wake_capable = agpio->wake_capable; + + if (agpio->vendor_length && agpio->vendor_data) { + length = min_t(int, agpio->vendor_length, MAX_VENDOR_DATA_LEN); + memcpy(walk_ctx->gpios[walk_ctx->count].vendor_data, + agpio->vendor_data, length); + walk_ctx->gpios[walk_ctx->count].vendor_data[length] = '\0'; + } else { + walk_ctx->gpios[walk_ctx->count].vendor_data[0] = '\0'; + } + + if (agpio->resource_source.string_ptr) { + length = min_t(int, agpio->resource_source.string_length, 15); + memcpy(walk_ctx->gpios[walk_ctx->count].resource_source, + agpio->resource_source.string_ptr, length); + walk_ctx->gpios[walk_ctx->count].resource_source[length] = '\0'; + } else { + walk_ctx->gpios[walk_ctx->count].resource_source[0] = '\0'; + } + walk_ctx->gpios[walk_ctx->count].resource_source_index = + agpio->resource_source.index; + walk_ctx->count++; + return AE_OK; +} + +/** + * cx7_hp_walk_acpi_gpios - Walk ACPI _CRS to collect all GPIO resources + * @pdev: Platform device + * @walk_ctx: Context structure to fill with GPIO information + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_walk_acpi_gpios(struct platform_device *pdev, + struct acpi_gpio_walk_context *walk_ctx) +{ + struct acpi_device *adev; + acpi_status status; + + adev = ACPI_COMPANION(&pdev->dev); + if (!adev) { + dev_err(&pdev->dev, "Failed to get ACPI companion device\n"); + return -ENODEV; + } + + memset(walk_ctx, 0, sizeof(*walk_ctx)); + walk_ctx->dev = &pdev->dev; + + status = acpi_walk_resources(adev->handle, METHOD_NAME__CRS, + acpi_gpio_collect_handler, walk_ctx); + if (ACPI_FAILURE(status)) { + dev_err(&pdev->dev, "Failed to walk ACPI GPIO resources: %s\n", + acpi_format_exception(status)); + return -EIO; + } + + dev_dbg(&pdev->dev, "Found %d GPIO resources via ACPI walk\n", + walk_ctx->count); + + if (walk_ctx->count == 0) { + dev_err(&pdev->dev, "No GPIO resources found in ACPI _CRS\n"); + return -ENODEV; + } + + return 0; +} + +/** + * acpi_gpio_lookup_handler - ACPI resource handler to look up a specific GPIO pin + * @ares: ACPI resource being processed + * @context: Pointer to acpi_gpio_parse_context + * + * Returns: AE_OK to continue iteration + */ +static acpi_status acpi_gpio_lookup_handler(struct acpi_resource *ares, + void *context) +{ + struct acpi_gpio_parse_context *parse_ctx = context; + struct gpio_acpi_context *ctx = parse_ctx->ctx; + struct cx7_hp_dev *hp_dev = parse_ctx->hp_dev; + struct acpi_resource_gpio *agpio; + int length; + + if (ares->type != ACPI_RESOURCE_TYPE_GPIO) + return AE_OK; + + agpio = &ares->data.gpio; + + if (ctx->pin != agpio->pin_table[0]) + return AE_OK; + + ctx->valid = 1; + ctx->debounce_timeout_us = agpio->debounce_timeout * 10; + ctx->wake_capable = agpio->wake_capable; + ctx->triggering = agpio->triggering; + ctx->polarity = agpio->polarity; + ctx->connection_type = agpio->connection_type; + + if (agpio->vendor_length && agpio->vendor_data && hp_dev) { + length = min_t(int, agpio->vendor_length, MAX_VENDOR_DATA_LEN); + memcpy(&ctx->vendor_data[0], agpio->vendor_data, length); + ctx->vendor_data[length] = '\0'; + + if (!strncmp("BOOT", ctx->vendor_data, strlen("BOOT"))) + hp_dev->boot_pin = ctx->pin; + else if (!strncmp("PRSNT", ctx->vendor_data, strlen("PRSNT"))) + hp_dev->prsnt_pin = ctx->pin; + } + + if (agpio->triggering == ACPI_EDGE_SENSITIVE) { + if (agpio->polarity == ACPI_ACTIVE_LOW) + ctx->irq_flags = IRQF_TRIGGER_FALLING; + else if (agpio->polarity == ACPI_ACTIVE_HIGH) + ctx->irq_flags = IRQF_TRIGGER_RISING; + else + ctx->irq_flags = + (IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING); + } else { + if (agpio->polarity == ACPI_ACTIVE_LOW) + ctx->irq_flags = IRQF_TRIGGER_LOW; + else + ctx->irq_flags = IRQF_TRIGGER_HIGH; + } + + return AE_OK; +} + +/** + * pci_devices_present_on_domain() - Check if PCI devices exist on a domain + * @domain: PCI domain number to check + * + * Returns: true if any PCI devices are present on the specified domain, + * false otherwise. This is used as a safety check before hardware shutdown. + */ +static bool pci_devices_present_on_domain(int domain) +{ + struct pci_bus *bus; + struct pci_dev *dev; + bool has_endpoint_devices = false; + + bus = pci_find_bus(domain, 1); + if (!bus) + return false; + + list_for_each_entry(dev, &bus->devices, bus_list) { + has_endpoint_devices = true; + break; + } + + return has_endpoint_devices; +} + +static ssize_t debug_state_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cx7_hp_dev *hp_dev = dev_get_drvdata(dev); + + if (!hp_dev) + return -EINVAL; + + return scnprintf(buf, PAGE_SIZE, "%d\n", hp_dev->debug_state); +} + +static ssize_t debug_state_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct cx7_hp_dev *hp_dev = dev_get_drvdata(dev); + unsigned long val, flags; + int err, i; + + if (!hp_dev || !hp_dev->pd) + return -EINVAL; + + err = kstrtoul(buf, 10, &val); + if (err) + return err; + + spin_lock_irqsave(&hp_dev->lock, flags); + if (!hp_dev->hotplug_enabled) { + spin_unlock_irqrestore(&hp_dev->lock, flags); + dev_info(dev, "Hotplug is disabled.\n"); + return -EPERM; + } + spin_unlock_irqrestore(&hp_dev->lock, flags); + + switch (val) { + case CX7_HP_DEBUG_PLUG_OUT: + /* Safety check: Verify no devices on the bus before hardware shutdown. */ + for (i = 0; i < hp_dev->pd->port_nums; i++) { + if (pci_devices_present_on_domain + (hp_dev->pd->ports[i].domain)) { + dev_err(dev, + "PCI devices still present, remove them first\n"); + return -EBUSY; + } + } + + spin_lock_irqsave(&hp_dev->lock, flags); + hp_dev->state = STATE_PLUG_OUT; + hp_dev->debug_state = val; + spin_unlock_irqrestore(&hp_dev->lock, flags); + remove_device(hp_dev); + return count; + + case CX7_HP_DEBUG_PLUG_IN: + for (i = 0; i < hp_dev->pd->port_nums; i++) { + if (pci_devices_present_on_domain + (hp_dev->pd->ports[i].domain)) { + dev_err(dev, + "PCI devices already present, cannot reinitialize hardware\n"); + return -EBUSY; + } + } + + spin_lock_irqsave(&hp_dev->lock, flags); + hp_dev->state = STATE_PLUG_IN; + hp_dev->debug_state = val; + spin_unlock_irqrestore(&hp_dev->lock, flags); + dev_info(dev, "Cable plugin\n"); + gpiod_set_value(hp_dev->pins[PCIE_PIN_EN].desc, 1); + return count; + + default: + return -EINVAL; + } + + return count; +} + +DEVICE_ATTR_RW(debug_state); + +static ssize_t hotplug_enabled_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cx7_hp_dev *hp_dev = dev_get_drvdata(dev); + + if (!hp_dev) + return -EINVAL; + + return scnprintf(buf, PAGE_SIZE, "%d\n", hp_dev->hotplug_enabled ? 1 : 0); +} + +static ssize_t hotplug_enabled_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct cx7_hp_dev *hp_dev = dev_get_drvdata(dev); + unsigned long val; + int err; + + if (!hp_dev) + return -EINVAL; + + err = kstrtoul(buf, 10, &val); + if (err) + return err; + + hp_dev->hotplug_enabled = (val != 0); + dev_info(dev, "Hotplug %s\n", hp_dev->hotplug_enabled ? "enabled" : "disabled"); + + return count; +} + +DEVICE_ATTR_RW(hotplug_enabled); + +static struct attribute *cx7_hp_attrs[] = { + &dev_attr_debug_state.attr, + &dev_attr_hotplug_enabled.attr, + NULL +}; + +static const struct attribute_group cx7_hp_attr_group = { + .name = "pcie_hotplug", + .attrs = cx7_hp_attrs +}; + +/** + * gpio_acpi_setup - Setup GPIO ACPI context from _CRS + * @pdev: platform device + * @desc: GPIO descriptor + * @hp_dev: hotplug device + * @gpio_index: GPIO index + * + * Returns: GPIO ACPI context on success, NULL on failure + */ +static struct gpio_acpi_context *gpio_acpi_setup(struct platform_device *pdev, + struct gpio_desc *desc, + struct cx7_hp_dev *hp_dev, + int gpio_index) +{ + struct acpi_gpio_parse_context parse_ctx; + struct gpio_acpi_context *ctx; + struct acpi_device *adev; + acpi_status status; + + adev = ACPI_COMPANION(&pdev->dev); + if (!adev) { + dev_err(&pdev->dev, "Failed to get ACPI companion device\n"); + return NULL; + } + + ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return NULL; + + ctx->pin = + desc_to_gpio(desc) - + gpio_device_get_base(gpiod_to_gpio_device(desc)); + ctx->dev = &pdev->dev; + + parse_ctx.ctx = ctx; + parse_ctx.hp_dev = hp_dev; + + status = acpi_walk_resources(adev->handle, METHOD_NAME__CRS, + acpi_gpio_lookup_handler, &parse_ctx); + if (ACPI_FAILURE(status)) { + devm_kfree(&pdev->dev, ctx); + return NULL; + } + + if (ctx->valid) { + if (gpio_index == PCIE_PIN_BOOT && hp_dev->boot_pin == -1) { + hp_dev->boot_pin = ctx->pin; + } else if (gpio_index == PCIE_PIN_PRSNT + && hp_dev->prsnt_pin == -1) { + hp_dev->prsnt_pin = ctx->pin; + } + return ctx; + } + + devm_kfree(&pdev->dev, ctx); + return NULL; +} + +/** + * cx7_hp_setup_irq - Setup IRQ for GPIO + * @app_ctx: GPIO context + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_setup_irq(struct cx7_hp_gpio_ctx *app_ctx) +{ + struct gpio_acpi_context *ctx = app_ctx->ctx; + int irq, ret; + + irq = gpiod_to_irq(app_ctx->desc); + if (irq < 0) { + dev_err(ctx->dev, "Failed to get IRQ for GPIO\n"); + return irq; + } + + if (ctx->wake_capable) + enable_irq_wake(irq); + + ret = devm_request_threaded_irq(ctx->dev, irq, + hotplug_irq_handler, cx7_hp_work, + ctx->irq_flags | IRQF_ONESHOT, + "pcie_hotplug", app_ctx); + if (ret) + dev_err(ctx->dev, "Failed to request IRQ %d: %d\n", irq, ret); + + return ret; +} + +/** + * cx7_hp_put_gpio_device - Release GPIO device reference + * @data: GPIO device pointer + */ +static void cx7_hp_put_gpio_device(void *data) +{ + struct gpio_device *gdev = data; + + gpio_device_put(gdev); +} + +/** + * cx7_hp_discover_pcie_devices - Discover existing PCI devices on managed ports + * @pdev: platform device + * @pd: platform data + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_discover_pcie_devices(struct platform_device *pdev, + struct cx7_hp_plat_data *pd) +{ + struct pci_dev *pci_dev = NULL; + int device_count = 0; + int i; + + if (!pd->vendor_id || !pd->device_id) + return 0; + + while ((pci_dev = pci_get_device(pd->vendor_id, + pd->device_id, pci_dev)) != NULL) { + if (!pci_dev->state_saved) { + pci_dev_put(pci_dev); + return -EPROBE_DEFER; + } + + for (i = 0; i < pd->port_nums; i++) { + if (pci_domain_nr(pci_dev->bus) == pd->ports[i].domain) + break; + } + + if (i == pd->port_nums) { + dev_err(&pdev->dev, + "Device %s found on unexpected domain %d\n", + pci_name(pci_dev), pci_domain_nr(pci_dev->bus)); + pci_dev_put(pci_dev); + return -ENODEV; + } + + device_count++; + } + + if (pd->num_devices && device_count != pd->num_devices) { + dev_err(&pdev->dev, + "Required number of devices not found. Expected=%d Actual=%d\n", + pd->num_devices, device_count); + return -ENODEV; + } + + return 0; +} + +/** + * cx7_hp_init_pcie_data - Initialize PCIe data from _DSD and discover devices + * @pdev: platform device + * @pd: platform data to populate + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_init_pcie_data(struct platform_device *pdev, + struct cx7_hp_plat_data *pd) +{ + int ret; + + ret = cx7_hp_parse_pcie_config_dsd(pdev, pd); + if (ret) { + dev_err(&pdev->dev, + "Failed to parse PCIe configuration _DSD properties: %d\n", + ret); + return ret; + } + + if (pd->port_nums == 0 || pd->port_nums >= HP_PORT_MAX) { + dev_err(&pdev->dev, + "Invalid port count from _DSD: %d (must be 1-%d)\n", + pd->port_nums, HP_PORT_MAX - 1); + return -EINVAL; + } + + ret = cx7_hp_discover_pcie_devices(pdev, pd); + if (ret) { + dev_dbg(&pdev->dev, "Device discovery failed: %d\n", ret); + return ret; + } + + return 0; +} + +/** + * cx7_hp_enumerate_gpios - Enumerate GPIOs from ACPI + * @pdev: Platform device + * @hp_dev: Hotplug device structure + * + * Returns: Number of GPIOs found, or negative error code + */ +static int cx7_hp_enumerate_gpios(struct platform_device *pdev, + struct cx7_hp_dev *hp_dev) +{ + struct acpi_gpio_walk_context walk_ctx; + struct fwnode_handle *gpio_fwnode = NULL; + struct acpi_device *gpio_adev = NULL; + acpi_handle gpio_handle; + acpi_status status; + int ret, i; + + ret = cx7_hp_walk_acpi_gpios(pdev, &walk_ctx); + if (ret) { + dev_err(&pdev->dev, "Failed to walk ACPI GPIO resources: %d\n", + ret); + return ret; + } + + if (walk_ctx.count < CX7_HP_MIN_GPIO_COUNT) { + dev_err(&pdev->dev, + "Insufficient GPIOs from ACPI: required at least %d, got %d\n", + CX7_HP_MIN_GPIO_COUNT, walk_ctx.count); + return -ENODEV; + } + + /* Find GPIO device using resource_source from first GPIO */ + if (walk_ctx.count == 0 || walk_ctx.gpios[0].resource_source[0] == '\0') { + dev_err(&pdev->dev, + "No resource_source in ACPI GPIO resources\n"); + return -ENODEV; + } + + status = + acpi_get_handle(NULL, walk_ctx.gpios[0].resource_source, + &gpio_handle); + if (ACPI_FAILURE(status)) { + dev_err(&pdev->dev, + "Failed to get ACPI handle for GPIO controller %s\n", + walk_ctx.gpios[0].resource_source); + return -ENODEV; + } + + gpio_adev = acpi_fetch_acpi_dev(gpio_handle); + if (!gpio_adev) { + dev_err(&pdev->dev, + "Failed to get ACPI device for GPIO controller %s\n", + walk_ctx.gpios[0].resource_source); + return -ENODEV; + } + + gpio_fwnode = acpi_fwnode_handle(gpio_adev); + hp_dev->gdev = gpio_device_find_by_fwnode(gpio_fwnode); + if (!hp_dev->gdev) { + return dev_err_probe(&pdev->dev, -EPROBE_DEFER, + "GPIO controller not available\n"); + } + + /* Successfully found GPIO device - manage reference */ + ret = devm_add_action_or_reset(&pdev->dev, cx7_hp_put_gpio_device, + hp_dev->gdev); + if (ret) { + gpio_device_put(hp_dev->gdev); + hp_dev->gdev = NULL; + dev_err(&pdev->dev, "Failed to register GPIO device cleanup\n"); + return ret; + } + + hp_dev->gpio_count = walk_ctx.count; + + hp_dev->pins = devm_kzalloc(&pdev->dev, + sizeof(struct cx7_hp_gpio_ctx) * + hp_dev->gpio_count, GFP_KERNEL); + if (!hp_dev->pins) { + dev_err(&pdev->dev, "Failed to allocate memory for GPIOs\n"); + return -ENOMEM; + } + + for (i = 0; i < hp_dev->gpio_count; i++) { + struct cx7_hp_gpio_ctx *app_ctx = &hp_dev->pins[i]; + + app_ctx->desc = + gpio_device_get_desc(hp_dev->gdev, walk_ctx.gpios[i].pin); + if (IS_ERR(app_ctx->desc)) { + dev_err(&pdev->dev, + "Failed to get GPIO descriptor for ACPI pin %u (index %d): %ld\n", + walk_ctx.gpios[i].pin, i, + PTR_ERR(app_ctx->desc)); + return PTR_ERR(app_ctx->desc); + } + + app_ctx->hp_dev = hp_dev; + } + + return hp_dev->gpio_count; +} + +/** + * cx7_hp_pci_notifier - PCI bus notifier to configure MPS for CX7 devices + * @nb: notifier block + * @action: bus notification action + * @data: pointer to device being added/removed + * + * Returns: NOTIFY_OK on success, NOTIFY_DONE if not a CX7 device + */ +static int cx7_hp_pci_notifier(struct notifier_block *nb, unsigned long action, + void *data) +{ + struct device *dev = data; + struct pci_dev *pdev = to_pci_dev(dev); + struct cx7_hp_dev *hp_dev; + unsigned long flags; + + if (action != BUS_NOTIFY_ADD_DEVICE) + return NOTIFY_DONE; + + hp_dev = container_of(nb, struct cx7_hp_dev, pci_notifier); + if (!hp_dev || !hp_dev->pd) + return NOTIFY_DONE; + + spin_lock_irqsave(&hp_dev->lock, flags); + if (!hp_dev->hotplug_enabled) { + spin_unlock_irqrestore(&hp_dev->lock, flags); + return NOTIFY_DONE; + } + spin_unlock_irqrestore(&hp_dev->lock, flags); + + if (!pdev || !hp_dev->pd->vendor_id || !hp_dev->pd->device_id) + return NOTIFY_DONE; + + if (pdev->vendor != hp_dev->pd->vendor_id || + pdev->device != hp_dev->pd->device_id) + return NOTIFY_DONE; + + if (pdev->bus) + pcie_bus_configure_settings(pdev->bus); + + return NOTIFY_OK; +} + +/** + * cx7_hp_probe - Platform device probe function + * @pdev: platform device + * + * Initializes the PCIe hotplug driver, parses ACPI resources, and sets up + * GPIO interrupts and sysfs interface. + * + * Returns: 0 on success, negative error code on failure + */ +static int cx7_hp_probe(struct platform_device *pdev) +{ + struct cx7_hp_plat_data *pd; + struct cx7_hp_gpio_ctx *app_ctx; + struct cx7_hp_dev *hp_dev; + int ret, i; + + pd = devm_kzalloc(&pdev->dev, sizeof(*pd), GFP_KERNEL); + if (!pd) { + dev_err(&pdev->dev, + "Failed to allocate memory for platform data\n"); + return -ENOMEM; + } + + ret = cx7_hp_init_pcie_data(pdev, pd); + if (ret) + return ret; + + hp_dev = devm_kzalloc(&pdev->dev, sizeof(*hp_dev), GFP_KERNEL); + if (!hp_dev) { + dev_err(&pdev->dev, + "Failed to allocate memory for hotplug device\n"); + return -ENOMEM; + } + + hp_dev->pdev = pdev; + hp_dev->pd = pd; + hp_dev->state = STATE_READY; + hp_dev->boot_pin = -1; + hp_dev->prsnt_pin = -1; + hp_dev->hotplug_enabled = false; + spin_lock_init(&hp_dev->lock); + + for (i = 0; i < HP_PORT_MAX; i++) + hp_dev->cached_root_ports[i] = NULL; + + ret = cx7_hp_enumerate_gpios(pdev, hp_dev); + if (ret < 0) { + dev_err(&pdev->dev, "Failed to enumerate GPIOs from ACPI: %d\n", + ret); + return ret; + } + + for (i = 0; i < hp_dev->gpio_count; i++) { + app_ctx = &hp_dev->pins[i]; + + app_ctx->ctx = gpio_acpi_setup(pdev, app_ctx->desc, hp_dev, i); + if (!app_ctx->ctx) { + dev_err(&pdev->dev, "Failed to setup GPIO %d\n", i); + return -ENODEV; + } + + gpiod_set_debounce(app_ctx->desc, + app_ctx->ctx->debounce_timeout_us); + + if (app_ctx->ctx->connection_type == + ACPI_RESOURCE_GPIO_TYPE_INT) { + ret = cx7_hp_setup_irq(app_ctx); + if (ret) { + dev_err(&pdev->dev, + "Failed to setup IRQ for GPIO %d\n", i); + return ret; + } + } + } + + platform_set_drvdata(pdev, hp_dev); + + ret = cx7_hp_pinctrl_init(hp_dev); + if (ret) { + dev_err(&pdev->dev, "Pinmux init failed, ret: %d\n", ret); + return ret; + } + + ret = sysfs_create_group(&pdev->dev.kobj, &cx7_hp_attr_group); + if (ret) { + dev_err(&pdev->dev, "Sysfs creation failed: %d\n", ret); + goto pinctrl_remove; + } + + cx7_hp_rp_bus_protect(hp_dev, 0, BUS_PROTECT_INIT); + + hp_dev->pci_notifier.notifier_call = cx7_hp_pci_notifier; + ret = bus_register_notifier(&pci_bus_type, &hp_dev->pci_notifier); + if (ret) { + dev_err(&pdev->dev, "Failed to register PCI bus notifier: %d\n", + ret); + goto sysfs_remove; + } + + if (gpiod_get_value(hp_dev->pins[PCIE_PIN_PRSNT].desc)) { + hp_dev->debug_state = CX7_HP_DEBUG_PLUG_OUT; + cx7_hp_send_uevent(hp_dev, REMOVAL_EVT); + } else { + hp_dev->debug_state = CX7_HP_DEBUG_PLUG_IN; + cx7_hp_send_uevent(hp_dev, PLUG_IN_EVT); + } + + dev_info(&pdev->dev, "PCIe hotplug driver initialized successfully\n"); + return 0; + +sysfs_remove: + sysfs_remove_group(&pdev->dev.kobj, &cx7_hp_attr_group); +pinctrl_remove: + cx7_hp_pinctrl_remove(hp_dev); + return ret; +} + +/** + * cx7_hp_remove - Platform device remove function + * @pdev: platform device + * + * Cleans up GPIO pins, pinctrl, sysfs interface, and bus protection. + */ +static void cx7_hp_remove(struct platform_device *pdev) +{ + struct cx7_hp_dev *hp_dev = platform_get_drvdata(pdev); + int i; + + if (!hp_dev) + return; + + sysfs_remove_group(&pdev->dev.kobj, &cx7_hp_attr_group); + + bus_unregister_notifier(&pci_bus_type, &hp_dev->pci_notifier); + + cx7_hp_rp_bus_protect(hp_dev, 0, BUS_PROTECT_CLEANUP); + + cx7_hp_pinctrl_remove(hp_dev); + + for (i = 0; i < hp_dev->pd->port_nums; i++) { + if (hp_dev->cached_root_ports[i]) + pci_dev_put(hp_dev->cached_root_ports[i]); + } + + platform_set_drvdata(pdev, NULL); +} + +static const struct acpi_device_id cx7_hp_acpi_match[] = { + {"MTKP0001", 0}, + {} +}; + +MODULE_DEVICE_TABLE(acpi, cx7_hp_acpi_match); + +static struct platform_driver cx7_hp_driver = { + .probe = cx7_hp_probe, + .remove = cx7_hp_remove, + .driver = { + .name = "cx7-pcie-hotplug", + .acpi_match_table = ACPI_PTR(cx7_hp_acpi_match), + }, +}; + +module_platform_driver(cx7_hp_driver); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("CX7 PCIe Hotplug Driver for NVIDIA DGX Systems"); From 0c521e62ea9a61b6546b3eacf0c2977965895fb9 Mon Sep 17 00:00:00 2001 From: Brad Figg Date: Fri, 29 Mar 2024 13:31:34 -0700 Subject: [PATCH 093/464] NVIDIA: [Packaging] Add nvidia-fs build dependencies BugLink: https://bugs.launchpad.net/bugs/2059814 Signed-off-by: Brad Figg Acked-by: Brad Figg Acked-by: Ian May Signed-off-by: Ian May Signed-off-by: Jacob Martin (cherry picked from commit a64b5977c0cb9bb66af5f3d9fd7ed2a7eaebc131 linux-nvidia-6.14) Signed-off-by: Abdur Rahman (cherry picked from commit 67713ae343c95f540b3f881160dee756ea132a3c noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/control.stub.in | 2 ++ 1 file changed, 2 insertions(+) diff --git a/debian.nvidia-bos/control.stub.in b/debian.nvidia-bos/control.stub.in index 9c20f84ca0721..6b9adaf973954 100644 --- a/debian.nvidia-bos/control.stub.in +++ b/debian.nvidia-bos/control.stub.in @@ -53,6 +53,8 @@ Build-Depends: uuid-dev , zstd , bpftool:native [amd64 arm64] , + nvidia-dkms-kernel [amd64 arm64] , + nvidia-kernel-source [amd64 arm64] , Build-Depends-Indep: asciidoc , bzip2 , From 7a6864dd9542de0fe7c381f06065a8139bbcb70a Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Mon, 1 Dec 2025 13:48:51 +0530 Subject: [PATCH 094/464] NVIDIA: VR: SAUCE: soc/tegra: misc: Use SMCCC to get chipid BugLink: https://bugs.launchpad.net/bugs/2138329 Tegra410 and Tegra241 have deprecated HIDREV register. It is recommended to use ARM SMCCC calls to get chip_id, major and minor revisions. Use ARM SMCCC to get chip_id, major and minor revision. Signed-off-by: Kartik Rajput Signed-off-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 676a37a1ce6720c5f239e1da3693e7fcee9f8153 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/soc/tegra/fuse/tegra-apbmisc.c | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/drivers/soc/tegra/fuse/tegra-apbmisc.c b/drivers/soc/tegra/fuse/tegra-apbmisc.c index 0ce94fdc536fb..7ef1782534d5e 100644 --- a/drivers/soc/tegra/fuse/tegra-apbmisc.c +++ b/drivers/soc/tegra/fuse/tegra-apbmisc.c @@ -4,6 +4,7 @@ */ #include +#include #include #include #include @@ -27,6 +28,11 @@ #define PMC_STRAPPING_OPT_A_RAM_CODE_MASK_SHORT \ (0x3 << PMC_STRAPPING_OPT_A_RAM_CODE_SHIFT) +#define TEGRA_SMCCC_PLATFORM(x) ((x >> 8) & 0xff) +#define TEGRA_SMCCC_CHIP_ID(x) ((x >> 4) & 0xff) +#define TEGRA_SMCCC_MAJOR_REV(x) (x & 0xf) +#define TEGRA_SMCCC_MINOR_REV(x) (x & 0xf) + static void __iomem *apbmisc_base; static bool long_ram_code; static u32 strapping; @@ -41,21 +47,46 @@ u32 tegra_read_chipid(void) u8 tegra_get_chip_id(void) { +#ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY + s32 soc_id = arm_smccc_get_soc_id_version(); + + if (soc_id >= 0) + return TEGRA_SMCCC_CHIP_ID(soc_id); +#endif return (tegra_read_chipid() >> 8) & 0xff; } u8 tegra_get_major_rev(void) { +#ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY + s32 soc_id = arm_smccc_get_soc_id_version(); + + if (soc_id >= 0) + return TEGRA_SMCCC_MAJOR_REV(soc_id); +#endif return (tegra_read_chipid() >> 4) & 0xf; } u8 tegra_get_minor_rev(void) { +#ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY + s32 revision = arm_smccc_get_soc_id_revision(); + + if (revision >= 0) + return TEGRA_SMCCC_MINOR_REV(revision); +#endif return (tegra_read_chipid() >> 16) & 0xf; + } u8 tegra_get_platform(void) { +#ifdef CONFIG_HAVE_ARM_SMCCC_DISCOVERY + s32 revision = arm_smccc_get_soc_id_revision(); + + if (revision >= 0) + return TEGRA_SMCCC_PLATFORM(revision); +#endif return (tegra_read_chipid() >> 20) & 0xf; } From 4e18e989101ac34c02bb6b7c10bf2a0d32177576 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Thu, 20 Feb 2025 20:39:06 -0800 Subject: [PATCH 095/464] NVIDIA: VR: SAUCE: cxl: add support for cxl reset BugLink: https://bugs.launchpad.net/bugs/2138266 Type 2 devices are being introduced and will require finer-grained reset mechanisms beyond bus-wide reset methods. Add support for CXL reset per CXL v3.2 Section 9.6/9.7 Signed-off-by: Srirangan Madhavan (backported from https://lore.kernel.org/all/20250221043906.1593189-3-smadhavan@nvidia.com/) [Nirmoy: Add #include "../cxl/cxlpci.h" and fix a compile error with if (reg & CXL_DVSEC_CXL_RST_CAPABLE == 0)] Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (backported from commit 0e06082184df86b7fe96d3f4d5a0fd075aac4002 noble:linux-nvidia-6.17) [jacobmartin: adjust CXL register definitions to match file location and naming of upstream commit 0f7afd80d81b ("PCI: Move CXL DVSEC definitions into uapi/linux/pci_regs.h") using https://lore.kernel.org/all/20260306092322.148765-2-smadhavan@nvidia.com/ as a reference.] Signed-off-by: Jacob Martin --- drivers/pci/pci.c | 146 ++++++++++++++++++++++++++++++++++ include/linux/pci.h | 2 +- include/uapi/linux/pci_regs.h | 14 ++++ 3 files changed, 161 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 9298a461bd302..0b4b7371341f6 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4980,6 +4980,151 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe) return rc; } +static int cxl_reset_prepare(struct pci_dev *dev, u16 dvsec) +{ + u32 timeout_us = 100, timeout_tot_us = 10000; + u16 reg, cap; + int rc; + + if (!pci_wait_for_pending_transaction(dev)) + pci_err(dev, "timed out waiting for pending transaction; performing cxl reset anyway\n"); + + /* Check if the device is cache capable. */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, &cap); + if (rc) + return rc; + + if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE)) + return 0; + + /* Disable cache. WB and invalidate cache if capability is advertised */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); + if (rc) + return rc; + reg |= PCI_DVSEC_CXL_DISABLE_CACHING; + /* + * DEVCTL2 bits are written only once. So check WB+I capability while + * keeping disable caching set. + */ + if (cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE) + reg |= PCI_DVSEC_CXL_INIT_CACHE_WBI; + pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); + + /* + * From Section 9.6: "Software may leverage the cache size reported in + * the DVSEC CXL Capability2 register to compute a suitable timeout + * value". + * Given there is no conversion factor for cache size -> timeout, + * setting timer for default 10ms. + */ + do { + if (timeout_tot_us == 0) + return -ETIMEDOUT; + usleep_range(timeout_us, timeout_us + 1); + timeout_tot_us -= timeout_us; + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, + ®); + if (rc) + return rc; + } while (!(reg & PCI_DVSEC_CXL_CACHE_INV)); + + return 0; +} + +static int cxl_reset_init(struct pci_dev *dev, u16 dvsec) +{ + /* + * Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers, + * under section 8.1.3.1 DVSEC CXL Capability. + */ + u32 reset_timeouts_ms[] = { 10, 100, 1000, 10000, 100000 }; + u16 reg; + u32 timeout_ms; + int rc, ind; + + /* Check if CXL Reset MEM CLR is supported. */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); + if (rc) + return rc; + + if (reg & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE) { + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, + ®); + if (rc) + return rc; + + reg |= PCI_DVSEC_CXL_RST_MEM_CLR_EN; + pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); + } + + /* Read timeout value. */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); + if (rc) + return rc; + ind = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, reg); + timeout_ms = reset_timeouts_ms[ind]; + + /* Write reset config. */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); + if (rc) + return rc; + + reg |= PCI_DVSEC_CXL_INIT_CXL_RST; + pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); + + /* Wait till timeout and then check reset status is complete. */ + msleep(timeout_ms); + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_STATUS2, ®); + if (rc) + return rc; + if (reg & PCI_DVSEC_CXL_RST_ERR || + ~reg & PCI_DVSEC_CXL_RST_DONE) + return -ETIMEDOUT; + + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); + if (rc) + return rc; + reg &= (~PCI_DVSEC_CXL_DISABLE_CACHING); + pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); + + return 0; +} + +/** + * cxl_reset - initiate a cxl reset + * @dev: device to reset + * @probe: if true, return 0 if device can be reset this way + * + * Initiate a cxl reset on @dev. + */ +static int cxl_reset(struct pci_dev *dev, bool probe) +{ + u16 dvsec, reg; + int rc; + + dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return -ENOTTY; + + /* Check if CXL Reset is supported. */ + rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); + if (rc) + return -ENOTTY; + + if ((reg & PCI_DVSEC_CXL_RST_CAPABLE) == 0) + return -ENOTTY; + + if (probe) + return 0; + + rc = cxl_reset_prepare(dev, dvsec); + if (rc) + return rc; + + return cxl_reset_init(dev, dvsec); +} + void pci_dev_lock(struct pci_dev *dev) { /* block PM suspend, driver probe, etc. */ @@ -5067,6 +5212,7 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, + { cxl_reset, .name = "cxl_reset" }, { pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" }, diff --git a/include/linux/pci.h b/include/linux/pci.h index 14bbdea91919e..9758af74a5452 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -51,7 +51,7 @@ PCI_STATUS_PARITY) /* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 8 +#define PCI_NUM_RESET_METHODS 9 #define PCI_RESET_PROBE true #define PCI_RESET_DO_RESET false diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 14f634ab9350d..e830dea141267 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1349,10 +1349,24 @@ /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */ #define PCI_DVSEC_CXL_DEVICE 0 #define PCI_DVSEC_CXL_CAP 0xA +#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0) #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2) #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11) #define PCI_DVSEC_CXL_CTRL 0xC #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) +#define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) +#define PCI_DVSEC_CXL_STATUS2 0x12 +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2) #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0) From 0249654d9e6ccee0049d2cc476b15eed4195e6ef Mon Sep 17 00:00:00 2001 From: Vishal Aslot Date: Tue, 14 Oct 2025 19:40:05 -0700 Subject: [PATCH 096/464] NVIDIA: VR: SAUCE: cxl_test: enable zero sized decoders under hb0 BugLink: https://bugs.launchpad.net/bugs/2138266 The cxl core in linux updated to supported committed decoders of zero size, because this is allowed by the CXL spec. This patch updates cxl_test to enable decoders 1 and 2 in the host-bridge 0 port, in a switch uport under hb0, and the endpoints ports with size zero simulating committed zero sized decoders. Signed-off-by: Vishal Aslot (backported from https://lore.kernel.org/all/20251015024019.1189713-1-vaslot@nvidia.com/) Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit a40b0390737baececc5d0b6ee4fb174516373ac9 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- tools/testing/cxl/test/cxl.c | 96 +++++++++++++++++++++++++++++++++++- 1 file changed, 94 insertions(+), 2 deletions(-) diff --git a/tools/testing/cxl/test/cxl.c b/tools/testing/cxl/test/cxl.c index f4c26441fc41a..60b52cd9d64db 100644 --- a/tools/testing/cxl/test/cxl.c +++ b/tools/testing/cxl/test/cxl.c @@ -759,6 +759,45 @@ static void default_mock_decoder(struct cxl_decoder *cxld) cxld->reset = mock_decoder_reset; } +static void size_zero_mock_decoder_ep(struct cxl_decoder *cxld, u64 base) +{ + struct cxl_endpoint_decoder *cxled; + + cxled = to_cxl_endpoint_decoder(&cxld->dev); + cxld->hpa_range = (struct range){ + .start = base, + .end = base - 1, /* Size 0 */ + }; + + cxld->interleave_ways = 2; + cxld->interleave_granularity = 4096; + cxld->target_type = CXL_DECODER_HOSTONLYMEM; + cxld->flags = CXL_DECODER_F_ENABLE; + cxled->state = CXL_DECODER_STATE_AUTO; + cxld->commit = mock_decoder_commit; + cxld->reset = mock_decoder_reset; +} + +static void size_zero_mock_decoder_sw(struct device *dev, u64 base, int i) +{ + struct cxl_switch_decoder *cxlsd; + struct cxl_decoder *cxld; + + cxlsd = to_cxl_switch_decoder(dev); + cxld = &cxlsd->cxld; + cxld->flags = CXL_DECODER_F_ENABLE; + cxld->target_type = CXL_DECODER_HOSTONLYMEM; + if (i == 0) + cxld->interleave_ways = 2; + else + cxld->interleave_ways = 1; + cxld->interleave_granularity = 4096; + cxld->hpa_range = (struct range) { + .start = base, + .end = base - 1, /* Size 0 */ + }; +} + static int first_decoder(struct device *dev, const void *data) { struct cxl_decoder *cxld; @@ -771,6 +810,30 @@ static int first_decoder(struct device *dev, const void *data) return 0; } +static int second_decoder(struct device *dev, const void *data) +{ + struct cxl_decoder *cxld; + + if (!is_switch_decoder(dev)) + return 0; + cxld = to_cxl_decoder(dev); + if (cxld->id == 1) + return 1; + return 0; +} + +static int third_decoder(struct device *dev, const void *data) +{ + struct cxl_decoder *cxld; + + if (!is_switch_decoder(dev)) + return 0; + cxld = to_cxl_decoder(dev); + if (cxld->id == 2) + return 1; + return 0; +} + static void mock_init_hdm_decoder(struct cxl_decoder *cxld) { struct acpi_cedt_cfmws *window = mock_cfmws[0]; @@ -782,7 +845,7 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld) struct cxl_dport *dport; struct device *dev; bool hb0 = false; - u64 base; + u64 base = window->base_hpa; int i; if (is_endpoint_decoder(&cxld->dev)) { @@ -806,6 +869,20 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld) port = cxled_to_port(cxled); } + /* + * Decoders 1 and 2 of the endpoint under host bridge 0 should be enabled as zero-sized. + * It would be even better to make sure that the parent switch uport decoder was + * also enabled before enabling the size zero decoders but there is no harm in doing it + * anyway. + */ + if (hb0 && (cxld->id == 1 || cxld->id == 2)) { + port = to_cxl_port(cxld->dev.parent); + size_zero_mock_decoder_ep(cxld, base); + /* Commit the zero-sized decoder */ + port->commit_end = cxld->id; + return; + } + /* * The first decoder on the first 2 devices on the first switch * attached to host-bridge0 mock a fake / static RAM region. All @@ -819,7 +896,6 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld) return; } - base = window->base_hpa; if (extended_linear_cache) base += mock_auto_region_size; cxld->hpa_range = (struct range) { @@ -886,6 +962,22 @@ static void mock_init_hdm_decoder(struct cxl_decoder *cxld) .end = base + mock_auto_region_size - 1, }; put_device(dev); + + /* Enable the next two decoders also and make them zero sized */ + dev = device_find_child(&iter->dev, NULL, second_decoder); + WARN_ON(!dev); + if (dev) { + size_zero_mock_decoder_sw(dev, base, i); + iter->commit_end = 1; + put_device(dev); + } + dev = device_find_child(&iter->dev, NULL, third_decoder); + WARN_ON(!dev); + if (dev) { + size_zero_mock_decoder_sw(dev, base, i); + iter->commit_end = 2; + put_device(dev); + } } } From 61c9ec84aa93f0c5d7d05952c28abf7b96a01745 Mon Sep 17 00:00:00 2001 From: Vishal Aslot Date: Tue, 14 Oct 2025 19:40:06 -0700 Subject: [PATCH 097/464] NVIDIA: VR: SAUCE: cxl: Allow zero sized HDM decoders BugLink: https://bugs.launchpad.net/bugs/2138266 CXL spec permits committing zero sized decoders. Linux currently considers them as an error. Zero-sized decoders are helpful when the BIOS is committing them. Often BIOS will also lock them to prevent them being changed due to the TSP requirement. For example, if the type 3 device is part of a TCB. The host bridge, switch, and end-point decoders can all be committed with zero-size. If they are locked along the VH, it is often to prevent hotplugging of a new device that could not be attested post boot and cannot be included in TCB. The caller leaves the decoder allocated but does not add it. It simply continues to the next decoder. Signed-off-by: Vishal Aslot (backported from https://lore.kernel.org/all/20251015024019.1189713-1-vaslot@nvidia.com/) Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit aa8224a0ffff58f91499084437badac56740dcc7 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/cxl/core/hdm.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index cb5d5a047a9da..64d96b3310e73 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -1031,13 +1031,14 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld, return -ENXIO; } + port->commit_end = cxld->id; + if (size == 0) { - dev_warn(&port->dev, + dev_dbg(&port->dev, "decoder%d.%d: Committed with zero size\n", port->id, cxld->id); - return -ENXIO; + return -ENOSPC; } - port->commit_end = cxld->id; } else { if (cxled) { struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); @@ -1193,6 +1194,8 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm, rc = init_hdm_decoder(port, cxld, hdm, i, &dpa_base, info); if (rc) { + if (rc == -ENOSPC) + continue; dev_warn(&port->dev, "Failed to initialize decoder%d.%d\n", port->id, i); From 84b4a9ddf2288194eb11f4df1a14c8f3ca5ae3c4 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Tue, 25 Nov 2025 13:07:35 +0000 Subject: [PATCH 098/464] NVIDIA: VR: SAUCE: cxl/region: Validate partition index before array access BugLink: https://bugs.launchpad.net/bugs/2138266 Check partition index bounds before accessing cxlds->part[] to prevent out-of-bounds when part is -1 or invalid. Fixes: 5ec67596e368) cxl/region: Drop goto pattern of construct_region() Signed-off-by: Koba Ko Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit d769d573d8adfcaa6c588b7f079b05962716316a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/cxl/core/region.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index c37ae0b28bbbc..9ad83813a5e60 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -3902,6 +3902,14 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, int rc, part = READ_ONCE(cxled->part); struct cxl_region *cxlr; + if (part < 0 || part >= cxlds->nr_partitions) { + dev_err(cxlmd->dev.parent, + "%s:%s: invalid partition index %d (max %u)\n", + dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), + part, cxlds->nr_partitions); + return ERR_PTR(-ENXIO); + } + do { cxlr = __create_region(cxlrd, cxlds->part[part].mode, atomic_read(&cxlrd->region_id)); From 834e27aedf17a83df7679232a3b34c55b780fd69 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 12 Jan 2026 11:14:02 -0800 Subject: [PATCH 099/464] NVIDIA: VR: SAUCE: [Config] Add a CXL config for CXL type 3 devices BugLink: https://bugs.launchpad.net/bugs/2138266 Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 60b3bd4564da77d31e508b9c88a7e69f49f1dd41 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 4d07edf8ec02a..fe8266527dfd2 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -105,6 +105,9 @@ CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'LP: #2028576: Perf governo CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL note<'LP: #2028576: Perf governor required for NVIDIA workloads'> +CONFIG_CXL_MEM_RAW_COMMANDS policy<{'amd64': 'n', 'arm64': 'y'}> +CONFIG_CXL_MEM_RAW_COMMANDS note<'Enable CXL raw commands for memory devices'> + CONFIG_DRM_NOUVEAU policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_DRM_NOUVEAU note<'Disable nouveau for NVIDIA kernels'> From 39ab9b2ff933f29321007ed0dc0c5c2d95295775 Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Wed, 7 Jan 2026 19:56:46 +0530 Subject: [PATCH 100/464] NVIDIA: VR: SAUCE: i2c: tegra: Introduce tegra_i2c_variant to identify DVC and VI BugLink: https://bugs.launchpad.net/bugs/2138238 Replace the per-instance boolean flags with an enum tegra_i2c_variant since DVC and VI are mutually exclusive. Update IS_DVC/IS_VI and variant initialization accordingly. Suggested-by: Jon Hunter Signed-off-by: Kartik Rajput (backported from https://lore.kernel.org/all/20260107142649.14917-1-kkartik@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 3d4a2674958462e3f4a9f5977d48255574211da6 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/i2c/busses/i2c-tegra.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c index 23c9b91b14dd8..396f5c9934fe8 100644 --- a/drivers/i2c/busses/i2c-tegra.c +++ b/drivers/i2c/busses/i2c-tegra.c @@ -171,6 +171,18 @@ enum msg_end_type { MSG_END_CONTINUE, }; +/* + * tegra_i2c_variant: Identifies the variant of I2C controller. + * @TEGRA_I2C_VARIANT_DEFAULT: Identifies the default I2C controller. + * @TEGRA_I2C_VARIANT_DVC: Identifies the DVC I2C controller, has a different register layout. + * @TEGRA_I2C_VARIANT_VI: Identifies the VI I2C controller, has a different register layout. + */ +enum tegra_i2c_variant { + TEGRA_I2C_VARIANT_DEFAULT, + TEGRA_I2C_VARIANT_DVC, + TEGRA_I2C_VARIANT_VI, +}; + /** * struct tegra_i2c_hw_feature : per hardware generation features * @has_continue_xfer_support: continue-transfer supported @@ -268,8 +280,7 @@ struct tegra_i2c_hw_feature { * @base_phys: physical base address of the I2C controller * @cont_id: I2C controller ID, used for packet header * @irq: IRQ number of transfer complete interrupt - * @is_dvc: identifies the DVC I2C controller, has a different register layout - * @is_vi: identifies the VI I2C controller, has a different register layout + * @variant: This represents the I2C controller variant. * @msg_complete: transfer completion notifier * @msg_buf_remaining: size of unsent data in the message buffer * @msg_len: length of message in current transfer @@ -321,12 +332,13 @@ struct tegra_i2c_dev { bool atomic_mode; bool dma_mode; bool msg_read; - bool is_dvc; - bool is_vi; + enum tegra_i2c_variant variant; }; -#define IS_DVC(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && (dev)->is_dvc) -#define IS_VI(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && (dev)->is_vi) +#define IS_DVC(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && \ + (dev)->variant == TEGRA_I2C_VARIANT_DVC) +#define IS_VI(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && \ + (dev)->variant == TEGRA_I2C_VARIANT_VI) static void dvc_writel(struct tegra_i2c_dev *i2c_dev, u32 val, unsigned int reg) @@ -1912,13 +1924,15 @@ static void tegra_i2c_parse_dt(struct tegra_i2c_dev *i2c_dev) multi_mode = device_property_read_bool(i2c_dev->dev, "multi-master"); i2c_dev->multimaster_mode = multi_mode; + i2c_dev->variant = TEGRA_I2C_VARIANT_DEFAULT; + if (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && of_device_is_compatible(np, "nvidia,tegra20-i2c-dvc")) - i2c_dev->is_dvc = true; + i2c_dev->variant = TEGRA_I2C_VARIANT_DVC; if (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && of_device_is_compatible(np, "nvidia,tegra210-i2c-vi")) - i2c_dev->is_vi = true; + i2c_dev->variant = TEGRA_I2C_VARIANT_VI; } static int tegra_i2c_init_clocks(struct tegra_i2c_dev *i2c_dev) From c61b80a67e53332c48edb4f1a7760acf35710dbd Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Wed, 7 Jan 2026 19:56:47 +0530 Subject: [PATCH 101/464] NVIDIA: VR: SAUCE: i2c: tegra: Move variant to tegra_i2c_hw_feature BugLink: https://bugs.launchpad.net/bugs/2138238 Move the variant field into tegra_i2c_hw_feature and populate it for all SoCs. Add dedicated SoC data for "nvidia,tegra20-i2c-dvc" and "nvidia,tegra210-i2c-vi" compatibles. Drop the compatible-string checks from tegra_i2c_parse_dt to initialize the Tegra I2C variant. Signed-off-by: Kartik Rajput (backported from https://lore.kernel.org/all/20260107142649.14917-1-kkartik@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit ad4783c1b992b8912089c1c1ad4848d6cd08ce4a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/i2c/busses/i2c-tegra.c | 98 ++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 17 deletions(-) diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c index 396f5c9934fe8..a59478cd5d33c 100644 --- a/drivers/i2c/busses/i2c-tegra.c +++ b/drivers/i2c/busses/i2c-tegra.c @@ -235,6 +235,7 @@ enum tegra_i2c_variant { * timing settings. * @enable_hs_mode_support: Enable support for high speed (HS) mode transfers. * @has_mutex: Has mutex register for mutual exclusion with other firmwares or VMs. + * @variant: This represents the I2C controller variant. */ struct tegra_i2c_hw_feature { bool has_continue_xfer_support; @@ -266,6 +267,7 @@ struct tegra_i2c_hw_feature { bool has_interface_timing_reg; bool enable_hs_mode_support; bool has_mutex; + enum tegra_i2c_variant variant; }; /** @@ -280,7 +282,6 @@ struct tegra_i2c_hw_feature { * @base_phys: physical base address of the I2C controller * @cont_id: I2C controller ID, used for packet header * @irq: IRQ number of transfer complete interrupt - * @variant: This represents the I2C controller variant. * @msg_complete: transfer completion notifier * @msg_buf_remaining: size of unsent data in the message buffer * @msg_len: length of message in current transfer @@ -332,13 +333,12 @@ struct tegra_i2c_dev { bool atomic_mode; bool dma_mode; bool msg_read; - enum tegra_i2c_variant variant; }; #define IS_DVC(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && \ - (dev)->variant == TEGRA_I2C_VARIANT_DVC) + (dev)->hw->variant == TEGRA_I2C_VARIANT_DVC) #define IS_VI(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && \ - (dev)->variant == TEGRA_I2C_VARIANT_VI) + (dev)->hw->variant == TEGRA_I2C_VARIANT_VI) static void dvc_writel(struct tegra_i2c_dev *i2c_dev, u32 val, unsigned int reg) @@ -1646,8 +1646,42 @@ static const struct tegra_i2c_hw_feature tegra20_i2c_hw = { .has_interface_timing_reg = false, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; +#if IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) +static const struct tegra_i2c_hw_feature tegra20_dvc_i2c_hw = { + .has_continue_xfer_support = false, + .has_per_pkt_xfer_complete_irq = false, + .clk_divisor_hs_mode = 3, + .clk_divisor_std_mode = 0, + .clk_divisor_fast_mode = 0, + .clk_divisor_fast_plus_mode = 0, + .has_config_load_reg = false, + .has_multi_master_mode = false, + .has_slcg_override_reg = false, + .has_mst_fifo = false, + .has_mst_reset = false, + .quirks = &tegra_i2c_quirks, + .supports_bus_clear = false, + .has_apb_dma = true, + .tlow_std_mode = 0x4, + .thigh_std_mode = 0x2, + .tlow_fast_mode = 0x4, + .thigh_fast_mode = 0x2, + .tlow_fastplus_mode = 0x4, + .thigh_fastplus_mode = 0x2, + .setup_hold_time_std_mode = 0x0, + .setup_hold_time_fast_mode = 0x0, + .setup_hold_time_fastplus_mode = 0x0, + .setup_hold_time_hs_mode = 0x0, + .has_interface_timing_reg = false, + .enable_hs_mode_support = false, + .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DVC, +}; +#endif + static const struct tegra_i2c_hw_feature tegra30_i2c_hw = { .has_continue_xfer_support = true, .has_per_pkt_xfer_complete_irq = false, @@ -1676,6 +1710,7 @@ static const struct tegra_i2c_hw_feature tegra30_i2c_hw = { .has_interface_timing_reg = false, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra114_i2c_hw = { @@ -1706,6 +1741,7 @@ static const struct tegra_i2c_hw_feature tegra114_i2c_hw = { .has_interface_timing_reg = false, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra124_i2c_hw = { @@ -1736,6 +1772,7 @@ static const struct tegra_i2c_hw_feature tegra124_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra210_i2c_hw = { @@ -1766,8 +1803,42 @@ static const struct tegra_i2c_hw_feature tegra210_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; +#if IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) +static const struct tegra_i2c_hw_feature tegra210_vi_i2c_hw = { + .has_continue_xfer_support = true, + .has_per_pkt_xfer_complete_irq = true, + .clk_divisor_hs_mode = 1, + .clk_divisor_std_mode = 0x19, + .clk_divisor_fast_mode = 0x19, + .clk_divisor_fast_plus_mode = 0x10, + .has_config_load_reg = true, + .has_multi_master_mode = false, + .has_slcg_override_reg = true, + .has_mst_fifo = false, + .has_mst_reset = false, + .quirks = &tegra_i2c_quirks, + .supports_bus_clear = true, + .has_apb_dma = true, + .tlow_std_mode = 0x4, + .thigh_std_mode = 0x2, + .tlow_fast_mode = 0x4, + .thigh_fast_mode = 0x2, + .tlow_fastplus_mode = 0x4, + .thigh_fastplus_mode = 0x2, + .setup_hold_time_std_mode = 0, + .setup_hold_time_fast_mode = 0, + .setup_hold_time_fastplus_mode = 0, + .setup_hold_time_hs_mode = 0, + .has_interface_timing_reg = true, + .enable_hs_mode_support = false, + .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_VI, +}; +#endif + static const struct tegra_i2c_hw_feature tegra186_i2c_hw = { .has_continue_xfer_support = true, .has_per_pkt_xfer_complete_irq = true, @@ -1796,6 +1867,7 @@ static const struct tegra_i2c_hw_feature tegra186_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = false, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra194_i2c_hw = { @@ -1828,6 +1900,7 @@ static const struct tegra_i2c_hw_feature tegra194_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = true, .has_mutex = false, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra256_i2c_hw = { @@ -1860,6 +1933,7 @@ static const struct tegra_i2c_hw_feature tegra256_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = true, .has_mutex = true, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct tegra_i2c_hw_feature tegra264_i2c_hw = { @@ -1892,6 +1966,7 @@ static const struct tegra_i2c_hw_feature tegra264_i2c_hw = { .has_interface_timing_reg = true, .enable_hs_mode_support = true, .has_mutex = true, + .variant = TEGRA_I2C_VARIANT_DEFAULT, }; static const struct of_device_id tegra_i2c_of_match[] = { @@ -1900,7 +1975,7 @@ static const struct of_device_id tegra_i2c_of_match[] = { { .compatible = "nvidia,tegra194-i2c", .data = &tegra194_i2c_hw, }, { .compatible = "nvidia,tegra186-i2c", .data = &tegra186_i2c_hw, }, #if IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) - { .compatible = "nvidia,tegra210-i2c-vi", .data = &tegra210_i2c_hw, }, + { .compatible = "nvidia,tegra210-i2c-vi", .data = &tegra210_vi_i2c_hw, }, #endif { .compatible = "nvidia,tegra210-i2c", .data = &tegra210_i2c_hw, }, { .compatible = "nvidia,tegra124-i2c", .data = &tegra124_i2c_hw, }, @@ -1908,7 +1983,7 @@ static const struct of_device_id tegra_i2c_of_match[] = { { .compatible = "nvidia,tegra30-i2c", .data = &tegra30_i2c_hw, }, { .compatible = "nvidia,tegra20-i2c", .data = &tegra20_i2c_hw, }, #if IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) - { .compatible = "nvidia,tegra20-i2c-dvc", .data = &tegra20_i2c_hw, }, + { .compatible = "nvidia,tegra20-i2c-dvc", .data = &tegra20_dvc_i2c_hw, }, #endif {}, }; @@ -1916,23 +1991,12 @@ MODULE_DEVICE_TABLE(of, tegra_i2c_of_match); static void tegra_i2c_parse_dt(struct tegra_i2c_dev *i2c_dev) { - struct device_node *np = i2c_dev->dev->of_node; bool multi_mode; i2c_parse_fw_timings(i2c_dev->dev, &i2c_dev->timings, true); multi_mode = device_property_read_bool(i2c_dev->dev, "multi-master"); i2c_dev->multimaster_mode = multi_mode; - - i2c_dev->variant = TEGRA_I2C_VARIANT_DEFAULT; - - if (IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) && - of_device_is_compatible(np, "nvidia,tegra20-i2c-dvc")) - i2c_dev->variant = TEGRA_I2C_VARIANT_DVC; - - if (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && - of_device_is_compatible(np, "nvidia,tegra210-i2c-vi")) - i2c_dev->variant = TEGRA_I2C_VARIANT_VI; } static int tegra_i2c_init_clocks(struct tegra_i2c_dev *i2c_dev) From 099f7a5a3341427bd8d4103b10a77500830f7f88 Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Wed, 7 Jan 2026 19:56:48 +0530 Subject: [PATCH 102/464] NVIDIA: VR: SAUCE: i2c: tegra: Add logic to support different register offsets BugLink: https://bugs.launchpad.net/bugs/2138238 Tegra410 use different offsets for existing I2C registers, update the logic to use appropriate offsets per SoC. As the registers offsets are now also defined for dvc and vi, following function are not required and they are removed: - tegra_i2c_reg_addr(): No translation required. - dvc_writel(): Replaced with i2c_writel() with DVC check. - dvc_readl(): Replaced with i2c_readl(). Signed-off-by: Kartik Rajput (backported from https://lore.kernel.org/all/20260107142649.14917-1-kkartik@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit f8e3f2341c695145bc45b71433775eea9a2ad9db noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/i2c/busses/i2c-tegra.c | 393 +++++++++++++++++++++------------ 1 file changed, 251 insertions(+), 142 deletions(-) diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c index a59478cd5d33c..07f8fa001fb2b 100644 --- a/drivers/i2c/busses/i2c-tegra.c +++ b/drivers/i2c/busses/i2c-tegra.c @@ -30,47 +30,37 @@ #define BYTES_PER_FIFO_WORD 4 -#define I2C_CNFG 0x000 #define I2C_CNFG_DEBOUNCE_CNT GENMASK(14, 12) #define I2C_CNFG_PACKET_MODE_EN BIT(10) #define I2C_CNFG_NEW_MASTER_FSM BIT(11) #define I2C_CNFG_MULTI_MASTER_MODE BIT(17) -#define I2C_STATUS 0x01c -#define I2C_SL_CNFG 0x020 + #define I2C_SL_CNFG_NACK BIT(1) #define I2C_SL_CNFG_NEWSL BIT(2) -#define I2C_SL_ADDR1 0x02c -#define I2C_SL_ADDR2 0x030 -#define I2C_TLOW_SEXT 0x034 -#define I2C_TX_FIFO 0x050 -#define I2C_RX_FIFO 0x054 -#define I2C_PACKET_TRANSFER_STATUS 0x058 -#define I2C_FIFO_CONTROL 0x05c + #define I2C_FIFO_CONTROL_TX_FLUSH BIT(1) #define I2C_FIFO_CONTROL_RX_FLUSH BIT(0) #define I2C_FIFO_CONTROL_TX_TRIG(x) (((x) - 1) << 5) #define I2C_FIFO_CONTROL_RX_TRIG(x) (((x) - 1) << 2) -#define I2C_FIFO_STATUS 0x060 + #define I2C_FIFO_STATUS_TX GENMASK(7, 4) #define I2C_FIFO_STATUS_RX GENMASK(3, 0) -#define I2C_INT_MASK 0x064 -#define I2C_INT_STATUS 0x068 + #define I2C_INT_BUS_CLR_DONE BIT(11) #define I2C_INT_PACKET_XFER_COMPLETE BIT(7) #define I2C_INT_NO_ACK BIT(3) #define I2C_INT_ARBITRATION_LOST BIT(2) #define I2C_INT_TX_FIFO_DATA_REQ BIT(1) #define I2C_INT_RX_FIFO_DATA_REQ BIT(0) -#define I2C_CLK_DIVISOR 0x06c + #define I2C_CLK_DIVISOR_STD_FAST_MODE GENMASK(31, 16) #define I2C_CLK_DIVISOR_HSMODE GENMASK(15, 0) -#define DVC_CTRL_REG1 0x000 #define DVC_CTRL_REG1_INTR_EN BIT(10) -#define DVC_CTRL_REG3 0x008 + #define DVC_CTRL_REG3_SW_PROG BIT(26) #define DVC_CTRL_REG3_I2C_DONE_INTR_EN BIT(30) -#define DVC_STATUS 0x00c + #define DVC_STATUS_I2C_DONE_INTR BIT(30) #define I2C_ERR_NONE 0x00 @@ -94,50 +84,38 @@ #define I2C_HEADER_CONTINUE_XFER BIT(15) #define I2C_HEADER_SLAVE_ADDR_SHIFT 1 -#define I2C_BUS_CLEAR_CNFG 0x084 #define I2C_BC_SCLK_THRESHOLD GENMASK(23, 16) #define I2C_BC_STOP_COND BIT(2) #define I2C_BC_TERMINATE BIT(1) #define I2C_BC_ENABLE BIT(0) -#define I2C_BUS_CLEAR_STATUS 0x088 + #define I2C_BC_STATUS BIT(0) -#define I2C_CONFIG_LOAD 0x08c #define I2C_MSTR_CONFIG_LOAD BIT(0) -#define I2C_CLKEN_OVERRIDE 0x090 #define I2C_MST_CORE_CLKEN_OVR BIT(0) -#define I2C_INTERFACE_TIMING_0 0x094 -#define I2C_INTERFACE_TIMING_THIGH GENMASK(13, 8) -#define I2C_INTERFACE_TIMING_TLOW GENMASK(5, 0) -#define I2C_INTERFACE_TIMING_1 0x098 -#define I2C_INTERFACE_TIMING_TBUF GENMASK(29, 24) -#define I2C_INTERFACE_TIMING_TSU_STO GENMASK(21, 16) -#define I2C_INTERFACE_TIMING_THD_STA GENMASK(13, 8) -#define I2C_INTERFACE_TIMING_TSU_STA GENMASK(5, 0) - -#define I2C_HS_INTERFACE_TIMING_0 0x09c -#define I2C_HS_INTERFACE_TIMING_THIGH GENMASK(13, 8) -#define I2C_HS_INTERFACE_TIMING_TLOW GENMASK(5, 0) -#define I2C_HS_INTERFACE_TIMING_1 0x0a0 -#define I2C_HS_INTERFACE_TIMING_TSU_STO GENMASK(21, 16) -#define I2C_HS_INTERFACE_TIMING_THD_STA GENMASK(13, 8) -#define I2C_HS_INTERFACE_TIMING_TSU_STA GENMASK(5, 0) - -#define I2C_MST_FIFO_CONTROL 0x0b4 +#define I2C_INTERFACE_TIMING_THIGH GENMASK(13, 8) +#define I2C_INTERFACE_TIMING_TLOW GENMASK(5, 0) +#define I2C_INTERFACE_TIMING_TBUF GENMASK(29, 24) +#define I2C_INTERFACE_TIMING_TSU_STO GENMASK(21, 16) +#define I2C_INTERFACE_TIMING_THD_STA GENMASK(13, 8) +#define I2C_INTERFACE_TIMING_TSU_STA GENMASK(5, 0) + +#define I2C_HS_INTERFACE_TIMING_THIGH GENMASK(13, 8) +#define I2C_HS_INTERFACE_TIMING_TLOW GENMASK(5, 0) +#define I2C_HS_INTERFACE_TIMING_TSU_STO GENMASK(21, 16) +#define I2C_HS_INTERFACE_TIMING_THD_STA GENMASK(13, 8) +#define I2C_HS_INTERFACE_TIMING_TSU_STA GENMASK(5, 0) + #define I2C_MST_FIFO_CONTROL_RX_FLUSH BIT(0) #define I2C_MST_FIFO_CONTROL_TX_FLUSH BIT(1) #define I2C_MST_FIFO_CONTROL_RX_TRIG(x) (((x) - 1) << 4) #define I2C_MST_FIFO_CONTROL_TX_TRIG(x) (((x) - 1) << 16) -#define I2C_MST_FIFO_STATUS 0x0b8 #define I2C_MST_FIFO_STATUS_TX GENMASK(23, 16) #define I2C_MST_FIFO_STATUS_RX GENMASK(7, 0) -#define I2C_MASTER_RESET_CNTRL 0x0a8 - -#define I2C_SW_MUTEX 0x0ec #define I2C_SW_MUTEX_REQUEST GENMASK(3, 0) #define I2C_SW_MUTEX_GRANT GENMASK(7, 4) #define I2C_SW_MUTEX_ID_CCPLEX 9 @@ -159,6 +137,149 @@ */ #define I2C_PIO_MODE_PREFERRED_LEN 32 +struct tegra_i2c_regs { + unsigned int cnfg; + unsigned int status; + unsigned int sl_cnfg; + unsigned int sl_addr1; + unsigned int sl_addr2; + unsigned int tlow_sext; + unsigned int tx_fifo; + unsigned int rx_fifo; + unsigned int packet_transfer_status; + unsigned int fifo_control; + unsigned int fifo_status; + unsigned int int_mask; + unsigned int int_status; + unsigned int clk_divisor; + unsigned int bus_clear_cnfg; + unsigned int bus_clear_status; + unsigned int config_load; + unsigned int clken_override; + unsigned int interface_timing_0; + unsigned int interface_timing_1; + unsigned int hs_interface_timing_0; + unsigned int hs_interface_timing_1; + unsigned int master_reset_cntrl; + unsigned int mst_fifo_control; + unsigned int mst_fifo_status; + unsigned int sw_mutex; + unsigned int dvc_ctrl_reg1; + unsigned int dvc_ctrl_reg3; + unsigned int dvc_status; +}; + +static const struct tegra_i2c_regs tegra20_i2c_regs = { + .cnfg = 0x000, + .status = 0x01c, + .sl_cnfg = 0x020, + .sl_addr1 = 0x02c, + .sl_addr2 = 0x030, + .tx_fifo = 0x050, + .rx_fifo = 0x054, + .packet_transfer_status = 0x058, + .fifo_control = 0x05c, + .fifo_status = 0x060, + .int_mask = 0x064, + .int_status = 0x068, + .clk_divisor = 0x06c, + .bus_clear_cnfg = 0x084, + .bus_clear_status = 0x088, + .config_load = 0x08c, + .clken_override = 0x090, + .interface_timing_0 = 0x094, + .interface_timing_1 = 0x098, + .hs_interface_timing_0 = 0x09c, + .hs_interface_timing_1 = 0x0a0, + .master_reset_cntrl = 0x0a8, + .mst_fifo_control = 0x0b4, + .mst_fifo_status = 0x0b8, +}; + +#if IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) +static const struct tegra_i2c_regs tegra20_dvc_i2c_regs = { + .dvc_ctrl_reg1 = 0x000, + .dvc_ctrl_reg3 = 0x008, + .dvc_status = 0x00c, + .cnfg = 0x040, + .status = 0x05c, + .tx_fifo = 0x060, + .rx_fifo = 0x064, + .packet_transfer_status = 0x068, + .fifo_control = 0x06c, + .fifo_status = 0x070, + .int_mask = 0x074, + .int_status = 0x078, + .clk_divisor = 0x07c, + .bus_clear_cnfg = 0x0c4, + .bus_clear_status = 0x0c8, + .config_load = 0x0cc, + .clken_override = 0x0d0, + .interface_timing_0 = 0x0d4, + .interface_timing_1 = 0x0d8, + .hs_interface_timing_0 = 0x0dc, + .hs_interface_timing_1 = 0x0e0, + .master_reset_cntrl = 0x0e8, + .mst_fifo_control = 0x0c4, + .mst_fifo_status = 0x0c8, +}; +#endif + +#if IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) +static const struct tegra_i2c_regs tegra210_vi_i2c_regs = { + .cnfg = 0x0c00, + .status = 0x0c70, + .tlow_sext = 0x0cd0, + .tx_fifo = 0x0d40, + .rx_fifo = 0x0d50, + .packet_transfer_status = 0x0d60, + .fifo_control = 0x0d70, + .fifo_status = 0x0d80, + .int_mask = 0x0d90, + .int_status = 0x0da0, + .clk_divisor = 0x0db0, + .bus_clear_cnfg = 0x0e10, + .bus_clear_status = 0x0e20, + .config_load = 0x0e30, + .clken_override = 0x0e40, + .interface_timing_0 = 0x0e50, + .interface_timing_1 = 0x0e60, + .hs_interface_timing_0 = 0x0e70, + .hs_interface_timing_1 = 0x0e80, + .master_reset_cntrl = 0x0ea0, + .mst_fifo_control = 0x0ed0, + .mst_fifo_status = 0x0ee0, +}; +#endif + +static const struct tegra_i2c_regs tegra264_i2c_regs = { + .cnfg = 0x000, + .status = 0x01c, + .sl_cnfg = 0x020, + .sl_addr1 = 0x02c, + .sl_addr2 = 0x030, + .tx_fifo = 0x050, + .rx_fifo = 0x054, + .packet_transfer_status = 0x058, + .fifo_control = 0x05c, + .fifo_status = 0x060, + .int_mask = 0x064, + .int_status = 0x068, + .clk_divisor = 0x06c, + .bus_clear_cnfg = 0x084, + .bus_clear_status = 0x088, + .config_load = 0x08c, + .clken_override = 0x090, + .interface_timing_0 = 0x094, + .interface_timing_1 = 0x098, + .hs_interface_timing_0 = 0x09c, + .hs_interface_timing_1 = 0x0a0, + .master_reset_cntrl = 0x0a8, + .mst_fifo_control = 0x0b4, + .mst_fifo_status = 0x0b8, + .sw_mutex = 0x0ec, +}; + /* * msg_end_type: The bus control which needs to be sent at end of transfer. * @MSG_END_STOP: Send stop pulse. @@ -236,6 +357,7 @@ enum tegra_i2c_variant { * @enable_hs_mode_support: Enable support for high speed (HS) mode transfers. * @has_mutex: Has mutex register for mutual exclusion with other firmwares or VMs. * @variant: This represents the I2C controller variant. + * @regs: Register offsets for the specific SoC variant. */ struct tegra_i2c_hw_feature { bool has_continue_xfer_support; @@ -268,6 +390,7 @@ struct tegra_i2c_hw_feature { bool enable_hs_mode_support; bool has_mutex; enum tegra_i2c_variant variant; + const struct tegra_i2c_regs *regs; }; /** @@ -340,51 +463,26 @@ struct tegra_i2c_dev { #define IS_VI(dev) (IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) && \ (dev)->hw->variant == TEGRA_I2C_VARIANT_VI) -static void dvc_writel(struct tegra_i2c_dev *i2c_dev, u32 val, - unsigned int reg) -{ - writel_relaxed(val, i2c_dev->base + reg); -} - -static u32 dvc_readl(struct tegra_i2c_dev *i2c_dev, unsigned int reg) -{ - return readl_relaxed(i2c_dev->base + reg); -} - -/* - * If necessary, i2c_writel() and i2c_readl() will offset the register - * in order to talk to the I2C block inside the DVC block. - */ -static u32 tegra_i2c_reg_addr(struct tegra_i2c_dev *i2c_dev, unsigned int reg) -{ - if (IS_DVC(i2c_dev)) - reg += (reg >= I2C_TX_FIFO) ? 0x10 : 0x40; - else if (IS_VI(i2c_dev)) - reg = 0xc00 + (reg << 2); - - return reg; -} - static void i2c_writel(struct tegra_i2c_dev *i2c_dev, u32 val, unsigned int reg) { - writel_relaxed(val, i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg)); + writel_relaxed(val, i2c_dev->base + reg); /* read back register to make sure that register writes completed */ - if (reg != I2C_TX_FIFO) - readl_relaxed(i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg)); + if (!IS_DVC(i2c_dev) && reg != i2c_dev->hw->regs->tx_fifo) + readl_relaxed(i2c_dev->base + reg); else if (IS_VI(i2c_dev)) - readl_relaxed(i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, I2C_INT_STATUS)); + readl_relaxed(i2c_dev->base + i2c_dev->hw->regs->int_status); } static u32 i2c_readl(struct tegra_i2c_dev *i2c_dev, unsigned int reg) { - return readl_relaxed(i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg)); + return readl_relaxed(i2c_dev->base + reg); } static void i2c_writesl(struct tegra_i2c_dev *i2c_dev, void *data, unsigned int reg, unsigned int len) { - writesl(i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg), data, len); + writesl(i2c_dev->base + reg, data, len); } static void i2c_writesl_vi(struct tegra_i2c_dev *i2c_dev, void *data, @@ -405,12 +503,12 @@ static void i2c_writesl_vi(struct tegra_i2c_dev *i2c_dev, void *data, static void i2c_readsl(struct tegra_i2c_dev *i2c_dev, void *data, unsigned int reg, unsigned int len) { - readsl(i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg), data, len); + readsl(i2c_dev->base + reg, data, len); } static bool tegra_i2c_mutex_acquired(struct tegra_i2c_dev *i2c_dev) { - unsigned int reg = tegra_i2c_reg_addr(i2c_dev, I2C_SW_MUTEX); + unsigned int reg = i2c_dev->hw->regs->sw_mutex; u32 val, id; val = readl(i2c_dev->base + reg); @@ -421,7 +519,7 @@ static bool tegra_i2c_mutex_acquired(struct tegra_i2c_dev *i2c_dev) static bool tegra_i2c_mutex_trylock(struct tegra_i2c_dev *i2c_dev) { - unsigned int reg = tegra_i2c_reg_addr(i2c_dev, I2C_SW_MUTEX); + unsigned int reg = i2c_dev->hw->regs->sw_mutex; u32 val, id; val = readl(i2c_dev->base + reg); @@ -459,7 +557,7 @@ static int tegra_i2c_mutex_lock(struct tegra_i2c_dev *i2c_dev) static void tegra_i2c_mutex_unlock(struct tegra_i2c_dev *i2c_dev) { - unsigned int reg = tegra_i2c_reg_addr(i2c_dev, I2C_SW_MUTEX); + unsigned int reg = i2c_dev->hw->regs->sw_mutex; u32 val, id; if (!i2c_dev->hw->has_mutex) @@ -479,16 +577,16 @@ static void tegra_i2c_mask_irq(struct tegra_i2c_dev *i2c_dev, u32 mask) { u32 int_mask; - int_mask = i2c_readl(i2c_dev, I2C_INT_MASK) & ~mask; - i2c_writel(i2c_dev, int_mask, I2C_INT_MASK); + int_mask = i2c_readl(i2c_dev, i2c_dev->hw->regs->int_mask) & ~mask; + i2c_writel(i2c_dev, int_mask, i2c_dev->hw->regs->int_mask); } static void tegra_i2c_unmask_irq(struct tegra_i2c_dev *i2c_dev, u32 mask) { u32 int_mask; - int_mask = i2c_readl(i2c_dev, I2C_INT_MASK) | mask; - i2c_writel(i2c_dev, int_mask, I2C_INT_MASK); + int_mask = i2c_readl(i2c_dev, i2c_dev->hw->regs->int_mask) | mask; + i2c_writel(i2c_dev, int_mask, i2c_dev->hw->regs->int_mask); } static void tegra_i2c_dma_complete(void *args) @@ -616,14 +714,14 @@ static void tegra_dvc_init(struct tegra_i2c_dev *i2c_dev) { u32 val; - val = dvc_readl(i2c_dev, DVC_CTRL_REG3); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->dvc_ctrl_reg3); val |= DVC_CTRL_REG3_SW_PROG; val |= DVC_CTRL_REG3_I2C_DONE_INTR_EN; - dvc_writel(i2c_dev, val, DVC_CTRL_REG3); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->dvc_ctrl_reg3); - val = dvc_readl(i2c_dev, DVC_CTRL_REG1); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->dvc_ctrl_reg1); val |= DVC_CTRL_REG1_INTR_EN; - dvc_writel(i2c_dev, val, DVC_CTRL_REG1); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->dvc_ctrl_reg1); } static void tegra_i2c_vi_init(struct tegra_i2c_dev *i2c_dev) @@ -632,34 +730,34 @@ static void tegra_i2c_vi_init(struct tegra_i2c_dev *i2c_dev) value = FIELD_PREP(I2C_INTERFACE_TIMING_THIGH, 2) | FIELD_PREP(I2C_INTERFACE_TIMING_TLOW, 4); - i2c_writel(i2c_dev, value, I2C_INTERFACE_TIMING_0); + i2c_writel(i2c_dev, value, i2c_dev->hw->regs->interface_timing_0); value = FIELD_PREP(I2C_INTERFACE_TIMING_TBUF, 4) | FIELD_PREP(I2C_INTERFACE_TIMING_TSU_STO, 7) | FIELD_PREP(I2C_INTERFACE_TIMING_THD_STA, 4) | FIELD_PREP(I2C_INTERFACE_TIMING_TSU_STA, 4); - i2c_writel(i2c_dev, value, I2C_INTERFACE_TIMING_1); + i2c_writel(i2c_dev, value, i2c_dev->hw->regs->interface_timing_1); value = FIELD_PREP(I2C_HS_INTERFACE_TIMING_THIGH, 3) | FIELD_PREP(I2C_HS_INTERFACE_TIMING_TLOW, 8); - i2c_writel(i2c_dev, value, I2C_HS_INTERFACE_TIMING_0); + i2c_writel(i2c_dev, value, i2c_dev->hw->regs->hs_interface_timing_0); value = FIELD_PREP(I2C_HS_INTERFACE_TIMING_TSU_STO, 11) | FIELD_PREP(I2C_HS_INTERFACE_TIMING_THD_STA, 11) | FIELD_PREP(I2C_HS_INTERFACE_TIMING_TSU_STA, 11); - i2c_writel(i2c_dev, value, I2C_HS_INTERFACE_TIMING_1); + i2c_writel(i2c_dev, value, i2c_dev->hw->regs->hs_interface_timing_1); value = FIELD_PREP(I2C_BC_SCLK_THRESHOLD, 9) | I2C_BC_STOP_COND; - i2c_writel(i2c_dev, value, I2C_BUS_CLEAR_CNFG); + i2c_writel(i2c_dev, value, i2c_dev->hw->regs->bus_clear_cnfg); - i2c_writel(i2c_dev, 0x0, I2C_TLOW_SEXT); + i2c_writel(i2c_dev, 0x0, i2c_dev->hw->regs->tlow_sext); } static int tegra_i2c_poll_register(struct tegra_i2c_dev *i2c_dev, u32 reg, u32 mask, u32 delay_us, u32 timeout_us) { - void __iomem *addr = i2c_dev->base + tegra_i2c_reg_addr(i2c_dev, reg); + void __iomem *addr = i2c_dev->base + reg; u32 val; if (!i2c_dev->atomic_mode) @@ -678,11 +776,11 @@ static int tegra_i2c_flush_fifos(struct tegra_i2c_dev *i2c_dev) if (i2c_dev->hw->has_mst_fifo) { mask = I2C_MST_FIFO_CONTROL_TX_FLUSH | I2C_MST_FIFO_CONTROL_RX_FLUSH; - offset = I2C_MST_FIFO_CONTROL; + offset = i2c_dev->hw->regs->mst_fifo_control; } else { mask = I2C_FIFO_CONTROL_TX_FLUSH | I2C_FIFO_CONTROL_RX_FLUSH; - offset = I2C_FIFO_CONTROL; + offset = i2c_dev->hw->regs->fifo_control; } val = i2c_readl(i2c_dev, offset); @@ -705,9 +803,9 @@ static int tegra_i2c_wait_for_config_load(struct tegra_i2c_dev *i2c_dev) if (!i2c_dev->hw->has_config_load_reg) return 0; - i2c_writel(i2c_dev, I2C_MSTR_CONFIG_LOAD, I2C_CONFIG_LOAD); + i2c_writel(i2c_dev, I2C_MSTR_CONFIG_LOAD, i2c_dev->hw->regs->config_load); - err = tegra_i2c_poll_register(i2c_dev, I2C_CONFIG_LOAD, 0xffffffff, + err = tegra_i2c_poll_register(i2c_dev, i2c_dev->hw->regs->config_load, 0xffffffff, 1000, I2C_CONFIG_LOAD_TIMEOUT); if (err) { dev_err(i2c_dev->dev, "failed to load config\n"); @@ -728,10 +826,10 @@ static int tegra_i2c_master_reset(struct tegra_i2c_dev *i2c_dev) * SW needs to wait for 2us after assertion and de-assertion of this soft * reset. */ - i2c_writel(i2c_dev, 0x1, I2C_MASTER_RESET_CNTRL); + i2c_writel(i2c_dev, 0x1, i2c_dev->hw->regs->master_reset_cntrl); fsleep(2); - i2c_writel(i2c_dev, 0x0, I2C_MASTER_RESET_CNTRL); + i2c_writel(i2c_dev, 0x0, i2c_dev->hw->regs->master_reset_cntrl); fsleep(2); return 0; @@ -773,8 +871,8 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) if (i2c_dev->hw->has_multi_master_mode) val |= I2C_CNFG_MULTI_MASTER_MODE; - i2c_writel(i2c_dev, val, I2C_CNFG); - i2c_writel(i2c_dev, 0, I2C_INT_MASK); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->cnfg); + i2c_writel(i2c_dev, 0, i2c_dev->hw->regs->int_mask); if (IS_VI(i2c_dev)) tegra_i2c_vi_init(i2c_dev); @@ -819,12 +917,12 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) clk_divisor = FIELD_PREP(I2C_CLK_DIVISOR_HSMODE, i2c_dev->hw->clk_divisor_hs_mode) | FIELD_PREP(I2C_CLK_DIVISOR_STD_FAST_MODE, non_hs_mode); - i2c_writel(i2c_dev, clk_divisor, I2C_CLK_DIVISOR); + i2c_writel(i2c_dev, clk_divisor, i2c_dev->hw->regs->clk_divisor); if (i2c_dev->hw->has_interface_timing_reg) { val = FIELD_PREP(I2C_INTERFACE_TIMING_THIGH, thigh) | FIELD_PREP(I2C_INTERFACE_TIMING_TLOW, tlow); - i2c_writel(i2c_dev, val, I2C_INTERFACE_TIMING_0); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->interface_timing_0); } /* @@ -832,7 +930,7 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) * Otherwise, preserve the chip default values. */ if (i2c_dev->hw->has_interface_timing_reg && tsu_thd) - i2c_writel(i2c_dev, tsu_thd, I2C_INTERFACE_TIMING_1); + i2c_writel(i2c_dev, tsu_thd, i2c_dev->hw->regs->interface_timing_1); /* Write HS mode registers. These will get used only for HS mode*/ if (i2c_dev->hw->enable_hs_mode_support) { @@ -842,8 +940,8 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) val = FIELD_PREP(I2C_HS_INTERFACE_TIMING_THIGH, thigh) | FIELD_PREP(I2C_HS_INTERFACE_TIMING_TLOW, tlow); - i2c_writel(i2c_dev, val, I2C_HS_INTERFACE_TIMING_0); - i2c_writel(i2c_dev, tsu_thd, I2C_HS_INTERFACE_TIMING_1); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->hs_interface_timing_0); + i2c_writel(i2c_dev, tsu_thd, i2c_dev->hw->regs->hs_interface_timing_1); } clk_multiplier = (tlow + thigh + 2) * (non_hs_mode + 1); @@ -856,12 +954,12 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) } if (!IS_DVC(i2c_dev) && !IS_VI(i2c_dev)) { - u32 sl_cfg = i2c_readl(i2c_dev, I2C_SL_CNFG); + u32 sl_cfg = i2c_readl(i2c_dev, i2c_dev->hw->regs->sl_cnfg); sl_cfg |= I2C_SL_CNFG_NACK | I2C_SL_CNFG_NEWSL; - i2c_writel(i2c_dev, sl_cfg, I2C_SL_CNFG); - i2c_writel(i2c_dev, 0xfc, I2C_SL_ADDR1); - i2c_writel(i2c_dev, 0x00, I2C_SL_ADDR2); + i2c_writel(i2c_dev, sl_cfg, i2c_dev->hw->regs->sl_cnfg); + i2c_writel(i2c_dev, 0xfc, i2c_dev->hw->regs->sl_addr1); + i2c_writel(i2c_dev, 0x00, i2c_dev->hw->regs->sl_addr2); } err = tegra_i2c_flush_fifos(i2c_dev); @@ -869,7 +967,7 @@ static int tegra_i2c_init(struct tegra_i2c_dev *i2c_dev) return err; if (i2c_dev->multimaster_mode && i2c_dev->hw->has_slcg_override_reg) - i2c_writel(i2c_dev, I2C_MST_CORE_CLKEN_OVR, I2C_CLKEN_OVERRIDE); + i2c_writel(i2c_dev, I2C_MST_CORE_CLKEN_OVR, i2c_dev->hw->regs->clken_override); err = tegra_i2c_wait_for_config_load(i2c_dev); if (err) @@ -890,9 +988,9 @@ static int tegra_i2c_disable_packet_mode(struct tegra_i2c_dev *i2c_dev) */ udelay(DIV_ROUND_UP(2 * 1000000, i2c_dev->timings.bus_freq_hz)); - cnfg = i2c_readl(i2c_dev, I2C_CNFG); + cnfg = i2c_readl(i2c_dev, i2c_dev->hw->regs->cnfg); if (cnfg & I2C_CNFG_PACKET_MODE_EN) - i2c_writel(i2c_dev, cnfg & ~I2C_CNFG_PACKET_MODE_EN, I2C_CNFG); + i2c_writel(i2c_dev, cnfg & ~I2C_CNFG_PACKET_MODE_EN, i2c_dev->hw->regs->cnfg); return tegra_i2c_wait_for_config_load(i2c_dev); } @@ -912,10 +1010,10 @@ static int tegra_i2c_empty_rx_fifo(struct tegra_i2c_dev *i2c_dev) return -EINVAL; if (i2c_dev->hw->has_mst_fifo) { - val = i2c_readl(i2c_dev, I2C_MST_FIFO_STATUS); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->mst_fifo_status); rx_fifo_avail = FIELD_GET(I2C_MST_FIFO_STATUS_RX, val); } else { - val = i2c_readl(i2c_dev, I2C_FIFO_STATUS); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->fifo_status); rx_fifo_avail = FIELD_GET(I2C_FIFO_STATUS_RX, val); } @@ -924,7 +1022,7 @@ static int tegra_i2c_empty_rx_fifo(struct tegra_i2c_dev *i2c_dev) if (words_to_transfer > rx_fifo_avail) words_to_transfer = rx_fifo_avail; - i2c_readsl(i2c_dev, buf, I2C_RX_FIFO, words_to_transfer); + i2c_readsl(i2c_dev, buf, i2c_dev->hw->regs->rx_fifo, words_to_transfer); buf += words_to_transfer * BYTES_PER_FIFO_WORD; buf_remaining -= words_to_transfer * BYTES_PER_FIFO_WORD; @@ -940,7 +1038,7 @@ static int tegra_i2c_empty_rx_fifo(struct tegra_i2c_dev *i2c_dev) * when (words_to_transfer was > rx_fifo_avail) earlier * in this function. */ - val = i2c_readl(i2c_dev, I2C_RX_FIFO); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->rx_fifo); val = cpu_to_le32(val); memcpy(buf, &val, buf_remaining); buf_remaining = 0; @@ -965,10 +1063,10 @@ static int tegra_i2c_fill_tx_fifo(struct tegra_i2c_dev *i2c_dev) u32 val; if (i2c_dev->hw->has_mst_fifo) { - val = i2c_readl(i2c_dev, I2C_MST_FIFO_STATUS); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->mst_fifo_status); tx_fifo_avail = FIELD_GET(I2C_MST_FIFO_STATUS_TX, val); } else { - val = i2c_readl(i2c_dev, I2C_FIFO_STATUS); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->fifo_status); tx_fifo_avail = FIELD_GET(I2C_FIFO_STATUS_TX, val); } @@ -999,9 +1097,9 @@ static int tegra_i2c_fill_tx_fifo(struct tegra_i2c_dev *i2c_dev) i2c_dev->msg_buf = buf + words_to_transfer * BYTES_PER_FIFO_WORD; if (IS_VI(i2c_dev)) - i2c_writesl_vi(i2c_dev, buf, I2C_TX_FIFO, words_to_transfer); + i2c_writesl_vi(i2c_dev, buf, i2c_dev->hw->regs->tx_fifo, words_to_transfer); else - i2c_writesl(i2c_dev, buf, I2C_TX_FIFO, words_to_transfer); + i2c_writesl(i2c_dev, buf, i2c_dev->hw->regs->tx_fifo, words_to_transfer); buf += words_to_transfer * BYTES_PER_FIFO_WORD; } @@ -1023,7 +1121,7 @@ static int tegra_i2c_fill_tx_fifo(struct tegra_i2c_dev *i2c_dev) i2c_dev->msg_buf_remaining = 0; i2c_dev->msg_buf = NULL; - i2c_writel(i2c_dev, val, I2C_TX_FIFO); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->tx_fifo); } return 0; @@ -1035,13 +1133,13 @@ static irqreturn_t tegra_i2c_isr(int irq, void *dev_id) struct tegra_i2c_dev *i2c_dev = dev_id; u32 status; - status = i2c_readl(i2c_dev, I2C_INT_STATUS); + status = i2c_readl(i2c_dev, i2c_dev->hw->regs->int_status); if (status == 0) { dev_warn(i2c_dev->dev, "IRQ status 0 %08x %08x %08x\n", - i2c_readl(i2c_dev, I2C_PACKET_TRANSFER_STATUS), - i2c_readl(i2c_dev, I2C_STATUS), - i2c_readl(i2c_dev, I2C_CNFG)); + i2c_readl(i2c_dev, i2c_dev->hw->regs->packet_transfer_status), + i2c_readl(i2c_dev, i2c_dev->hw->regs->status), + i2c_readl(i2c_dev, i2c_dev->hw->regs->cnfg)); i2c_dev->msg_err |= I2C_ERR_UNKNOWN_INTERRUPT; goto err; } @@ -1084,9 +1182,9 @@ static irqreturn_t tegra_i2c_isr(int irq, void *dev_id) } } - i2c_writel(i2c_dev, status, I2C_INT_STATUS); + i2c_writel(i2c_dev, status, i2c_dev->hw->regs->int_status); if (IS_DVC(i2c_dev)) - dvc_writel(i2c_dev, DVC_STATUS_I2C_DONE_INTR, DVC_STATUS); + i2c_writel(i2c_dev, DVC_STATUS_I2C_DONE_INTR, i2c_dev->hw->regs->dvc_status); /* * During message read XFER_COMPLETE interrupt is triggered prior to @@ -1122,10 +1220,10 @@ static irqreturn_t tegra_i2c_isr(int irq, void *dev_id) if (i2c_dev->hw->supports_bus_clear) tegra_i2c_mask_irq(i2c_dev, I2C_INT_BUS_CLR_DONE); - i2c_writel(i2c_dev, status, I2C_INT_STATUS); + i2c_writel(i2c_dev, status, i2c_dev->hw->regs->int_status); if (IS_DVC(i2c_dev)) - dvc_writel(i2c_dev, DVC_STATUS_I2C_DONE_INTR, DVC_STATUS); + i2c_writel(i2c_dev, DVC_STATUS_I2C_DONE_INTR, i2c_dev->hw->regs->dvc_status); if (i2c_dev->dma_mode) { dmaengine_terminate_async(i2c_dev->dma_chan); @@ -1145,9 +1243,9 @@ static void tegra_i2c_config_fifo_trig(struct tegra_i2c_dev *i2c_dev, int err; if (i2c_dev->hw->has_mst_fifo) - reg = I2C_MST_FIFO_CONTROL; + reg = i2c_dev->hw->regs->mst_fifo_control; else - reg = I2C_FIFO_CONTROL; + reg = i2c_dev->hw->regs->fifo_control; if (i2c_dev->dma_mode) { if (len & 0xF) @@ -1158,7 +1256,7 @@ static void tegra_i2c_config_fifo_trig(struct tegra_i2c_dev *i2c_dev, dma_burst = 8; if (i2c_dev->msg_read) { - reg_offset = tegra_i2c_reg_addr(i2c_dev, I2C_RX_FIFO); + reg_offset = i2c_dev->hw->regs->rx_fifo; slv_config.src_addr = i2c_dev->base_phys + reg_offset; slv_config.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; @@ -1169,7 +1267,7 @@ static void tegra_i2c_config_fifo_trig(struct tegra_i2c_dev *i2c_dev, else val = I2C_FIFO_CONTROL_RX_TRIG(dma_burst); } else { - reg_offset = tegra_i2c_reg_addr(i2c_dev, I2C_TX_FIFO); + reg_offset = i2c_dev->hw->regs->tx_fifo; slv_config.dst_addr = i2c_dev->base_phys + reg_offset; slv_config.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; @@ -1212,7 +1310,7 @@ static unsigned long tegra_i2c_poll_completion(struct tegra_i2c_dev *i2c_dev, ktime_t ktimeout = ktime_add_ms(ktime, timeout_ms); do { - u32 status = i2c_readl(i2c_dev, I2C_INT_STATUS); + u32 status = i2c_readl(i2c_dev, i2c_dev->hw->regs->int_status); if (status) tegra_i2c_isr(i2c_dev->irq, i2c_dev); @@ -1271,14 +1369,14 @@ static int tegra_i2c_issue_bus_clear(struct i2c_adapter *adap) val = FIELD_PREP(I2C_BC_SCLK_THRESHOLD, 9) | I2C_BC_STOP_COND | I2C_BC_TERMINATE; - i2c_writel(i2c_dev, val, I2C_BUS_CLEAR_CNFG); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->bus_clear_cnfg); err = tegra_i2c_wait_for_config_load(i2c_dev); if (err) return err; val |= I2C_BC_ENABLE; - i2c_writel(i2c_dev, val, I2C_BUS_CLEAR_CNFG); + i2c_writel(i2c_dev, val, i2c_dev->hw->regs->bus_clear_cnfg); tegra_i2c_unmask_irq(i2c_dev, I2C_INT_BUS_CLR_DONE); time_left = tegra_i2c_wait_completion(i2c_dev, &i2c_dev->msg_complete, 50); @@ -1289,7 +1387,7 @@ static int tegra_i2c_issue_bus_clear(struct i2c_adapter *adap) return -ETIMEDOUT; } - val = i2c_readl(i2c_dev, I2C_BUS_CLEAR_STATUS); + val = i2c_readl(i2c_dev, i2c_dev->hw->regs->bus_clear_status); if (!(val & I2C_BC_STATUS)) { dev_err(i2c_dev->dev, "un-recovered arbitration lost\n"); return -EIO; @@ -1314,14 +1412,14 @@ static void tegra_i2c_push_packet_header(struct tegra_i2c_dev *i2c_dev, if (i2c_dev->dma_mode && !i2c_dev->msg_read) *dma_buf++ = packet_header; else - i2c_writel(i2c_dev, packet_header, I2C_TX_FIFO); + i2c_writel(i2c_dev, packet_header, i2c_dev->hw->regs->tx_fifo); packet_header = i2c_dev->msg_len - 1; if (i2c_dev->dma_mode && !i2c_dev->msg_read) *dma_buf++ = packet_header; else - i2c_writel(i2c_dev, packet_header, I2C_TX_FIFO); + i2c_writel(i2c_dev, packet_header, i2c_dev->hw->regs->tx_fifo); packet_header = I2C_HEADER_IE_ENABLE; @@ -1349,7 +1447,7 @@ static void tegra_i2c_push_packet_header(struct tegra_i2c_dev *i2c_dev, if (i2c_dev->dma_mode && !i2c_dev->msg_read) *dma_buf++ = packet_header; else - i2c_writel(i2c_dev, packet_header, I2C_TX_FIFO); + i2c_writel(i2c_dev, packet_header, i2c_dev->hw->regs->tx_fifo); } static int tegra_i2c_error_recover(struct tegra_i2c_dev *i2c_dev, @@ -1470,7 +1568,7 @@ static int tegra_i2c_xfer_msg(struct tegra_i2c_dev *i2c_dev, tegra_i2c_unmask_irq(i2c_dev, int_mask); dev_dbg(i2c_dev->dev, "unmasked IRQ: %02x\n", - i2c_readl(i2c_dev, I2C_INT_MASK)); + i2c_readl(i2c_dev, i2c_dev->hw->regs->int_mask)); if (i2c_dev->dma_mode) { time_left = tegra_i2c_wait_completion(i2c_dev, @@ -1647,6 +1745,7 @@ static const struct tegra_i2c_hw_feature tegra20_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; #if IS_ENABLED(CONFIG_ARCH_TEGRA_2x_SOC) @@ -1679,6 +1778,7 @@ static const struct tegra_i2c_hw_feature tegra20_dvc_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DVC, + .regs = &tegra20_dvc_i2c_regs, }; #endif @@ -1711,6 +1811,7 @@ static const struct tegra_i2c_hw_feature tegra30_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra114_i2c_hw = { @@ -1742,6 +1843,7 @@ static const struct tegra_i2c_hw_feature tegra114_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra124_i2c_hw = { @@ -1773,6 +1875,7 @@ static const struct tegra_i2c_hw_feature tegra124_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra210_i2c_hw = { @@ -1804,6 +1907,7 @@ static const struct tegra_i2c_hw_feature tegra210_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; #if IS_ENABLED(CONFIG_ARCH_TEGRA_210_SOC) @@ -1836,6 +1940,7 @@ static const struct tegra_i2c_hw_feature tegra210_vi_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_VI, + .regs = &tegra210_vi_i2c_regs, }; #endif @@ -1868,6 +1973,7 @@ static const struct tegra_i2c_hw_feature tegra186_i2c_hw = { .enable_hs_mode_support = false, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra194_i2c_hw = { @@ -1901,6 +2007,7 @@ static const struct tegra_i2c_hw_feature tegra194_i2c_hw = { .enable_hs_mode_support = true, .has_mutex = false, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra256_i2c_hw = { @@ -1934,6 +2041,7 @@ static const struct tegra_i2c_hw_feature tegra256_i2c_hw = { .enable_hs_mode_support = true, .has_mutex = true, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra20_i2c_regs, }; static const struct tegra_i2c_hw_feature tegra264_i2c_hw = { @@ -1967,6 +2075,7 @@ static const struct tegra_i2c_hw_feature tegra264_i2c_hw = { .enable_hs_mode_support = true, .has_mutex = true, .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra264_i2c_regs, }; static const struct of_device_id tegra_i2c_of_match[] = { From 93127a0bf8f01299e34938dad1e76492666562b6 Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Wed, 7 Jan 2026 19:56:49 +0530 Subject: [PATCH 103/464] NVIDIA: VR: SAUCE: i2c: tegra: Add support for Tegra410 BugLink: https://bugs.launchpad.net/bugs/2138238 Add support for the Tegra410 SoC, which has 4 I2C controllers. The controllers are feature-equivalent to Tegra264; only the register offsets differ. Signed-off-by: Kartik Rajput (backported from https://lore.kernel.org/all/20260107142649.14917-1-kkartik@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 35f6b577a3cecac4f0d78d18439ff833cae9e65a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/i2c/busses/i2c-tegra.c | 64 ++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c index 07f8fa001fb2b..21cd16db74e85 100644 --- a/drivers/i2c/busses/i2c-tegra.c +++ b/drivers/i2c/busses/i2c-tegra.c @@ -280,6 +280,35 @@ static const struct tegra_i2c_regs tegra264_i2c_regs = { .sw_mutex = 0x0ec, }; +static const struct tegra_i2c_regs tegra410_i2c_regs = { + .cnfg = 0x000, + .status = 0x01c, + .sl_cnfg = 0x020, + .sl_addr1 = 0x02c, + .sl_addr2 = 0x030, + .tlow_sext = 0x034, + .tx_fifo = 0x054, + .rx_fifo = 0x058, + .packet_transfer_status = 0x05c, + .fifo_control = 0x060, + .fifo_status = 0x064, + .int_mask = 0x068, + .int_status = 0x06c, + .clk_divisor = 0x070, + .bus_clear_cnfg = 0x088, + .bus_clear_status = 0x08c, + .config_load = 0x090, + .clken_override = 0x094, + .interface_timing_0 = 0x098, + .interface_timing_1 = 0x09c, + .hs_interface_timing_0 = 0x0a0, + .hs_interface_timing_1 = 0x0a4, + .master_reset_cntrl = 0x0ac, + .mst_fifo_control = 0x0b8, + .mst_fifo_status = 0x0bc, + .sw_mutex = 0x0f0, +}; + /* * msg_end_type: The bus control which needs to be sent at end of transfer. * @MSG_END_STOP: Send stop pulse. @@ -2078,6 +2107,40 @@ static const struct tegra_i2c_hw_feature tegra264_i2c_hw = { .regs = &tegra264_i2c_regs, }; +static const struct tegra_i2c_hw_feature tegra410_i2c_hw = { + .has_continue_xfer_support = true, + .has_per_pkt_xfer_complete_irq = true, + .clk_divisor_hs_mode = 1, + .clk_divisor_std_mode = 0x3f, + .clk_divisor_fast_mode = 0x2c, + .clk_divisor_fast_plus_mode = 0x11, + .has_config_load_reg = true, + .has_multi_master_mode = true, + .has_slcg_override_reg = true, + .has_mst_fifo = true, + .has_mst_reset = true, + .quirks = &tegra194_i2c_quirks, + .supports_bus_clear = true, + .has_apb_dma = false, + .tlow_std_mode = 0x8, + .thigh_std_mode = 0x7, + .tlow_fast_mode = 0x2, + .thigh_fast_mode = 0x2, + .tlow_fastplus_mode = 0x2, + .thigh_fastplus_mode = 0x2, + .tlow_hs_mode = 0x8, + .thigh_hs_mode = 0x6, + .setup_hold_time_std_mode = 0x08080808, + .setup_hold_time_fast_mode = 0x02020202, + .setup_hold_time_fastplus_mode = 0x02020202, + .setup_hold_time_hs_mode = 0x0b0b0b, + .has_interface_timing_reg = true, + .enable_hs_mode_support = true, + .has_mutex = true, + .variant = TEGRA_I2C_VARIANT_DEFAULT, + .regs = &tegra410_i2c_regs, +}; + static const struct of_device_id tegra_i2c_of_match[] = { { .compatible = "nvidia,tegra264-i2c", .data = &tegra264_i2c_hw, }, { .compatible = "nvidia,tegra256-i2c", .data = &tegra256_i2c_hw, }, @@ -2391,6 +2454,7 @@ static const struct acpi_device_id tegra_i2c_acpi_match[] = { {.id = "NVDA0101", .driver_data = (kernel_ulong_t)&tegra210_i2c_hw}, {.id = "NVDA0201", .driver_data = (kernel_ulong_t)&tegra186_i2c_hw}, {.id = "NVDA0301", .driver_data = (kernel_ulong_t)&tegra194_i2c_hw}, + {.id = "NVDA2017", .driver_data = (kernel_ulong_t)&tegra410_i2c_hw}, { } }; MODULE_DEVICE_TABLE(acpi, tegra_i2c_acpi_match); From 2321671f9fa5d20ebd817fc33afad13b34e5dd62 Mon Sep 17 00:00:00 2001 From: Muteeb Akram Date: Tue, 6 Jan 2026 01:58:48 +0000 Subject: [PATCH 104/464] NVIDIA: SAUCE: r8127: print GPL_CLAIM with KERN_INFO BugLink: https://bugs.launchpad.net/bugs/2137588 Add KERN_INFO log level for GPL_CLAIM to downgrade warning messages Signed-off-by: ChunHao Lin Signed-off-by: Revanth Kumar Uppala Signed-off-by: Muteeb Akram Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 3c12da38a1f4ad9723a55cc8aeb0f800ad6e995e noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/r8127_n.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/ethernet/realtek/r8127/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c index 496fec1320d12..9e39016ea2c80 100755 --- a/drivers/net/ethernet/realtek/r8127/r8127_n.c +++ b/drivers/net/ethernet/realtek/r8127/r8127_n.c @@ -14298,7 +14298,7 @@ rtl8127_init_one(struct pci_dev *pdev, rtl8127_sysfs_init(dev); #endif /* ENABLE_R8127_SYSFS */ - printk("%s", GPL_CLAIM); + printk(KERN_INFO "%s", GPL_CLAIM); out: return rc; From 83af2f504fc02100c8a4841f9c2a7cc9dec42004 Mon Sep 17 00:00:00 2001 From: Lucas De Marchi Date: Fri, 9 Jan 2026 11:49:19 -0600 Subject: [PATCH 105/464] NVIDIA: SAUCE: vfio: Fix missing prototype warning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2138132 Fix this warning about missing prototype: ../drivers/vfio/vfio_main.c:1369:21: warning: no previous prototype for ‘vfio_device_from_file’ [-Wmissing-prototypes] 1369 | struct vfio_device *vfio_device_from_file(struct file *file) | ^~~~~~~~~~~~~~~~~~~~~ Add the declaration in the header since it's no different than e.g. vfio_file_is_valid() that is there and remove the extern in the .c. Signed-off-by: Lucas De Marchi Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (backported from commit 006a8656e31361ea8425d655ceef919fd7814d31 noble:linux-nvidia-6.17) [jacobmartin: upstream did not have the extern in nvgrace-gpu/main.c] Signed-off-by: Jacob Martin --- include/linux/vfio.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index ef02a4996d451..313fca4b5636b 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -335,6 +335,7 @@ static inline bool vfio_file_has_dev(struct file *file, struct vfio_device *devi return false; } #endif +struct vfio_device *vfio_device_from_file(struct file *file); bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); From 2c56a83eaabaca4b7550566f463d2a0049cc2c6f Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Sat, 10 Jan 2026 14:22:06 -0600 Subject: [PATCH 106/464] NVIDIA: VR: SAUCE: arm64: Add workaround to convert MT_NORMAL_NC to Device-nGnRE BugLink: https://bugs.launchpad.net/bugs/2138952 Add CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE configuration option that enables conversion of MT_NORMAL_NC (Normal Non-Cacheable) memory attribute to Device-nGnRE memory type in MAIR_EL1 for hardware that requires stricter memory ordering or has issues with Non-Cacheable memory mappings. Key changes: 1. New memory type MT_NORMAL_NC_DMA (Attr5): - Introduced specifically for DMA coherent memory mappings - Configured with the same Normal Non-Cacheable attribute (0x44) as MT_NORMAL_NC (Attr2) by default - pgprot_dmacoherent uses MT_NORMAL_NC_DMA when workaround is enabled, MT_NORMAL_NC otherwise 2. MAIR_EL1 conversion via alternatives framework: - arch/arm64/mm/proc.S uses ARM64 alternatives to patch MAIR_EL1 during early boot - Converts MT_NORMAL_NC (Attr2) from 0x44 to 0x04 (Device-nGnRE) using efficient bfi instruction - MT_NORMAL_NC_DMA (Attr5) keeps the same attribute value as MT_NORMAL_NC originally had - Zero performance overhead when workaround is disabled 3. Boot-time configuration: - Enabled via kernel command line: mair_el1_nc_to_ngnre=1 - Boot CPU fixup in enable_nc_to_ngnre() applies conversion before alternatives are patched - Secondary CPUs automatically use patched alternatives in __cpu_setup - Runtime changes not supported as alternatives cannot be re-patched after boot 4. Errata framework integration: - Registered in arm64_errata[] array as ARM64_WORKAROUND_NC_TO_NGNRE - Capability type: ARM64_CPUCAP_BOOT_CPU_FEATURE - Uses cpucap_is_possible() for build-time capability checking The workaround preserves pgprot_dmacoherent behavior while allowing MT_NORMAL_NC to be converted to Device memory type for other mappings that may be affected by hardware issues. Ensure NC memory attribute assignment is prevented for passthrough device MMIO regions. Signed-off-by: Shanker Donthineni Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Nirmoy Das Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit b32a3cbb3fd22c37336f97f17707a8b54f988cb0 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- arch/arm64/Kconfig | 30 +++++++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/pgtable.h | 6 ++ arch/arm64/kernel/cpu_errata.c | 105 +++++++++++++++++++++++++++++++ arch/arm64/kvm/mmu.c | 3 +- arch/arm64/mm/proc.S | 11 +++- arch/arm64/tools/cpucaps | 1 + 8 files changed, 157 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index d4b97eebf9965..7d181d4d6aebe 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1717,6 +1717,36 @@ config ARM64_TAGGED_ADDR_ABI to system calls as pointer arguments. For details, see Documentation/arch/arm64/tagged-address-abi.rst. +config ARM64_WORKAROUND_NC_TO_NGNRE + bool "Workaround: Convert MT_NORMAL_NC to Device-nGnRE" + default y + help + This option enables a workaround that converts the MT_NORMAL_NC + (Non-Cacheable) memory attribute to Device-nGnRE memory type in + MAIR_EL1 (Memory Attribute Indirection Register). + + This workaround is useful for hardware that requires stricter + memory ordering or has issues with Non-Cacheable memory mappings. + + A new memory type index MT_NORMAL_NC_DMA (Attr5) has been introduced + specifically for DMA coherent memory mappings (pgprot_dmacoherent), + configured with the same Normal Non-Cacheable attribute (0x44) as + MT_NORMAL_NC (Attr2). When this workaround is enabled, it converts + the NC attribute to Device-nGnRE (0x04), and pgprot_dmacoherent + behavior remains the same as before. + + The workaround uses the ARM64 alternatives framework for efficient + runtime patching with no performance overhead when disabled. + + This workaround can only be enabled at boot time via kernel command + line parameter. Runtime changes are not supported because CPU + alternatives cannot be re-patched after boot. + + Boot-time activation (kernel command line): + mair_el1_nc_to_ngnre=1 + + If unsure, say Y. + menuconfig COMPAT bool "Kernel support for 32-bit EL0" depends on ARM64_4K_PAGES || EXPERT diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 177c691914f87..6a252fefec696 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -71,6 +71,8 @@ cpucap_is_possible(const unsigned int cap) return true; case ARM64_HAS_PMUV3: return IS_ENABLED(CONFIG_HW_PERF_EVENTS); + case ARM64_WORKAROUND_NC_TO_NGNRE: + return IS_ENABLED(CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE); } return true; diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index a2b7a33966ff1..b67ab60a407b7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -173,6 +173,7 @@ #define MT_NORMAL_NC 2 #define MT_DEVICE_nGnRnE 3 #define MT_DEVICE_nGnRE 4 +#define MT_NORMAL_NC_DMA 5 /* * Memory types for Stage-2 translation when HCR_EL2.FWB=0. See R_HMNDG, diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index dd062179b9b66..320ed19f4291c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -771,9 +771,15 @@ static inline void __set_puds(struct mm_struct *mm, unsigned long addr, * requires strict alignment and can also force write responses to come from the * endpoint. */ +#ifdef CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE +#define pgprot_dmacoherent(prot) \ + __pgprot_modify(prot, PTE_ATTRINDX_MASK, \ + PTE_ATTRINDX(MT_NORMAL_NC_DMA) | PTE_PXN | PTE_UXN) +#else #define pgprot_dmacoherent(prot) \ __pgprot_modify(prot, PTE_ATTRINDX_MASK, \ PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN) +#endif #define __HAVE_PHYS_MEM_ACCESS_PROT struct file; diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 5c0ab6bfd44a6..059f5e322eb81 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -8,11 +8,16 @@ #include #include #include +#include #include #include #include #include #include +#include +#include +#include +#include static u64 target_impl_cpu_num; static struct target_impl_cpu *target_impl_cpus; @@ -590,6 +595,97 @@ static const struct midr_range erratum_ac04_cpu_23_list[] = { }; #endif +#ifdef CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE +/* + * MAIR_EL1 MT_NORMAL_NC to Device-nGnRE Conversion + * + * Boot-time workaround that converts MT_NORMAL_NC attribute to Device-nGnRE + * (0x04) via kernel parameter for hardware-specific issues. + * + */ + +/* + * Flag indicating if MT_NORMAL_NC to nGnRE conversion is enabled + */ +static int mair_el1_nc_to_ngnre __read_mostly = -1; + +/* + * Parse kernel command line parameter at boot: + * mair_el1_nc_to_ngnre=1 + * Enables MT_NORMAL_NC to Device-nGnRE conversion + */ +static int __init mair_el1_nc_setup(char *str) +{ + bool enable; + int ret; + + ret = kstrtobool(str, &enable); + if (ret) + return ret; + + mair_el1_nc_to_ngnre = enable ? 1 : 0; + + pr_info("MAIR_EL1: MT_NORMAL_NC to Device-nGnRE conversion %s\n", + enable ? "enabled" : "disabled"); + + return 0; +} +early_param("mair_el1_nc_to_ngnre", mair_el1_nc_setup); + +/* Cpufeature capability check for MAIR NC to nGnRE workaround */ +static bool has_nc_ngnre_workaround(const struct arm64_cpu_capabilities *entry, + int scope) +{ + /* Lazy initialization: check only once */ + if (mair_el1_nc_to_ngnre == -1) { + if ((arm_smccc_get_soc_id_version() == 0x036b0410) && + (arm_smccc_get_soc_id_revision() < 5)) { + mair_el1_nc_to_ngnre = 1; + return true; + } + mair_el1_nc_to_ngnre = 0; + } + + return mair_el1_nc_to_ngnre > 0; +} + +/* + * Called by cpufeature framework when CPU comes online + * For boot CPU: alternatives not yet patched, so apply NC to nGnRE here + * For secondary CPUs: alternatives already patched in __cpu_setup + */ +static void enable_nc_to_ngnre(struct arm64_cpu_capabilities const *cap) +{ + u64 attr_mask, current_mair, new_mair; + u8 current_attr; + + if (mair_el1_nc_to_ngnre <= 0) + return; + + current_mair = read_sysreg(mair_el1); + + /* Check if MT_NORMAL_NC is already Device-nGnRE */ + attr_mask = GENMASK_ULL((MT_NORMAL_NC * 8) + 7, MT_NORMAL_NC * 8); + current_attr = FIELD_GET(attr_mask, current_mair); + + /* Already set via alternatives (secondary CPU case) */ + if (current_attr == MAIR_ATTR_DEVICE_nGnRE) + return; + + /* Apply override for boot CPU */ + new_mair = (current_mair & ~attr_mask) | + ((u64)MAIR_ATTR_DEVICE_nGnRE << (MT_NORMAL_NC * 8)); + + write_sysreg(new_mair, mair_el1); + isb(); + local_flush_tlb_all(); + + pr_info("CPU%d: MAIR_EL1 updated 0x%016llx -> 0x%016llx\n", + smp_processor_id(), current_mair, new_mair); +} + +#endif /* CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE */ + const struct arm64_cpu_capabilities arm64_errata[] = { #ifdef CONFIG_ARM64_WORKAROUND_CLEAN_CACHE { @@ -938,6 +1034,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .matches = has_impdef_pmuv3, .cpu_enable = cpu_enable_impdef_pmuv3_traps, }, +#ifdef CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE + { + .desc = "MAIR_EL1 NC to nGnRE", + .capability = ARM64_WORKAROUND_NC_TO_NGNRE, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = has_nc_ngnre_workaround, + .cpu_enable = enable_nc_to_ngnre, + }, +#endif { } }; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 17d64a1e11e5c..a5af59bb30231 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1765,7 +1765,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn = ipa >> PAGE_SHIFT; mte_allowed = kvm_vma_mte_allowed(vma); - vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED; + if (!cpus_have_cap(ARM64_WORKAROUND_NC_TO_NGNRE)) + vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED; vm_flags = vma->vm_flags; diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 22866b49be372..45fe22fbe2fa1 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -76,7 +76,8 @@ MAIR_ATTRIDX(MAIR_ATTR_DEVICE_nGnRE, MT_DEVICE_nGnRE) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ - MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED)) + MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL_TAGGED) | \ + MAIR_ATTRIDX(MAIR_ATTR_NORMAL_NC, MT_NORMAL_NC_DMA)) #ifdef CONFIG_CPU_PM /** @@ -498,6 +499,14 @@ SYM_FUNC_START(__cpu_setup) tcr .req x16 tcr2 .req x15 mov_q mair, MAIR_EL1_SET + +#ifdef CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE +alternative_if ARM64_WORKAROUND_NC_TO_NGNRE + mov x9, #MAIR_ATTR_DEVICE_nGnRE + bfi mair, x9, #(MT_NORMAL_NC * 8), #8 +alternative_else_nop_endif +#endif + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS_MIN) | TCR_CACHE_FLAGS | \ TCR_SHARED | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_EL1_AS | \ TCR_EL1_TBI0 | TCR_EL1_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 7261553b644b2..3b9659d5b082e 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -118,6 +118,7 @@ WORKAROUND_CAVIUM_TX2_219_PRFM WORKAROUND_CAVIUM_TX2_219_TVM WORKAROUND_CLEAN_CACHE WORKAROUND_DEVICE_LOAD_ACQUIRE +WORKAROUND_NC_TO_NGNRE WORKAROUND_NVIDIA_CARMEL_CNP WORKAROUND_PMUV3_IMPDEF_TRAPS WORKAROUND_QCOM_FALKOR_E1003 From b9d02777ab21dc42c32e4f5fd8eff7d93f599ad7 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Tue, 20 Jan 2026 16:57:20 -0800 Subject: [PATCH 107/464] NVIDIA: VR: SAUCE: [Config] Enable arm64 NC-to-NGNRE workaround BugLink: https://bugs.launchpad.net/bugs/2138952 Enable the workaround to convert MT_NORMAL_NC to Device-nGnRE by setting CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE=y. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Nirmoy Das Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 3bf78d7604924cd9aa898be73722b4a77a6a6cae noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index fe8266527dfd2..d4eff215ec49b 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -27,6 +27,9 @@ CONFIG_ARM64_ERRATUM_2224489 note<'Required for Grace enablem CONFIG_ARM64_ERRATUM_2253138 policy<{'arm64': 'y'}> CONFIG_ARM64_ERRATUM_2253138 note<'Required for Grace enablement'> +CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE policy<{'arm64': 'y'}> +CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE note<'Required for Vera enablement'> + CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enablement'> From 87748e3895bc871af67d97865ee5a33e92feba1c Mon Sep 17 00:00:00 2001 From: Kartik Rajput Date: Thu, 22 Jan 2026 16:38:05 +0530 Subject: [PATCH 108/464] NVIDIA: VR: SAUCE: soc/tegra: pmc: Add PMC support for Tegra410 BugLink: https://bugs.launchpad.net/bugs/2139082 Tegra410 uses PMC driver only to retrieve system reset reason using PMC sysfs. Tegra410 uses ACPI to probe PMC, unlike device-tree boot it does not use the early initialisation sequence. Add PMC support for Tegra410. Signed-off-by: Kartik Rajput (backported from https://lore.kernel.org/all/20260122110805.97899-1-kkartik@nvidia.com/) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit ad4700cdcedfd220254e50de17cfc28b03f28000 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/soc/tegra/pmc.c | 128 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c index 6debaabdaa36a..4ee8159f26a31 100644 --- a/drivers/soc/tegra/pmc.c +++ b/drivers/soc/tegra/pmc.c @@ -11,6 +11,7 @@ #define pr_fmt(fmt) "tegra-pmc: " fmt +#include #include #include #include @@ -3098,12 +3099,30 @@ static void tegra_pmc_reset_suspend_mode(void *data) pmc->suspend_mode = TEGRA_SUSPEND_NOT_READY; } +static int tegra_pmc_acpi_probe(struct platform_device *pdev) +{ + pmc->soc = device_get_match_data(&pdev->dev); + pmc->dev = &pdev->dev; + + pmc->base = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(pmc->base)) + return PTR_ERR(pmc->base); + + tegra_pmc_reset_sysfs_init(pmc); + platform_set_drvdata(pdev, pmc); + + return 0; +} + static int tegra_pmc_probe(struct platform_device *pdev) { void __iomem *base; struct resource *res; int err; + if (is_acpi_node(dev_fwnode(&pdev->dev))) + return tegra_pmc_acpi_probe(pdev); + /* * Early initialisation should have configured an initial * register mapping and setup the soc data pointer. If these @@ -4618,6 +4637,108 @@ static const struct tegra_pmc_soc tegra264_pmc_soc = { .max_wake_vectors = 4, }; +static const char * const tegra410_reset_sources[] = { + "SYS_RESET_N", /* 0x0 */ + "CSDC_RTC_XTAL", + "VREFRO_POWER_BAD", + "FMON_32K", + "FMON_OSC", + "POD_RTC", + "POD_IO", + "POD_PLUS_IO_SPLL", + "POD_PLUS_IO_VMON", /* 0x8 */ + "POD_PLUS_SOC", + "VMON_PLUS_UV", + "VMON_PLUS_OV", + "FUSECRC_FAULT", + "OSC_FAULT", + "BPMP_BOOT_FAULT", + "SCPM_BPMP_CORE_CLK", + "SCPM_PSC_SE_CLK", /* 0x10 */ + "VMON_SOC_MIN", + "VMON_SOC_MAX", + "NVJTAG_SEL_MONITOR", + "L0_RST_REQ_N", + "NV_THERM_FAULT", + "PSC_SW", + "POD_C2C_LPI_0", + "POD_C2C_LPI_1", /* 0x18 */ + "BPMP_FMON", + "FMON_SPLL_OUT", + "L1_RST_REQ_N", + "OCP_RECOVERY", + "AO_WDT_POR", + "BPMP_WDT_POR", + "RAS_WDT_POR", + "TOP_0_WDT_POR", /* 0x20 */ + "TOP_1_WDT_POR", + "TOP_2_WDT_POR", + "PSC_WDT_POR", + "OOBHUB_WDT_POR", + "MSS_SEQ_WDT_POR", + "SW_MAIN", + "L0L1_RST_OUT_N", + "HSM", /* 0x28 */ + "CSITE_SW", + "AO_WDT_DBG", + "BPMP_WDT_DBG", + "RAS_WDT_DBG", + "TOP_0_WDT_DBG", + "TOP_1_WDT_DBG", + "TOP_2_WDT_DBG", + "PSC_WDT_DBG", /* 0x30 */ + "TSC_0_WDT_DBG", + "TSC_1_WDT_DBG", + "OOBHUB_WDT_DBG", + "MSS_SEQ_WDT_DBG", + "L2_RST_REQ_N", + "L2_RST_OUT_N", + "SC7" +}; + +static const struct tegra_pmc_regs tegra410_pmc_regs = { + .rst_status = 0x8, + .rst_source_shift = 0x2, + .rst_source_mask = 0xfc, + .rst_level_shift = 0x0, + .rst_level_mask = 0x3, +}; + +static const struct tegra_pmc_soc tegra410_pmc_soc = { + .supports_core_domain = false, + .num_powergates = 0, + .powergates = NULL, + .num_cpu_powergates = 0, + .cpu_powergates = NULL, + .has_tsense_reset = false, + .has_gpu_clamps = false, + .needs_mbist_war = false, + .has_impl_33v_pwr = false, + .maybe_tz_only = false, + .num_io_pads = 0, + .io_pads = NULL, + .num_pin_descs = 0, + .pin_descs = NULL, + .regs = &tegra410_pmc_regs, + .init = NULL, + .setup_irq_polarity = NULL, + .set_wake_filters = NULL, + .irq_set_wake = NULL, + .irq_set_type = NULL, + .reset_sources = tegra410_reset_sources, + .num_reset_sources = ARRAY_SIZE(tegra410_reset_sources), + .reset_levels = tegra186_reset_levels, + .num_reset_levels = ARRAY_SIZE(tegra186_reset_levels), + .num_wake_events = 0, + .wake_events = NULL, + .max_wake_events = 0, + .max_wake_vectors = 0, + .pmc_clks_data = NULL, + .num_pmc_clks = 0, + .has_blink_output = false, + .has_single_mmio_aperture = false, +}; + static const struct of_device_id tegra_pmc_match[] = { { .compatible = "nvidia,tegra264-pmc", .data = &tegra264_pmc_soc }, { .compatible = "nvidia,tegra234-pmc", .data = &tegra234_pmc_soc }, @@ -4632,6 +4753,12 @@ static const struct of_device_id tegra_pmc_match[] = { { } }; +static const struct acpi_device_id tegra_pmc_acpi_match[] = { + { .id = "NVDA2016", .driver_data = (kernel_ulong_t)&tegra410_pmc_soc }, + { } +}; +MODULE_DEVICE_TABLE(acpi, tegra_pmc_acpi_match); + static void tegra_pmc_sync_state(struct device *dev) { struct device_node *np, *child; @@ -4682,6 +4809,7 @@ static struct platform_driver tegra_pmc_driver = { .name = "tegra-pmc", .suppress_bind_attrs = true, .of_match_table = tegra_pmc_match, + .acpi_match_table = tegra_pmc_acpi_match, #if defined(CONFIG_PM_SLEEP) && defined(CONFIG_ARM) .pm = &tegra_pmc_pm_ops, #endif From 0a2389a406f3154a1066958aa57b6fdb2abe8f1a Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Fri, 16 Jan 2026 20:56:40 -0800 Subject: [PATCH 109/464] NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable devices BugLink: https://bugs.launchpad.net/bugs/2139088 Controlled by the IOMMU driver, ATS is usually enabled "on demand", when a device requests a translation service from its associated IOMMU HW running on the channel of a given PASID. This is working even when a device has no translation on its RID, i.e. RID is IOMMU bypassed. On the other hand, certain PCIe device requires non-PASID ATS, when its RID stream is IOMMU bypassed. Call this "always on". For instance, the CXL spec notes in "3.2.5.13 Memory Type on CXL.cache": "To source requests on CXL.cache, devices need to get the Host Physical Address (HPA) from the Host by means of an ATS request on CXL.io." In other word, the CXL.cache capability relies on ATS. Otherwise, it won't have access to the host physical memory. Introduce a new pci_ats_always_on() for IOMMU driver to scan a PCI device, to shift ATS policies between "on demand" and "always on". Add the support for CXL.cache devices first. Non-CXL devices will be added in quirks.c file. Suggested-by: Vikram Sethi Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1768624180.git.nicolinc@nvidia.com) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 72bd823fb4f1640564558f8b316cdb9ec24ebd8f noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/pci/ats.c | 44 +++++++++++++++++++++++++++++++++++ include/linux/pci-ats.h | 3 +++ include/uapi/linux/pci_regs.h | 5 ++++ 3 files changed, 52 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index ec6c8dbdc5e9c..1795131f0697f 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -205,6 +205,50 @@ int pci_ats_page_aligned(struct pci_dev *pdev) return 0; } +/* + * CXL r4.0, sec 3.2.5.13 Memory Type on CXL.cache notes: to source requests on + * CXL.cache, devices need to get the Host Physical Address (HPA) from the Host + * by means of an ATS request on CXL.io. + * + * In other world, CXL.cache devices cannot access physical memory without ATS. + */ +static bool pci_cxl_ats_always_on(struct pci_dev *pdev) +{ + int offset; + u16 cap; + + offset = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + CXL_DVSEC_PCIE_DEVICE); + if (!offset) + return false; + + pci_read_config_word(pdev, offset + CXL_DVSEC_CAP_OFFSET, &cap); + if (cap & CXL_DVSEC_CACHE_CAPABLE) + return true; + + return false; +} + +/** + * pci_ats_always_on - Whether the PCI device requires ATS to be always enabled + * @pdev: the PCI device + * + * Returns true, if the PCI device requires non-PASID ATS function on an IOMMU + * bypassed configuration. + */ +bool pci_ats_always_on(struct pci_dev *pdev) +{ + if (pci_ats_disabled() || !pci_ats_supported(pdev)) + return false; + + /* A VF inherits its PF's requirement for ATS function */ + if (pdev->is_virtfn) + pdev = pci_physfn(pdev); + + return pci_cxl_ats_always_on(pdev); +} +EXPORT_SYMBOL_GPL(pci_ats_always_on); + #ifdef CONFIG_PCI_PRI void pci_pri_init(struct pci_dev *pdev) { diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index 75c6c86cf09dc..d14ba727d38b3 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -12,6 +12,7 @@ int pci_prepare_ats(struct pci_dev *dev, int ps); void pci_disable_ats(struct pci_dev *dev); int pci_ats_queue_depth(struct pci_dev *dev); int pci_ats_page_aligned(struct pci_dev *dev); +bool pci_ats_always_on(struct pci_dev *dev); #else /* CONFIG_PCI_ATS */ static inline bool pci_ats_supported(struct pci_dev *d) { return false; } @@ -24,6 +25,8 @@ static inline int pci_ats_queue_depth(struct pci_dev *d) { return -ENODEV; } static inline int pci_ats_page_aligned(struct pci_dev *dev) { return 0; } +static inline bool pci_ats_always_on(struct pci_dev *dev) +{ return false; } #endif /* CONFIG_PCI_ATS */ #ifdef CONFIG_PCI_PRI diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index e830dea141267..c0371efe7f376 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1411,4 +1411,9 @@ #define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_ID __GENMASK(15, 8) #define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW __GENMASK(31, 16) +/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ +#define CXL_DVSEC_PCIE_DEVICE 0 +#define CXL_DVSEC_CAP_OFFSET 0xA +#define CXL_DVSEC_CACHE_CAPABLE BIT(0) + #endif /* LINUX_PCI_REGS_H */ From dbb46413e062e3caafee41e14cdad284aae08556 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Fri, 16 Jan 2026 20:56:41 -0800 Subject: [PATCH 110/464] NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for non-CXL NVIDIA GPUs BugLink: https://bugs.launchpad.net/bugs/2139088 Some non-CXL NVIDIA GPU devices support non-PASID ATS function when their RIDs are IOMMU bypassed. This is slightly different than the default ATS policy which would only enable ATS on demand: when a non-zero PASID line is enabled in SVA use cases. Introduce a pci_dev_specific_ats_always_on() quirk function to support a list of IDs for these device. Then, include it pci_ats_always_on(). Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1768624180.git.nicolinc@nvidia.com) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit c44923d41bbfbcfada8d4feed1061d5ccd91ce49 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/pci/ats.c | 3 ++- drivers/pci/pci.h | 9 +++++++++ drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 1795131f0697f..6db45ae2cc8e3 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -245,7 +245,8 @@ bool pci_ats_always_on(struct pci_dev *pdev) if (pdev->is_virtfn) pdev = pci_physfn(pdev); - return pci_cxl_ats_always_on(pdev); + return pci_cxl_ats_always_on(pdev) || + pci_dev_specific_ats_always_on(pdev); } EXPORT_SYMBOL_GPL(pci_ats_always_on); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 13d998fbacce6..13fa71f965900 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -1150,6 +1150,15 @@ static inline int pci_dev_specific_reset(struct pci_dev *dev, bool probe) } #endif +#if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_PCI_ATS) +bool pci_dev_specific_ats_always_on(struct pci_dev *dev); +#else +static inline bool pci_dev_specific_ats_always_on(struct pci_dev *dev) +{ + return false; +} +#endif + #if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_ARM64) int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment, struct resource *res); diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index cdb6271397f6a..6009eb1122c9b 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -5729,6 +5729,29 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1457, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1459, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145a, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145c, quirk_intel_e2000_no_ats); + +static const struct pci_dev_ats_always_on { + u16 vendor; + u16 device; +} pci_dev_ats_always_on[] = { + { PCI_VENDOR_ID_NVIDIA, 0x2e12, }, + { PCI_VENDOR_ID_NVIDIA, 0x2e2a, }, + { PCI_VENDOR_ID_NVIDIA, 0x2e2b, }, + { 0 } +}; + +/* Some non-CXL devices support ATS on RID when it is IOMMU-bypassed */ +bool pci_dev_specific_ats_always_on(struct pci_dev *pdev) +{ + const struct pci_dev_ats_always_on *i; + + for (i = pci_dev_ats_always_on; i->vendor; i++) { + if (i->vendor == pdev->vendor && i->device == pdev->device) + return true; + } + + return false; +} #endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ From 3a2c1d57503bac25f8040d89a1cf4c67dca8e101 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Fri, 16 Jan 2026 20:56:42 -0800 Subject: [PATCH 111/464] NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on BugLink: https://bugs.launchpad.net/bugs/2139088 When a device's default substream attaches to an identity domain, the SMMU driver currently sets the device's STE between two modes: Mode 1: Cfg=Translate, S1DSS=Bypass, EATS=1 Mode 2: Cfg=bypass (EATS is ignored by HW) When there is an active PASID (non-default substream), mode 1 is used. And when there is no PASID support or no active PASID, mode 2 is used. The driver will also downgrade an STE from mode 1 to mode 2, when the last active substream becomes inactive. However, there are PCIe devices that demand ATS to be always on. For these devices, their STEs have to use the mode 1 as HW ignores EATS with mode 2. Change the driver accordingly: - always use the mode 1 - never downgrade to mode 2 - allocate and retain a CD table (see note below) Note that these devices might not support PASID, i.e. doing non-PASID ATS. In such a case, the ssid_bits is set to 0. However, s1cdmax must be set to a !0 value in order to keep the S1DSS field effective. Thus, when a master requires ats_always_on, set its s1cdmax to minimal 1, meaning the CD table will have a dummy entry (SSID=1) that will be never used. Now, for these device, arm_smmu_cdtab_allocated() will always return true, v.s. false prior to this change. When its default substream is attached to an IDENTITY domain, its first CD is NULL in the table, which is a totally valid case. Thus, drop the WARN_ON(). Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1768624180.git.nicolinc@nvidia.com) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (backported from commit df59703f696af30f801c051b06b7e73fa53bc316 noble:linux-nvidia-6.17) [jacobmartin: adjust calls to arm_smmu_attach_dev_ste() to include old_domain, as upstream commit fd714986e4e4 ("iommu: Pass in old domain to attach_dev callback functions") requires it.] Signed-off-by: Jacob Martin --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 75 ++++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 65 insertions(+), 11 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3cf4f515753d1..9f3829d5e3028 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1489,7 +1489,7 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) if (!arm_smmu_cdtab_allocated(&master->cd_table)) return; cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (WARN_ON(!cdptr)) + if (!cdptr) return; arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } @@ -1503,6 +1503,22 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; cd_table->s1cdmax = master->ssid_bits; + + /* + * When a device doesn't support PASID (non default SSID), ssid_bits is + * set to 0. This also sets S1CDMAX to 0, which disables the substreams + * and ignores the S1DSS field. + * + * On the other hand, if a device demands ATS to be always on even when + * its default substream is IOMMU bypassed, it has to use EATS that is + * only effective with an STE (CFG=S1translate, S1DSS=Bypass). For such + * use cases, S1CDMAX has to be !0, in order to make use of S1DSS/EATS. + * + * Set S1CDMAX no lower than 1. This would add a dummy substream in the + * CD table but it should never be used by an actual CD. + */ + if (master->ats_always_on) + cd_table->s1cdmax = max_t(u8, cd_table->s1cdmax, 1); max_contexts = 1 << cd_table->s1cdmax; if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_CDTAB) || @@ -3257,7 +3273,8 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain, * When the last user of the CD table goes away downgrade the STE back * to a non-cd_table one, by re-attaching its sid_domain. */ - if (!arm_smmu_ssids_in_use(&master->cd_table)) { + if (!master->ats_always_on && + !arm_smmu_ssids_in_use(&master->cd_table)) { struct iommu_domain *sid_domain = iommu_driver_get_domain_for_dev(master->dev); @@ -3273,7 +3290,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, struct iommu_domain *old_domain, struct device *dev, struct arm_smmu_ste *ste, - unsigned int s1dss) + unsigned int s1dss, bool ats_always_on) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_attach_state state = { @@ -3292,7 +3309,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, * If the CD table is not in use we can use the provided STE, otherwise * we use a cdtable STE with the provided S1DSS. */ - if (arm_smmu_ssids_in_use(&master->cd_table)) { + if (ats_always_on || arm_smmu_ssids_in_use(&master->cd_table)) { /* * If a CD table has to be present then we need to run with ATS * on because we have to assume a PASID is using ATS. For @@ -3328,7 +3345,8 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, arm_smmu_master_clear_vmaster(master); arm_smmu_make_bypass_ste(master->smmu, &ste); arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste, - STRTAB_STE_1_S1DSS_BYPASS); + STRTAB_STE_1_S1DSS_BYPASS, + master->ats_always_on); return 0; } @@ -3351,7 +3369,8 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, arm_smmu_master_clear_vmaster(master); arm_smmu_make_abort_ste(&ste); arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste, - STRTAB_STE_1_S1DSS_TERMINATE); + STRTAB_STE_1_S1DSS_TERMINATE, + false); return 0; } @@ -3588,6 +3607,40 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master) kfree(master->streams); } +static int arm_smmu_master_prepare_ats(struct arm_smmu_master *master) +{ + bool s1p = master->smmu->features & ARM_SMMU_FEAT_TRANS_S1; + unsigned int stu = __ffs(master->smmu->pgsize_bitmap); + struct pci_dev *pdev = to_pci_dev(master->dev); + int ret; + + if (!arm_smmu_ats_supported(master)) + return 0; + + if (!pci_ats_always_on(pdev)) + goto out_prepare; + + /* + * S1DSS is required for ATS to be always on for identity domain cases. + * However, the S1DSS field is ignored if !IDR0_S1P or !IDR1_SSIDSIZE. + */ + if (!s1p || !master->smmu->ssid_bits) { + dev_info_once(master->dev, + "SMMU doesn't support ATS to be always on\n"); + goto out_prepare; + } + + master->ats_always_on = true; + + ret = arm_smmu_alloc_cd_tables(master); + if (ret) + return ret; + +out_prepare: + pci_prepare_ats(pdev, stu); + return 0; +} + static struct iommu_device *arm_smmu_probe_device(struct device *dev) { int ret; @@ -3636,14 +3689,14 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; - if (dev_is_pci(dev)) { - unsigned int stu = __ffs(smmu->pgsize_bitmap); - - pci_prepare_ats(to_pci_dev(dev), stu); - } + ret = arm_smmu_master_prepare_ats(master); + if (ret) + goto err_disable_pasid; return &smmu->iommu; +err_disable_pasid: + arm_smmu_disable_pasid(master); err_free_master: kfree(master); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 3c6d65d36164f..f966d474b61fd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -848,6 +848,7 @@ struct arm_smmu_master { bool ats_enabled : 1; bool ste_ats_enabled : 1; bool stall_enabled; + bool ats_always_on; unsigned int ssid_bits; unsigned int iopf_refcount; }; From 1b41f8dc6dace29856f436f46f78e22d0565c742 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:48 +0000 Subject: [PATCH 112/464] NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Rename doc to Tegra241 BugLink: https://bugs.launchpad.net/bugs/2139315 The documentation in nvidia-pmu.rst contains PMUs specific to NVIDIA Tegra241 SoC. Rename the file for this specific SoC to have better distinction with other NVIDIA SoC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit e9fedcf8f86bca14b82772337cbf0400ee199070 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- Documentation/admin-guide/perf/index.rst | 2 +- .../perf/{nvidia-pmu.rst => nvidia-tegra241-pmu.rst} | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) rename Documentation/admin-guide/perf/{nvidia-pmu.rst => nvidia-tegra241-pmu.rst} (98%) diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index 47d9a3df6329b..c407bb44b08e3 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -24,7 +24,7 @@ Performance monitor support thunderx2-pmu alibaba_pmu dwc_pcie_pmu - nvidia-pmu + nvidia-tegra241-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst similarity index 98% rename from Documentation/admin-guide/perf/nvidia-pmu.rst rename to Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst index f538ef67e0e8f..fad5bc4cee6c0 100644 --- a/Documentation/admin-guide/perf/nvidia-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst @@ -1,8 +1,8 @@ -========================================================= -NVIDIA Tegra SoC Uncore Performance Monitoring Unit (PMU) -========================================================= +============================================================ +NVIDIA Tegra241 SoC Uncore Performance Monitoring Unit (PMU) +============================================================ -The NVIDIA Tegra SoC includes various system PMUs to measure key performance +The NVIDIA Tegra241 SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Scalable Coherency Fabric (SCF) From da6cb548aa27a428129d5988d460f0382320a25f Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:49 +0000 Subject: [PATCH 113/464] NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Adds Unified Coherent Fabric PMU support in Tegra410 SOC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit f717536a488c5fc9a62f7e12ba7af0c41aabad3a noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- Documentation/admin-guide/perf/index.rst | 1 + .../admin-guide/perf/nvidia-tegra410-pmu.rst | 106 ++++++++++++++++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 90 ++++++++++++++- 3 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index c407bb44b08e3..aa12708ddb965 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -25,6 +25,7 @@ Performance monitor support alibaba_pmu dwc_pcie_pmu nvidia-tegra241-pmu + nvidia-tegra410-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst new file mode 100644 index 0000000000000..7b7ba5700ca19 --- /dev/null +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -0,0 +1,106 @@ +===================================================================== +NVIDIA Tegra410 SoC Uncore Performance Monitoring Unit (PMU) +===================================================================== + +The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance +metrics like memory bandwidth, latency, and utilization: + +* Unified Coherence Fabric (UCF) + +PMU Driver +---------- + +The PMU driver describes the available events and configuration of each PMU in +sysfs. Please see the sections below to get the sysfs path of each PMU. Like +other uncore PMU drivers, the driver provides "cpumask" sysfs attribute to show +the CPU id used to handle the PMU event. There is also "associated_cpus" +sysfs attribute, which contains a list of CPUs associated with the PMU instance. + +UCF PMU +------- + +The Unified Coherence Fabric (UCF) in the NVIDIA Tegra410 SoC serves as a +distributed cache, last level for CPU Memory and CXL Memory, and cache coherent +interconnect that supports hardware coherence across multiple coherently caching +agents, including: + + * CPU clusters + * GPU + * PCIe Ordering Controller Unit (OCU) + * Other IO-coherent requesters + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_ucf_pmu_. + +Some of the events available in this PMU can be used to measure bandwidth and +utilization: + + * slc_access_rd: count the number of read requests to SLC. + * slc_access_wr: count the number of write requests to SLC. + * slc_bytes_rd: count the number of bytes transferred by slc_access_rd. + * slc_bytes_wr: count the number of bytes transferred by slc_access_wr. + * mem_access_rd: count the number of read requests to local or remote memory. + * mem_access_wr: count the number of write requests to local or remote memory. + * mem_bytes_rd: count the number of bytes transferred by mem_access_rd. + * mem_bytes_wr: count the number of bytes transferred by mem_access_wr. + * cycles: counts the UCF cycles. + +The average bandwidth is calculated as:: + + AVG_SLC_READ_BANDWIDTH_IN_GBPS = SLC_BYTES_RD / ELAPSED_TIME_IN_NS + AVG_SLC_WRITE_BANDWIDTH_IN_GBPS = SLC_BYTES_WR / ELAPSED_TIME_IN_NS + AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS + AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_SLC_READ_REQUEST_RATE = SLC_ACCESS_RD / CYCLES + AVG_SLC_WRITE_REQUEST_RATE = SLC_ACCESS_WR / CYCLES + AVG_MEM_READ_REQUEST_RATE = MEM_ACCESS_RD / CYCLES + AVG_MEM_WRITE_REQUEST_RATE = MEM_ACCESS_WR / CYCLES + +More details about what other events are available can be found in Tegra410 SoC +technical reference manual. + +The events can be filtered based on source or destination. The source filter +indicates the traffic initiator to the SLC, e.g local CPU, non-CPU device, or +remote socket. The destination filter specifies the destination memory type, +e.g. local system memory (CMEM), local GPU memory (GMEM), or remote memory. The +local/remote classification of the destination filter is based on the home +socket of the address, not where the data actually resides. The available +filters are described in +/sys/bus/event_source/devices/nvidia_ucf_pmu_/format/. + +The list of UCF PMU event filters: + +* Source filter: + + * src_loc_cpu: if set, count events from local CPU + * src_loc_noncpu: if set, count events from local non-CPU device + * src_rem: if set, count events from CPU, GPU, PCIE devices of remote socket + +* Destination filter: + + * dst_loc_cmem: if set, count events to local system memory (CMEM) address + * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address + * dst_loc_other: if set, count events to local CXL memory address + * dst_rem: if set, count events to CPU, GPU, and CXL memory address of remote socket + +If the source is not specified, the PMU will count events from all sources. If +the destination is not specified, the PMU will count events to all destinations. + +Example usage: + +* Count event id 0x0 in socket 0 from all sources and to all destinations:: + + perf stat -a -e nvidia_ucf_pmu_0/event=0x0/ + +* Count event id 0x0 in socket 0 with source filter = local CPU and destination + filter = local system memory (CMEM):: + + perf stat -a -e nvidia_ucf_pmu_0/event=0x0,src_loc_cpu=0x1,dst_loc_cmem=0x1/ + +* Count event id 0x0 in socket 1 with source filter = local non-CPU device and + destination filter = remote memory:: + + perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index e06a06d3407b1..c67667097a3cd 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -21,6 +21,13 @@ #define NV_CNVL_PORT_COUNT 4ULL #define NV_CNVL_FILTER_ID_MASK GENMASK_ULL(NV_CNVL_PORT_COUNT - 1, 0) +#define NV_UCF_SRC_COUNT 3ULL +#define NV_UCF_DST_COUNT 4ULL +#define NV_UCF_FILTER_ID_MASK GENMASK_ULL(11, 0) +#define NV_UCF_FILTER_SRC GENMASK_ULL(2, 0) +#define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) +#define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -124,6 +131,37 @@ static struct attribute *mcf_pmu_event_attrs[] = { NULL, }; +static struct attribute *ucf_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(bus_cycles, 0x1D), + + ARM_CSPMU_EVENT_ATTR(slc_allocate, 0xF0), + ARM_CSPMU_EVENT_ATTR(slc_wb, 0xF3), + ARM_CSPMU_EVENT_ATTR(slc_refill_rd, 0x109), + ARM_CSPMU_EVENT_ATTR(slc_refill_wr, 0x10A), + ARM_CSPMU_EVENT_ATTR(slc_hit_rd, 0x119), + + ARM_CSPMU_EVENT_ATTR(slc_access_dataless, 0x183), + ARM_CSPMU_EVENT_ATTR(slc_access_atomic, 0x184), + + ARM_CSPMU_EVENT_ATTR(slc_access, 0xF2), + ARM_CSPMU_EVENT_ATTR(slc_access_rd, 0x111), + ARM_CSPMU_EVENT_ATTR(slc_access_wr, 0x112), + ARM_CSPMU_EVENT_ATTR(slc_bytes_rd, 0x113), + ARM_CSPMU_EVENT_ATTR(slc_bytes_wr, 0x114), + + ARM_CSPMU_EVENT_ATTR(mem_access_rd, 0x121), + ARM_CSPMU_EVENT_ATTR(mem_access_wr, 0x122), + ARM_CSPMU_EVENT_ATTR(mem_bytes_rd, 0x123), + ARM_CSPMU_EVENT_ATTR(mem_bytes_wr, 0x124), + + ARM_CSPMU_EVENT_ATTR(local_snoop, 0x180), + ARM_CSPMU_EVENT_ATTR(ext_snp_access, 0x181), + ARM_CSPMU_EVENT_ATTR(ext_snp_evict, 0x182), + + ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), + NULL, +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -152,6 +190,18 @@ static struct attribute *cnvlink_pmu_format_attrs[] = { NULL, }; +static struct attribute *ucf_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_EVENT_ATTR, + ARM_CSPMU_FORMAT_ATTR(src_loc_noncpu, "config1:0"), + ARM_CSPMU_FORMAT_ATTR(src_loc_cpu, "config1:1"), + ARM_CSPMU_FORMAT_ATTR(src_rem, "config1:2"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config1:8"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config1:9"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_other, "config1:10"), + ARM_CSPMU_FORMAT_ATTR(dst_rem, "config1:11"), + NULL, +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -236,6 +286,27 @@ static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, writel(filter, cspmu->base0 + PMCCFILTR); } +static u32 ucf_pmu_event_filter(const struct perf_event *event) +{ + u32 ret, filter, src, dst; + + filter = nv_cspmu_event_filter(event); + + /* Monitor all sources if none is selected. */ + src = FIELD_GET(NV_UCF_FILTER_SRC, filter); + if (src == 0) + src = GENMASK_ULL(NV_UCF_SRC_COUNT - 1, 0); + + /* Monitor all destinations if none is selected. */ + dst = FIELD_GET(NV_UCF_FILTER_DST, filter); + if (dst == 0) + dst = GENMASK_ULL(NV_UCF_DST_COUNT - 1, 0); + + ret = FIELD_PREP(NV_UCF_FILTER_SRC, src); + ret |= FIELD_PREP(NV_UCF_FILTER_DST, dst); + + return ret; +} enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, @@ -342,6 +413,23 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .init_data = NULL }, }, + { + .prodid = 0x2CF20000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_ucf_pmu_%u", + .name_fmt = NAME_FMT_SOCKET, + .template_ctx = { + .event_attr = ucf_pmu_event_attrs, + .format_attr = ucf_pmu_format_attrs, + .filter_mask = NV_UCF_FILTER_ID_MASK, + .filter_default_val = NV_UCF_FILTER_DEFAULT, + .filter2_mask = 0x0, + .filter2_default_val = 0x0, + .get_filter = ucf_pmu_event_filter, + .get_filter2 = NULL, + .init_data = NULL + }, + }, { .prodid = 0, .prodid_mask = 0, From 37123e0e097e15dce2baa406f6bf663dc3012a18 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:50 +0000 Subject: [PATCH 114/464] NVIDIA: VR: SAUCE: perf/arm_cspmu: Add arm_cspmu_acpi_dev_get BugLink: https://bugs.launchpad.net/bugs/2139315 Add interface to get ACPI device associated with the PMU. This ACPI device may contain additional properties not covered by the standard properties. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit 11ded2412355efd7e6f196d0c16e3d5706c2de09 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/perf/arm_cspmu/arm_cspmu.c | 24 +++++++++++++++++++++++- drivers/perf/arm_cspmu/arm_cspmu.h | 17 ++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/drivers/perf/arm_cspmu/arm_cspmu.c b/drivers/perf/arm_cspmu/arm_cspmu.c index 34430b68f6025..dadc9b765d801 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.c +++ b/drivers/perf/arm_cspmu/arm_cspmu.c @@ -16,7 +16,7 @@ * The user should refer to the vendor technical documentation to get details * about the supported events. * - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -1132,6 +1132,28 @@ static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) return 0; } + +struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) +{ + char hid[16]; + char uid[16]; + struct acpi_device *adev; + const struct acpi_apmt_node *apmt_node; + + apmt_node = arm_cspmu_apmt_node(cspmu->dev); + if (!apmt_node || apmt_node->type != ACPI_APMT_NODE_TYPE_ACPI) + return NULL; + + memset(hid, 0, sizeof(hid)); + memset(uid, 0, sizeof(uid)); + + memcpy(hid, &apmt_node->inst_primary, sizeof(apmt_node->inst_primary)); + snprintf(uid, sizeof(uid), "%u", apmt_node->inst_secondary); + + adev = acpi_dev_get_first_match_dev(hid, uid, -1); + return adev; +} +EXPORT_SYMBOL_GPL(arm_cspmu_acpi_dev_get); #else static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) { diff --git a/drivers/perf/arm_cspmu/arm_cspmu.h b/drivers/perf/arm_cspmu/arm_cspmu.h index cd65a58dbd884..3200966732003 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.h +++ b/drivers/perf/arm_cspmu/arm_cspmu.h @@ -1,13 +1,14 @@ /* SPDX-License-Identifier: GPL-2.0 * * ARM CoreSight Architecture PMU driver. - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ #ifndef __ARM_CSPMU_H__ #define __ARM_CSPMU_H__ +#include #include #include #include @@ -255,4 +256,18 @@ int arm_cspmu_impl_register(const struct arm_cspmu_impl_match *impl_match); /* Unregister vendor backend. */ void arm_cspmu_impl_unregister(const struct arm_cspmu_impl_match *impl_match); +#if defined(CONFIG_ACPI) +/** + * Get ACPI device associated with the PMU. + * The caller is responsible for calling acpi_dev_put() on the returned device. + */ +struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu); +#else +static inline struct acpi_device * +arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) +{ + return NULL; +} +#endif + #endif /* __ARM_CSPMU_H__ */ From 3d47d0e99a5c4849e1936c2d2003dfdfadc8c5c3 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:51 +0000 Subject: [PATCH 115/464] NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Adds PCIE PMU support in Tegra410 SOC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit 35f3a96b09d8c44def60cf6c5f0ecb0a3843890c noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 162 ++++++++++++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 208 +++++++++++++++++- 2 files changed, 368 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 7b7ba5700ca19..8528685ddb61e 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -6,6 +6,7 @@ The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) +* PCIE PMU Driver ---------- @@ -104,3 +105,164 @@ Example usage: destination filter = remote memory:: perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ + +PCIE PMU +-------- + +This PMU monitors all read/write traffic from the root port(s) or a particular +BDF in a PCIE root complex (RC) to local or remote memory. There is one PMU per +PCIE RC in the SoC. Each RC can have up to 16 lanes that can be bifurcated into +up to 8 root ports. The traffic from each root port can be filtered using RP or +BDF filter. For example, specifying "src_rp_mask=0xFF" means the PMU counter will +capture traffic from all RPs. Please see below for more details. + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_pcie_pmu__rc_. + +The events in this PMU can be used to measure bandwidth, utilization, and +latency: + + * rd_req: count the number of read requests by PCIE device. + * wr_req: count the number of write requests by PCIE device. + * rd_bytes: count the number of bytes transferred by rd_req. + * wr_bytes: count the number of bytes transferred by wr_req. + * rd_cum_outs: count outstanding rd_req each cycle. + * cycles: counts the PCIE cycles. + +The average bandwidth is calculated as:: + + AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS + AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_RD_REQUEST_RATE = RD_REQ / CYCLES + AVG_WR_REQUEST_RATE = WR_REQ / CYCLES + + +The average latency is calculated as:: + + FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ + AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ + +The PMU events can be filtered based on the traffic source and destination. +The source filter indicates the PCIE devices that will be monitored. The +destination filter specifies the destination memory type, e.g. local system +memory (CMEM), local GPU memory (GMEM), or remote memory. The local/remote +classification of the destination filter is based on the home socket of the +address, not where the data actually resides. These filters can be found in +/sys/bus/event_source/devices/nvidia_pcie_pmu__rc_/format/. + +The list of event filters: + +* Source filter: + + * src_rp_mask: bitmask of root ports that will be monitored. Each bit in this + bitmask represents the RP index in the RC. If the bit is set, all devices under + the associated RP will be monitored. E.g "src_rp_mask=0xF" will monitor + devices in root port 0 to 3. + * src_bdf: the BDF that will be monitored. This is a 16-bit value that + follows formula: (bus << 8) + (device << 3) + (function). For example, the + value of BDF 27:01.1 is 0x2781. + * src_bdf_en: enable the BDF filter. If this is set, the BDF filter value in + "src_bdf" is used to filter the traffic. + + Note that Root-Port and BDF filters are mutually exclusive and the PMU in + each RC can only have one BDF filter for the whole counters. If BDF filter + is enabled, the BDF filter value will be applied to all events. + +* Destination filter: + + * dst_loc_cmem: if set, count events to local system memory (CMEM) address + * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address + * dst_loc_pcie_p2p: if set, count events to local PCIE peer address + * dst_loc_pcie_cxl: if set, count events to local CXL memory address + * dst_rem: if set, count events to remote memory address + +If the source filter is not specified, the PMU will count events from all root +ports. If the destination filter is not specified, the PMU will count events +to all destinations. + +Example usage: + +* Count event id 0x0 from root port 0 of PCIE RC-0 on socket 0 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_0/event=0x0,src_rp_mask=0x1/ + +* Count event id 0x1 from root port 0 and 1 of PCIE RC-1 on socket 0 and + targeting just local CMEM of socket 0:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_1/event=0x1,src_rp_mask=0x3,dst_loc_cmem=0x1/ + +* Count event id 0x2 from root port 0 of PCIE RC-2 on socket 1 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_1_rc_2/event=0x2,src_rp_mask=0x1/ + +* Count event id 0x3 from root port 0 and 1 of PCIE RC-3 on socket 1 and + targeting just local CMEM of socket 1:: + + perf stat -a -e nvidia_pcie_pmu_1_rc_3/event=0x3,src_rp_mask=0x3,dst_loc_cmem=0x1/ + +* Count event id 0x4 from BDF 01:01.0 of PCIE RC-4 on socket 0 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ + +Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA +Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space +for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register +contains the following information to map PCIE devices under the RP back to its RC# : + + - Bus# (byte 0xc) : bus number as reported by the lspci output + - Segment# (byte 0xd) : segment number as reported by the lspci output + - RP# (byte 0xe) : port number as reported by LnkCap attribute from lspci for a device with Root Port capability + - RC# (byte 0xf): root complex number associated with the RP + - Socket# (byte 0x10): socket number associated with the RP + +Example script for mapping lspci BDF to RC# and socket#:: + + #!/bin/bash + while read bdf rest; do + dvsec4_reg=$(lspci -vv -s $bdf | awk ' + /Designated Vendor-Specific: Vendor=10de ID=0004/ { + match($0, /\[([0-9a-fA-F]+)/, arr); + print "0x" arr[1]; + exit + } + ') + if [ -n "$dvsec4_reg" ]; then + bus=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xc))).b) + segment=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xd))).b) + rp=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xe))).b) + rc=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xf))).b) + socket=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0x10))).b) + echo "$bdf: Bus=$bus, Segment=$segment, RP=$rp, RC=$rc, Socket=$socket" + fi + done < <(lspci -d 10de:) + +Example output:: + + 0001:00:00.0: Bus=00, Segment=01, RP=00, RC=00, Socket=00 + 0002:80:00.0: Bus=80, Segment=02, RP=01, RC=01, Socket=00 + 0002:a0:00.0: Bus=a0, Segment=02, RP=02, RC=01, Socket=00 + 0002:c0:00.0: Bus=c0, Segment=02, RP=03, RC=01, Socket=00 + 0002:e0:00.0: Bus=e0, Segment=02, RP=04, RC=01, Socket=00 + 0003:00:00.0: Bus=00, Segment=03, RP=00, RC=02, Socket=00 + 0004:00:00.0: Bus=00, Segment=04, RP=00, RC=03, Socket=00 + 0005:00:00.0: Bus=00, Segment=05, RP=00, RC=04, Socket=00 + 0005:40:00.0: Bus=40, Segment=05, RP=01, RC=04, Socket=00 + 0005:c0:00.0: Bus=c0, Segment=05, RP=02, RC=04, Socket=00 + 0006:00:00.0: Bus=00, Segment=06, RP=00, RC=05, Socket=00 + 0009:00:00.0: Bus=00, Segment=09, RP=00, RC=00, Socket=01 + 000a:80:00.0: Bus=80, Segment=0a, RP=01, RC=01, Socket=01 + 000a:a0:00.0: Bus=a0, Segment=0a, RP=02, RC=01, Socket=01 + 000a:e0:00.0: Bus=e0, Segment=0a, RP=03, RC=01, Socket=01 + 000b:00:00.0: Bus=00, Segment=0b, RP=00, RC=02, Socket=01 + 000c:00:00.0: Bus=00, Segment=0c, RP=00, RC=03, Socket=01 + 000d:00:00.0: Bus=00, Segment=0d, RP=00, RC=04, Socket=01 + 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 + 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 + 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index c67667097a3cd..3a5531d1f94c7 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -8,6 +8,7 @@ #include #include +#include #include #include "arm_cspmu.h" @@ -28,6 +29,19 @@ #define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) #define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) +#define NV_PCIE_V2_PORT_COUNT 8ULL +#define NV_PCIE_V2_FILTER_ID_MASK GENMASK_ULL(24, 0) +#define NV_PCIE_V2_FILTER_PORT GENMASK_ULL(NV_PCIE_V2_PORT_COUNT - 1, 0) +#define NV_PCIE_V2_FILTER_BDF_VAL GENMASK_ULL(23, NV_PCIE_V2_PORT_COUNT) +#define NV_PCIE_V2_FILTER_BDF_EN BIT(24) +#define NV_PCIE_V2_FILTER_BDF_VAL_EN GENMASK_ULL(24, NV_PCIE_V2_PORT_COUNT) +#define NV_PCIE_V2_FILTER_DEFAULT NV_PCIE_V2_FILTER_PORT + +#define NV_PCIE_V2_DST_COUNT 5ULL +#define NV_PCIE_V2_FILTER2_ID_MASK GENMASK_ULL(4, 0) +#define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) +#define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -162,6 +176,16 @@ static struct attribute *ucf_pmu_event_attrs[] = { NULL, }; +static struct attribute *pcie_v2_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), + ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), + ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), + ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), + ARM_CSPMU_EVENT_ATTR(rd_cum_outs, 0x4), + ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), + NULL, +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -202,6 +226,19 @@ static struct attribute *ucf_pmu_format_attrs[] = { NULL, }; +static struct attribute *pcie_v2_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_EVENT_ATTR, + ARM_CSPMU_FORMAT_ATTR(src_rp_mask, "config1:0-7"), + ARM_CSPMU_FORMAT_ATTR(src_bdf, "config1:8-23"), + ARM_CSPMU_FORMAT_ATTR(src_bdf_en, "config1:24"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config2:0"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config2:1"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_p2p, "config2:2"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_cxl, "config2:3"), + ARM_CSPMU_FORMAT_ATTR(dst_rem, "config2:4"), + NULL, +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -233,6 +270,32 @@ nv_cspmu_get_name(const struct arm_cspmu *cspmu) return ctx->name; } +#if defined(CONFIG_ACPI) +static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) +{ + struct fwnode_handle *fwnode; + struct acpi_device *adev; + int ret; + + adev = arm_cspmu_acpi_dev_get(cspmu); + if (!adev) + return -ENODEV; + + fwnode = acpi_fwnode_handle(adev); + ret = fwnode_property_read_u32(fwnode, "instance_id", id); + if (ret) + dev_err(cspmu->dev, "Failed to get instance ID\n"); + + acpi_dev_put(adev); + return ret; +} +#else +static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) +{ + return -EINVAL; +} +#endif + static u32 nv_cspmu_event_filter(const struct perf_event *event) { const struct nv_cspmu_ctx *ctx = @@ -278,6 +341,20 @@ static void nv_cspmu_set_ev_filter(struct arm_cspmu *cspmu, } } +static void nv_cspmu_reset_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + const struct nv_cspmu_ctx *ctx = + to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); + const u32 offset = 4 * event->hw.idx; + + if (ctx->get_filter) + writel(0, cspmu->base0 + PMEVFILTR + offset); + + if (ctx->get_filter2) + writel(0, cspmu->base0 + PMEVFILT2R + offset); +} + static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, const struct perf_event *event) { @@ -308,9 +385,103 @@ static u32 ucf_pmu_event_filter(const struct perf_event *event) return ret; } +static u32 pcie_v2_pmu_bdf_val_en(u32 filter) +{ + const u32 bdf_en = FIELD_GET(NV_PCIE_V2_FILTER_BDF_EN, filter); + + /* Returns both BDF value and enable bit if BDF filtering is enabled. */ + if (bdf_en) + return FIELD_GET(NV_PCIE_V2_FILTER_BDF_VAL_EN, filter); + + /* Ignore the BDF value if BDF filter is not enabled. */ + return 0; +} + +static u32 pcie_v2_pmu_event_filter(const struct perf_event *event) +{ + u32 filter, lead_filter, lead_bdf; + struct perf_event *leader; + const struct nv_cspmu_ctx *ctx = + to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); + + filter = event->attr.config1 & ctx->filter_mask; + if (filter != 0) + return filter; + + leader = event->group_leader; + + /* Use leader's filter value if its BDF filtering is enabled. */ + if (event != leader) { + lead_filter = pcie_v2_pmu_event_filter(leader); + lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); + if (lead_bdf != 0) + return lead_filter; + } + + /* Otherwise, return default filter value. */ + return ctx->filter_default_val; +} + +static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, + struct perf_event *new_ev) +{ + /* + * Make sure the events are using same BDF filter since the PCIE-SRC PMU + * only supports one common BDF filter setting for all of the counters. + */ + + int idx; + u32 new_filter, new_rp, new_bdf, new_lead_filter, new_lead_bdf; + struct perf_event *leader, *new_leader; + + if (cspmu->impl.ops.is_cycle_counter_event(new_ev)) + return 0; + + new_leader = new_ev->group_leader; + + new_filter = pcie_v2_pmu_event_filter(new_ev); + new_lead_filter = pcie_v2_pmu_event_filter(new_leader); + + new_bdf = pcie_v2_pmu_bdf_val_en(new_filter); + new_lead_bdf = pcie_v2_pmu_bdf_val_en(new_lead_filter); + + new_rp = FIELD_GET(NV_PCIE_V2_FILTER_PORT, new_filter); + + if (new_rp != 0 && new_bdf != 0) { + dev_err(cspmu->dev, + "RP and BDF filtering are mutually exclusive\n"); + return -EINVAL; + } + + if (new_bdf != new_lead_bdf) { + dev_err(cspmu->dev, + "sibling and leader BDF value should be equal\n"); + return -EINVAL; + } + + /* Compare BDF filter on existing events. */ + idx = find_first_bit(cspmu->hw_events.used_ctrs, + cspmu->cycle_counter_logical_idx); + + if (idx != cspmu->cycle_counter_logical_idx) { + leader = cspmu->hw_events.events[idx]->group_leader; + + const u32 lead_filter = pcie_v2_pmu_event_filter(leader); + const u32 lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); + + if (new_lead_bdf != lead_bdf) { + dev_err(cspmu->dev, "only one BDF value is supported\n"); + return -EINVAL; + } + } + + return 0; +} + enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, - NAME_FMT_SOCKET + NAME_FMT_SOCKET, + NAME_FMT_SOCKET_INST }; struct nv_cspmu_match { @@ -430,6 +601,27 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .init_data = NULL }, }, + { + .prodid = 0x10301000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_pcie_pmu_%u_rc_%u", + .name_fmt = NAME_FMT_SOCKET_INST, + .template_ctx = { + .event_attr = pcie_v2_pmu_event_attrs, + .format_attr = pcie_v2_pmu_format_attrs, + .filter_mask = NV_PCIE_V2_FILTER_ID_MASK, + .filter_default_val = NV_PCIE_V2_FILTER_DEFAULT, + .filter2_mask = NV_PCIE_V2_FILTER2_ID_MASK, + .filter2_default_val = NV_PCIE_V2_FILTER2_DEFAULT, + .get_filter = pcie_v2_pmu_event_filter, + .get_filter2 = nv_cspmu_event_filter2, + .init_data = NULL + }, + .ops = { + .validate_event = pcie_v2_pmu_validate_event, + .reset_ev_filter = nv_cspmu_reset_ev_filter, + } + }, { .prodid = 0, .prodid_mask = 0, @@ -453,7 +645,7 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, const struct nv_cspmu_match *match) { - char *name; + char *name = NULL; struct device *dev = cspmu->dev; static atomic_t pmu_generic_idx = {0}; @@ -467,6 +659,16 @@ static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, socket); break; } + case NAME_FMT_SOCKET_INST: { + const int cpu = cpumask_first(&cspmu->associated_cpus); + const int socket = cpu_to_node(cpu); + u32 inst_id; + + if (!nv_cspmu_get_inst_id(cspmu, &inst_id)) + name = devm_kasprintf(dev, GFP_KERNEL, + match->name_pattern, socket, inst_id); + break; + } case NAME_FMT_GENERIC: name = devm_kasprintf(dev, GFP_KERNEL, match->name_pattern, atomic_fetch_inc(&pmu_generic_idx)); @@ -514,8 +716,10 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) cspmu->impl.ctx = ctx; /* NVIDIA specific callbacks. */ + SET_OP(validate_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); + SET_OP(reset_ev_filter, impl_ops, match, NULL); SET_OP(get_event_attrs, impl_ops, match, nv_cspmu_get_event_attrs); SET_OP(get_format_attrs, impl_ops, match, nv_cspmu_get_format_attrs); SET_OP(get_name, impl_ops, match, nv_cspmu_get_name); From 7d06d911bbc02aadcdbcbfca45708b310d27a49b Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:52 +0000 Subject: [PATCH 116/464] NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Adds PCIE-TGT PMU support in Tegra410 SOC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit 6183f41d33fcc389f11db1f930593bd94be786e8 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 76 ++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 324 ++++++++++++++++++ 2 files changed, 400 insertions(+) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 8528685ddb61e..07dc447eead7c 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -7,6 +7,7 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE +* PCIE-TGT PMU Driver ---------- @@ -211,6 +212,11 @@ Example usage: perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ +.. _NVIDIA_T410_PCIE_PMU_RC_Mapping_Section: + +Mapping the RC# to lspci segment number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register @@ -266,3 +272,73 @@ Example output:: 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 + +PCIE-TGT PMU +------------ + +The PCIE-TGT PMU monitors traffic targeting PCIE BAR and CXL HDM ranges. +There is one PCIE-TGT PMU per PCIE root complex (RC) in the SoC. Each RC in +Tegra410 SoC can have up to 16 lanes that can be bifurcated into up to 8 root +ports (RP). The PMU provides RP filter to count PCIE BAR traffic to each RP and +address filter to count access to PCIE BAR or CXL HDM ranges. The details +of the filters are described in the following sections. + +Mapping the RC# to lspci segment number is similar to the PCIE PMU. +Please see :ref:`NVIDIA_T410_PCIE_PMU_RC_Mapping_Section` for more info. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_. + +The events in this PMU can be used to measure bandwidth and utilization: + + * rd_req: count the number of read requests to PCIE. + * wr_req: count the number of write requests to PCIE. + * rd_bytes: count the number of bytes transferred by rd_req. + * wr_bytes: count the number of bytes transferred by wr_req. + * cycles: counts the PCIE cycles. + +The average bandwidth is calculated as:: + + AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS + AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_RD_REQUEST_RATE = RD_REQ / CYCLES + AVG_WR_REQUEST_RATE = WR_REQ / CYCLES + +The PMU events can be filtered based on the destination root port or target +address range. Filtering based on RP is only available for PCIE BAR traffic. +Address filter works for both PCIE BAR and CXL HDM ranges. These filters can be +found in sysfs, see +/sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_/format/. + +Destination filter settings: + +* dst_rp_mask: bitmask to select the root port(s) to monitor. E.g. "dst_rp_mask=0xFF" + corresponds to all root ports (from 0 to 7) in the PCIE RC. Note that this filter is + only available for PCIE BAR traffic. +* dst_addr_base: BAR or CXL HDM filter base address. +* dst_addr_mask: BAR or CXL HDM filter address mask. +* dst_addr_en: enable BAR or CXL HDM address range filter. If this is set, the + address range specified by "dst_addr_base" and "dst_addr_mask" will be used to filter + the PCIE BAR and CXL HDM traffic address. The PMU uses the following comparison + to determine if the traffic destination address falls within the filter range:: + + (txn's addr & dst_addr_mask) == (dst_addr_base & dst_addr_mask) + + If the comparison succeeds, then the event will be counted. + +If the destination filter is not specified, the RP filter will be configured by default +to count PCIE BAR traffic to all root ports. + +Example usage: + +* Count event id 0x0 to root port 0 and 1 of PCIE RC-0 on socket 0:: + + perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_0/event=0x0,dst_rp_mask=0x3/ + +* Count event id 0x1 for accesses to PCIE BAR or CXL HDM address range + 0x10000 to 0x100FF on socket 0's PCIE RC-1:: + + perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index 3a5531d1f94c7..095d2f322c6f9 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -42,6 +42,24 @@ #define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) #define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST +#define NV_PCIE_TGT_PORT_COUNT 8ULL +#define NV_PCIE_TGT_EV_TYPE_CC 0x4 +#define NV_PCIE_TGT_EV_TYPE_COUNT 3ULL +#define NV_PCIE_TGT_EV_TYPE_MASK GENMASK_ULL(NV_PCIE_TGT_EV_TYPE_COUNT - 1, 0) +#define NV_PCIE_TGT_FILTER2_MASK GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT, 0) +#define NV_PCIE_TGT_FILTER2_PORT GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT - 1, 0) +#define NV_PCIE_TGT_FILTER2_ADDR_EN BIT(NV_PCIE_TGT_PORT_COUNT) +#define NV_PCIE_TGT_FILTER2_ADDR GENMASK_ULL(15, NV_PCIE_TGT_PORT_COUNT) +#define NV_PCIE_TGT_FILTER2_DEFAULT NV_PCIE_TGT_FILTER2_PORT + +#define NV_PCIE_TGT_ADDR_COUNT 8ULL +#define NV_PCIE_TGT_ADDR_STRIDE 20 +#define NV_PCIE_TGT_ADDR_CTRL 0xD38 +#define NV_PCIE_TGT_ADDR_BASE_LO 0xD3C +#define NV_PCIE_TGT_ADDR_BASE_HI 0xD40 +#define NV_PCIE_TGT_ADDR_MASK_LO 0xD44 +#define NV_PCIE_TGT_ADDR_MASK_HI 0xD48 + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -186,6 +204,15 @@ static struct attribute *pcie_v2_pmu_event_attrs[] = { NULL, }; +static struct attribute *pcie_tgt_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), + ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), + ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), + ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), + ARM_CSPMU_EVENT_ATTR(cycles, NV_PCIE_TGT_EV_TYPE_CC), + NULL, +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -239,6 +266,15 @@ static struct attribute *pcie_v2_pmu_format_attrs[] = { NULL, }; +static struct attribute *pcie_tgt_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_ATTR(event, "config:0-2"), + ARM_CSPMU_FORMAT_ATTR(dst_rp_mask, "config:3-10"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_en, "config:11"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_base, "config1:0-63"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_mask, "config2:0-63"), + NULL, +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -478,6 +514,268 @@ static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, return 0; } +struct pcie_tgt_addr_filter { + u32 refcount; + u64 base; + u64 mask; +}; + +struct pcie_tgt_data { + struct pcie_tgt_addr_filter addr_filter[NV_PCIE_TGT_ADDR_COUNT]; + void __iomem *addr_filter_reg; +}; + +#if defined(CONFIG_ACPI) +static int pcie_tgt_init_data(struct arm_cspmu *cspmu) +{ + int ret; + struct acpi_device *adev; + struct pcie_tgt_data *data; + struct list_head resource_list; + struct resource_entry *rentry; + struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); + struct device *dev = cspmu->dev; + + data = devm_kzalloc(dev, sizeof(struct pcie_tgt_data), GFP_KERNEL); + if (!data) + return -ENOMEM; + + adev = arm_cspmu_acpi_dev_get(cspmu); + if (!adev) { + dev_err(dev, "failed to get associated PCIE-TGT device\n"); + return -ENODEV; + } + + INIT_LIST_HEAD(&resource_list); + ret = acpi_dev_get_memory_resources(adev, &resource_list); + if (ret < 0) { + dev_err(dev, "failed to get PCIE-TGT device memory resources\n"); + acpi_dev_put(adev); + return ret; + } + + rentry = list_first_entry_or_null( + &resource_list, struct resource_entry, node); + if (rentry) { + data->addr_filter_reg = devm_ioremap_resource(dev, rentry->res); + ret = 0; + } + + if (IS_ERR(data->addr_filter_reg)) { + dev_err(dev, "failed to get address filter resource\n"); + ret = PTR_ERR(data->addr_filter_reg); + } + + acpi_dev_free_resource_list(&resource_list); + acpi_dev_put(adev); + + ctx->data = data; + + return ret; +} +#else +static int pcie_tgt_init_data(struct arm_cspmu *cspmu) +{ + return -ENODEV; +} +#endif + +static struct pcie_tgt_data *pcie_tgt_get_data(struct arm_cspmu *cspmu) +{ + struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); + + return ctx->data; +} + +/* Find the first available address filter slot. */ +static int pcie_tgt_find_addr_idx(struct arm_cspmu *cspmu, u64 base, u64 mask, + bool is_reset) +{ + int i; + struct pcie_tgt_data *data = pcie_tgt_get_data(cspmu); + + for (i = 0; i < NV_PCIE_TGT_ADDR_COUNT; i++) { + if (!is_reset && data->addr_filter[i].refcount == 0) + return i; + + if (data->addr_filter[i].base == base && + data->addr_filter[i].mask == mask) + return i; + } + + return -ENODEV; +} + +static u32 pcie_tgt_pmu_event_filter(const struct perf_event *event) +{ + u32 filter; + + filter = (event->attr.config >> NV_PCIE_TGT_EV_TYPE_COUNT) & + NV_PCIE_TGT_FILTER2_MASK; + + return filter; +} + +static bool pcie_tgt_pmu_addr_en(const struct perf_event *event) +{ + u32 filter = pcie_tgt_pmu_event_filter(event); + + return FIELD_GET(NV_PCIE_TGT_FILTER2_ADDR_EN, filter) != 0; +} + +static u32 pcie_tgt_pmu_port_filter(const struct perf_event *event) +{ + u32 filter = pcie_tgt_pmu_event_filter(event); + + return FIELD_GET(NV_PCIE_TGT_FILTER2_PORT, filter); +} + +static u64 pcie_tgt_pmu_dst_addr_base(const struct perf_event *event) +{ + return event->attr.config1; +} + +static u64 pcie_tgt_pmu_dst_addr_mask(const struct perf_event *event) +{ + return event->attr.config2; +} + +static int pcie_tgt_pmu_validate_event(struct arm_cspmu *cspmu, + struct perf_event *new_ev) +{ + u64 base, mask; + int idx; + + if (!pcie_tgt_pmu_addr_en(new_ev)) + return 0; + + /* Make sure there is a slot available for the address filter. */ + base = pcie_tgt_pmu_dst_addr_base(new_ev); + mask = pcie_tgt_pmu_dst_addr_mask(new_ev); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); + if (idx < 0) + return -EINVAL; + + return 0; +} + +static void pcie_tgt_pmu_config_addr_filter(struct arm_cspmu *cspmu, + bool en, u64 base, u64 mask, int idx) +{ + struct pcie_tgt_data *data; + struct pcie_tgt_addr_filter *filter; + void __iomem *filter_reg; + + data = pcie_tgt_get_data(cspmu); + filter = &data->addr_filter[idx]; + filter_reg = data->addr_filter_reg + (idx * NV_PCIE_TGT_ADDR_STRIDE); + + if (en) { + filter->refcount++; + if (filter->refcount == 1) { + filter->base = base; + filter->mask = mask; + + writel(lower_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); + writel(upper_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); + writel(lower_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); + writel(upper_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); + writel(1, filter_reg + NV_PCIE_TGT_ADDR_CTRL); + } + } else { + filter->refcount--; + if (filter->refcount == 0) { + writel(0, filter_reg + NV_PCIE_TGT_ADDR_CTRL); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); + + filter->base = 0; + filter->mask = 0; + } + } +} + +static void pcie_tgt_pmu_set_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + bool addr_filter_en; + int idx; + u32 filter2_val, filter2_offset, port_filter; + u64 base, mask; + + filter2_val = 0; + filter2_offset = PMEVFILT2R + (4 * event->hw.idx); + + addr_filter_en = pcie_tgt_pmu_addr_en(event); + if (addr_filter_en) { + base = pcie_tgt_pmu_dst_addr_base(event); + mask = pcie_tgt_pmu_dst_addr_mask(event); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); + + if (idx < 0) { + dev_err(cspmu->dev, + "Unable to find a slot for address filtering\n"); + writel(0, cspmu->base0 + filter2_offset); + return; + } + + /* Configure address range filter registers.*/ + pcie_tgt_pmu_config_addr_filter(cspmu, true, base, mask, idx); + + /* Config the counter to use the selected address filter slot. */ + filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_ADDR, 1U << idx); + } + + port_filter = pcie_tgt_pmu_port_filter(event); + + /* Monitor all ports if no filter is selected. */ + if (!addr_filter_en && port_filter == 0) + port_filter = NV_PCIE_TGT_FILTER2_PORT; + + filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_PORT, port_filter); + + writel(filter2_val, cspmu->base0 + filter2_offset); +} + +static void pcie_tgt_pmu_reset_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + bool addr_filter_en; + u64 base, mask; + int idx; + + addr_filter_en = pcie_tgt_pmu_addr_en(event); + if (!addr_filter_en) + return; + + base = pcie_tgt_pmu_dst_addr_base(event); + mask = pcie_tgt_pmu_dst_addr_mask(event); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, true); + + if (idx < 0) { + dev_err(cspmu->dev, + "Unable to find the address filter slot to reset\n"); + return; + } + + pcie_tgt_pmu_config_addr_filter( + cspmu, false, base, mask, idx); +} + +static u32 pcie_tgt_pmu_event_type(const struct perf_event *event) +{ + return event->attr.config & NV_PCIE_TGT_EV_TYPE_MASK; +} + +static bool pcie_tgt_pmu_is_cycle_counter_event(const struct perf_event *event) +{ + u32 event_type = pcie_tgt_pmu_event_type(event); + + return event_type == NV_PCIE_TGT_EV_TYPE_CC; +} + enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, NAME_FMT_SOCKET, @@ -622,6 +920,30 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .reset_ev_filter = nv_cspmu_reset_ev_filter, } }, + { + .prodid = 0x10700000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_pcie_tgt_pmu_%u_rc_%u", + .name_fmt = NAME_FMT_SOCKET_INST, + .template_ctx = { + .event_attr = pcie_tgt_pmu_event_attrs, + .format_attr = pcie_tgt_pmu_format_attrs, + .filter_mask = 0x0, + .filter_default_val = 0x0, + .filter2_mask = NV_PCIE_TGT_FILTER2_MASK, + .filter2_default_val = NV_PCIE_TGT_FILTER2_DEFAULT, + .get_filter = NULL, + .get_filter2 = NULL, + .init_data = pcie_tgt_init_data + }, + .ops = { + .is_cycle_counter_event = pcie_tgt_pmu_is_cycle_counter_event, + .event_type = pcie_tgt_pmu_event_type, + .validate_event = pcie_tgt_pmu_validate_event, + .set_ev_filter = pcie_tgt_pmu_set_ev_filter, + .reset_ev_filter = pcie_tgt_pmu_reset_ev_filter, + } + }, { .prodid = 0, .prodid_mask = 0, @@ -717,6 +1039,8 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) /* NVIDIA specific callbacks. */ SET_OP(validate_event, impl_ops, match, NULL); + SET_OP(event_type, impl_ops, match, NULL); + SET_OP(is_cycle_counter_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); SET_OP(reset_ev_filter, impl_ops, match, NULL); From 48c219284801c7f2b02d365b4db071039e0f2f49 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:53 +0000 Subject: [PATCH 117/464] NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 CPU Memory Latency PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Adds CPU Memory (CMEM) Latency PMU support in Tegra410 SOC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit e66c9ad006cb8dc191104858f30c99fcd567826e noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 25 + drivers/perf/Kconfig | 7 + drivers/perf/Makefile | 1 + drivers/perf/nvidia_t410_cmem_latency_pmu.c | 727 ++++++++++++++++++ 4 files changed, 760 insertions(+) create mode 100644 drivers/perf/nvidia_t410_cmem_latency_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 07dc447eead7c..11fc1c88346a0 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -8,6 +8,7 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE * PCIE-TGT +* CPU Memory (CMEM) Latency PMU Driver ---------- @@ -342,3 +343,27 @@ Example usage: 0x10000 to 0x100FF on socket 0's PCIE RC-1:: perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ + +CPU Memory (CMEM) Latency PMU +----------------------------- + +This PMU monitors latency events of memory read requests to local +CPU DRAM: + + * RD_REQ counters: count read requests (32B per request). + * RD_CUM_OUTS counters: accumulated outstanding request counter, which track + how many cycles the read requests are in flight. + * CYCLES counter: counts the number of elapsed cycles. + +The average latency is calculated as:: + + FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ + AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. + +Example usage:: + + perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 638321fc9800c..9fed3c41d5ea0 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -311,4 +311,11 @@ config MARVELL_PEM_PMU Enable support for PCIe Interface performance monitoring on Marvell platform. +config NVIDIA_TEGRA410_CMEM_LATENCY_PMU + tristate "NVIDIA Tegra410 CPU Memory Latency PMU" + depends on ARM64 + help + Enable perf support for CPU memory latency counters monitoring on + NVIDIA Tegra410 SoC. + endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index ea52711a87e32..4aa6aad393c2d 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -35,3 +35,4 @@ obj-$(CONFIG_DWC_PCIE_PMU) += dwc_pcie_pmu.o obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o +obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o diff --git a/drivers/perf/nvidia_t410_cmem_latency_pmu.c b/drivers/perf/nvidia_t410_cmem_latency_pmu.c new file mode 100644 index 0000000000000..9b466581c8fcc --- /dev/null +++ b/drivers/perf/nvidia_t410_cmem_latency_pmu.c @@ -0,0 +1,727 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * NVIDIA Tegra410 CPU Memory (CMEM) Latency PMU driver. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NUM_INSTANCES 14 +#define BCAST(pmu) pmu->base[NUM_INSTANCES] + +/* Register offsets. */ +#define CG_CTRL 0x800 +#define CTRL 0x808 +#define STATUS 0x810 +#define CYCLE_CNTR 0x818 +#define MC0_REQ_CNTR 0x820 +#define MC0_AOR_CNTR 0x830 +#define MC1_REQ_CNTR 0x838 +#define MC1_AOR_CNTR 0x848 +#define MC2_REQ_CNTR 0x850 +#define MC2_AOR_CNTR 0x860 + +/* CTRL values. */ +#define CTRL_DISABLE 0x0ULL +#define CTRL_ENABLE 0x1ULL +#define CTRL_CLR 0x2ULL + +/* CG_CTRL values. */ +#define CG_CTRL_DISABLE 0x0ULL +#define CG_CTRL_ENABLE 0x1ULL + +/* STATUS register field. */ +#define STATUS_CYCLE_OVF BIT(0) +#define STATUS_MC0_AOR_OVF BIT(1) +#define STATUS_MC0_REQ_OVF BIT(3) +#define STATUS_MC1_AOR_OVF BIT(4) +#define STATUS_MC1_REQ_OVF BIT(6) +#define STATUS_MC2_AOR_OVF BIT(7) +#define STATUS_MC2_REQ_OVF BIT(9) + +/* Events. */ +#define EVENT_CYCLES 0x0 +#define EVENT_REQ 0x1 +#define EVENT_AOR 0x2 + +#define NUM_EVENTS 0x3 +#define MASK_EVENT 0x3 +#define MAX_ACTIVE_EVENTS 32 + +#define ACTIVE_CPU_MASK 0x0 +#define ASSOCIATED_CPU_MASK 0x1 + +static unsigned long cmem_lat_pmu_cpuhp_state; + +struct cmem_lat_pmu_hw_events { + struct perf_event *events[MAX_ACTIVE_EVENTS]; + DECLARE_BITMAP(used_ctrs, MAX_ACTIVE_EVENTS); +}; + +struct cmem_lat_pmu { + struct pmu pmu; + struct device *dev; + const char *name; + const char *identifier; + void __iomem *base[NUM_INSTANCES + 1]; + cpumask_t associated_cpus; + cpumask_t active_cpu; + struct hlist_node node; + struct cmem_lat_pmu_hw_events hw_events; +}; + +#define to_cmem_lat_pmu(p) \ + container_of(p, struct cmem_lat_pmu, pmu) + + +/* Get event type from perf_event. */ +static inline u32 get_event_type(struct perf_event *event) +{ + return (event->attr.config) & MASK_EVENT; +} + +/* PMU operations. */ +static int cmem_lat_pmu_get_event_idx(struct cmem_lat_pmu_hw_events *hw_events, + struct perf_event *event) +{ + unsigned int idx; + + idx = find_first_zero_bit(hw_events->used_ctrs, MAX_ACTIVE_EVENTS); + if (idx >= MAX_ACTIVE_EVENTS) + return -EAGAIN; + + set_bit(idx, hw_events->used_ctrs); + + return idx; +} + +static bool cmem_lat_pmu_validate_event(struct pmu *pmu, + struct cmem_lat_pmu_hw_events *hw_events, + struct perf_event *event) +{ + if (is_software_event(event)) + return true; + + /* Reject groups spanning multiple HW PMUs. */ + if (event->pmu != pmu) + return false; + + return (cmem_lat_pmu_get_event_idx(hw_events, event) >= 0); +} + +/* + * Make sure the group of events can be scheduled at once + * on the PMU. + */ +static bool cmem_lat_pmu_validate_group(struct perf_event *event) +{ + struct perf_event *sibling, *leader = event->group_leader; + struct cmem_lat_pmu_hw_events fake_hw_events; + + if (event->group_leader == event) + return true; + + memset(&fake_hw_events, 0, sizeof(fake_hw_events)); + + if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, leader)) + return false; + + for_each_sibling_event(sibling, leader) { + if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, + sibling)) + return false; + } + + return cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, event); +} + +static int cmem_lat_pmu_event_init(struct perf_event *event) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u32 event_type = get_event_type(event); + + if (event->attr.type != event->pmu->type || + event_type >= NUM_EVENTS) + return -ENOENT; + + /* + * Following other "uncore" PMUs, we do not support sampling mode or + * attach to a task (per-process mode). + */ + if (is_sampling_event(event)) { + dev_dbg(cmem_lat_pmu->pmu.dev, + "Can't support sampling events\n"); + return -EOPNOTSUPP; + } + + if (event->cpu < 0 || event->attach_state & PERF_ATTACH_TASK) { + dev_dbg(cmem_lat_pmu->pmu.dev, + "Can't support per-task counters\n"); + return -EINVAL; + } + + /* + * Make sure the CPU assignment is on one of the CPUs associated with + * this PMU. + */ + if (!cpumask_test_cpu(event->cpu, &cmem_lat_pmu->associated_cpus)) { + dev_dbg(cmem_lat_pmu->pmu.dev, + "Requested cpu is not associated with the PMU\n"); + return -EINVAL; + } + + /* Enforce the current active CPU to handle the events in this PMU. */ + event->cpu = cpumask_first(&cmem_lat_pmu->active_cpu); + if (event->cpu >= nr_cpu_ids) + return -EINVAL; + + if (!cmem_lat_pmu_validate_group(event)) + return -EINVAL; + + hwc->idx = -1; + hwc->config = event_type; + + return 0; +} + +static u64 cmem_lat_pmu_read_status(struct cmem_lat_pmu *cmem_lat_pmu, + unsigned int inst) +{ + return readq(cmem_lat_pmu->base[inst] + STATUS); +} + +static u64 cmem_lat_pmu_read_cycle_counter(struct perf_event *event) +{ + const unsigned int instance = 0; + u64 status; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* + * Use the reading from first instance since all instances are + * identical. + */ + status = cmem_lat_pmu_read_status(cmem_lat_pmu, instance); + if (status & STATUS_CYCLE_OVF) + dev_warn(dev, "Cycle counter overflow\n"); + + return readq(cmem_lat_pmu->base[instance] + CYCLE_CNTR); +} + +static u64 cmem_lat_pmu_read_req_counter(struct perf_event *event) +{ + unsigned int i; + u64 status, val = 0; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* Sum up the counts from all instances. */ + for (i = 0; i < NUM_INSTANCES; i++) { + status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); + if (status & STATUS_MC0_REQ_OVF) + dev_warn(dev, "MC0 request counter overflow\n"); + if (status & STATUS_MC1_REQ_OVF) + dev_warn(dev, "MC1 request counter overflow\n"); + if (status & STATUS_MC2_REQ_OVF) + dev_warn(dev, "MC2 request counter overflow\n"); + + val += readq(cmem_lat_pmu->base[i] + MC0_REQ_CNTR); + val += readq(cmem_lat_pmu->base[i] + MC1_REQ_CNTR); + val += readq(cmem_lat_pmu->base[i] + MC2_REQ_CNTR); + } + + return val; +} + +static u64 cmem_lat_pmu_read_aor_counter(struct perf_event *event) +{ + unsigned int i; + u64 status, val = 0; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* Sum up the counts from all instances. */ + for (i = 0; i < NUM_INSTANCES; i++) { + status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); + if (status & STATUS_MC0_AOR_OVF) + dev_warn(dev, "MC0 AOR counter overflow\n"); + if (status & STATUS_MC1_AOR_OVF) + dev_warn(dev, "MC1 AOR counter overflow\n"); + if (status & STATUS_MC2_AOR_OVF) + dev_warn(dev, "MC2 AOR counter overflow\n"); + + val += readq(cmem_lat_pmu->base[i] + MC0_AOR_CNTR); + val += readq(cmem_lat_pmu->base[i] + MC1_AOR_CNTR); + val += readq(cmem_lat_pmu->base[i] + MC2_AOR_CNTR); + } + + return val; +} + +static u64 (*read_counter_fn[NUM_EVENTS])(struct perf_event *) = { + [EVENT_CYCLES] = cmem_lat_pmu_read_cycle_counter, + [EVENT_REQ] = cmem_lat_pmu_read_req_counter, + [EVENT_AOR] = cmem_lat_pmu_read_aor_counter, +}; + +static void cmem_lat_pmu_event_update(struct perf_event *event) +{ + u32 event_type; + u64 prev, now; + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + event_type = hwc->config; + + do { + prev = local64_read(&hwc->prev_count); + now = read_counter_fn[event_type](event); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); + + local64_add(now - prev, &event->count); + + hwc->state |= PERF_HES_UPTODATE; +} + +static void cmem_lat_pmu_start(struct perf_event *event, int pmu_flags) +{ + event->hw.state = 0; +} + +static void cmem_lat_pmu_stop(struct perf_event *event, int pmu_flags) +{ + event->hw.state |= PERF_HES_STOPPED; +} + +static int cmem_lat_pmu_add(struct perf_event *event, int flags) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx; + + if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &cmem_lat_pmu->associated_cpus))) + return -ENOENT; + + idx = cmem_lat_pmu_get_event_idx(hw_events, event); + if (idx < 0) + return idx; + + hw_events->events[idx] = event; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + cmem_lat_pmu_start(event, PERF_EF_RELOAD); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void cmem_lat_pmu_del(struct perf_event *event, int flags) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + cmem_lat_pmu_stop(event, PERF_EF_UPDATE); + + hw_events->events[idx] = NULL; + + clear_bit(idx, hw_events->used_ctrs); + + perf_event_update_userpage(event); +} + +static void cmem_lat_pmu_read(struct perf_event *event) +{ + cmem_lat_pmu_event_update(event); +} + +static inline void cmem_lat_pmu_cg_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, u64 val) +{ + writeq(val, BCAST(cmem_lat_pmu) + CG_CTRL); +} + +static inline void cmem_lat_pmu_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, u64 val) +{ + writeq(val, BCAST(cmem_lat_pmu) + CTRL); +} + +static void cmem_lat_pmu_enable(struct pmu *pmu) +{ + bool disabled; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + + disabled = bitmap_empty( + cmem_lat_pmu->hw_events.used_ctrs, MAX_ACTIVE_EVENTS); + + if (disabled) + return; + + /* Enable all the counters. */ + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_ENABLE); + cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_ENABLE); +} + +static void cmem_lat_pmu_disable(struct pmu *pmu) +{ + int idx; + struct perf_event *event; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + + /* Disable all the counters. */ + cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_DISABLE); + + /* + * The counters will start from 0 again on restart. + * Update the events immediately to avoid losing the counts. + */ + for_each_set_bit( + idx, cmem_lat_pmu->hw_events.used_ctrs, MAX_ACTIVE_EVENTS) { + event = cmem_lat_pmu->hw_events.events[idx]; + + if (!event) + continue; + + cmem_lat_pmu_event_update(event); + + local64_set(&event->hw.prev_count, 0ULL); + } + + cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_CLR); + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_DISABLE); +} + +/* PMU identifier attribute. */ + +static ssize_t cmem_lat_pmu_identifier_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(dev_get_drvdata(dev)); + + return sysfs_emit(page, "%s\n", cmem_lat_pmu->identifier); +} + +static struct device_attribute cmem_lat_pmu_identifier_attr = + __ATTR(identifier, 0444, cmem_lat_pmu_identifier_show, NULL); + +static struct attribute *cmem_lat_pmu_identifier_attrs[] = { + &cmem_lat_pmu_identifier_attr.attr, + NULL, +}; + +static struct attribute_group cmem_lat_pmu_identifier_attr_group = { + .attrs = cmem_lat_pmu_identifier_attrs, +}; + +/* Format attributes. */ + +#define NV_PMU_EXT_ATTR(_name, _func, _config) \ + (&((struct dev_ext_attribute[]){ \ + { \ + .attr = __ATTR(_name, 0444, _func, NULL), \ + .var = (void *)_config \ + } \ + })[0].attr.attr) + +static struct attribute *cmem_lat_pmu_formats[] = { + NV_PMU_EXT_ATTR(event, device_show_string, "config:0-1"), + NULL, +}; + +static const struct attribute_group cmem_lat_pmu_format_group = { + .name = "format", + .attrs = cmem_lat_pmu_formats, +}; + +/* Event attributes. */ + +static ssize_t cmem_lat_pmu_sysfs_event_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, typeof(*pmu_attr), attr); + return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); +} + +#define NV_PMU_EVENT_ATTR(_name, _config) \ + PMU_EVENT_ATTR_ID(_name, cmem_lat_pmu_sysfs_event_show, _config) + +static struct attribute *cmem_lat_pmu_events[] = { + NV_PMU_EVENT_ATTR(cycles, EVENT_CYCLES), + NV_PMU_EVENT_ATTR(rd_req, EVENT_REQ), + NV_PMU_EVENT_ATTR(rd_cum_outs, EVENT_AOR), + NULL +}; + +static const struct attribute_group cmem_lat_pmu_events_group = { + .name = "events", + .attrs = cmem_lat_pmu_events, +}; + +/* Cpumask attributes. */ + +static ssize_t cmem_lat_pmu_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pmu *pmu = dev_get_drvdata(dev); + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + struct dev_ext_attribute *eattr = + container_of(attr, struct dev_ext_attribute, attr); + unsigned long mask_id = (unsigned long)eattr->var; + const cpumask_t *cpumask; + + switch (mask_id) { + case ACTIVE_CPU_MASK: + cpumask = &cmem_lat_pmu->active_cpu; + break; + case ASSOCIATED_CPU_MASK: + cpumask = &cmem_lat_pmu->associated_cpus; + break; + default: + return 0; + } + return cpumap_print_to_pagebuf(true, buf, cpumask); +} + +#define NV_PMU_CPUMASK_ATTR(_name, _config) \ + NV_PMU_EXT_ATTR(_name, cmem_lat_pmu_cpumask_show, \ + (unsigned long)_config) + +static struct attribute *cmem_lat_pmu_cpumask_attrs[] = { + NV_PMU_CPUMASK_ATTR(cpumask, ACTIVE_CPU_MASK), + NV_PMU_CPUMASK_ATTR(associated_cpus, ASSOCIATED_CPU_MASK), + NULL, +}; + +static const struct attribute_group cmem_lat_pmu_cpumask_attr_group = { + .attrs = cmem_lat_pmu_cpumask_attrs, +}; + +/* Per PMU device attribute groups. */ + +static const struct attribute_group *cmem_lat_pmu_attr_groups[] = { + &cmem_lat_pmu_identifier_attr_group, + &cmem_lat_pmu_format_group, + &cmem_lat_pmu_events_group, + &cmem_lat_pmu_cpumask_attr_group, + NULL, +}; + +static int cmem_lat_pmu_cpu_online(unsigned int cpu, struct hlist_node *node) +{ + struct cmem_lat_pmu *cmem_lat_pmu = + hlist_entry_safe(node, struct cmem_lat_pmu, node); + + if (!cpumask_test_cpu(cpu, &cmem_lat_pmu->associated_cpus)) + return 0; + + /* If the PMU is already managed, there is nothing to do */ + if (!cpumask_empty(&cmem_lat_pmu->active_cpu)) + return 0; + + /* Use this CPU for event counting */ + cpumask_set_cpu(cpu, &cmem_lat_pmu->active_cpu); + + return 0; +} + +static int cmem_lat_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) +{ + unsigned int dst; + + struct cmem_lat_pmu *cmem_lat_pmu = + hlist_entry_safe(node, struct cmem_lat_pmu, node); + + /* Nothing to do if this CPU doesn't own the PMU */ + if (!cpumask_test_and_clear_cpu(cpu, &cmem_lat_pmu->active_cpu)) + return 0; + + /* Choose a new CPU to migrate ownership of the PMU to */ + dst = cpumask_any_and_but(&cmem_lat_pmu->associated_cpus, + cpu_online_mask, cpu); + if (dst >= nr_cpu_ids) + return 0; + + /* Use this CPU for event counting */ + perf_pmu_migrate_context(&cmem_lat_pmu->pmu, cpu, dst); + cpumask_set_cpu(dst, &cmem_lat_pmu->active_cpu); + + return 0; +} + +static int cmem_lat_pmu_get_cpus(struct cmem_lat_pmu *cmem_lat_pmu, + unsigned int socket) +{ + int ret = 0, cpu; + + for_each_possible_cpu(cpu) { + if (cpu_to_node(cpu) == socket) + cpumask_set_cpu(cpu, &cmem_lat_pmu->associated_cpus); + } + + if (cpumask_empty(&cmem_lat_pmu->associated_cpus)) { + dev_dbg(cmem_lat_pmu->dev, + "No cpu associated with PMU socket-%u\n", socket); + ret = -ENODEV; + } + + return ret; +} + +static int cmem_lat_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct acpi_device *acpi_dev; + struct cmem_lat_pmu *cmem_lat_pmu; + char *name, *uid_str; + int ret, i; + u32 socket; + + acpi_dev = ACPI_COMPANION(dev); + if (!acpi_dev) + return -ENODEV; + + uid_str = acpi_device_uid(acpi_dev); + if (!uid_str) + return -ENODEV; + + ret = kstrtou32(uid_str, 0, &socket); + if (ret) + return ret; + + cmem_lat_pmu = devm_kzalloc(dev, sizeof(*cmem_lat_pmu), GFP_KERNEL); + name = devm_kasprintf(dev, GFP_KERNEL, "nvidia_cmem_latency_pmu_%u", socket); + if (!cmem_lat_pmu || !name) + return -ENOMEM; + + cmem_lat_pmu->dev = dev; + cmem_lat_pmu->name = name; + cmem_lat_pmu->identifier = acpi_device_hid(acpi_dev); + platform_set_drvdata(pdev, cmem_lat_pmu); + + cmem_lat_pmu->pmu = (struct pmu) { + .parent = &pdev->dev, + .task_ctx_nr = perf_invalid_context, + .pmu_enable = cmem_lat_pmu_enable, + .pmu_disable = cmem_lat_pmu_disable, + .event_init = cmem_lat_pmu_event_init, + .add = cmem_lat_pmu_add, + .del = cmem_lat_pmu_del, + .start = cmem_lat_pmu_start, + .stop = cmem_lat_pmu_stop, + .read = cmem_lat_pmu_read, + .attr_groups = cmem_lat_pmu_attr_groups, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | + PERF_PMU_CAP_NO_INTERRUPT, + }; + + /* Map the address of all the instances plus one for the broadcast. */ + for (i = 0; i < NUM_INSTANCES + 1; i++) { + cmem_lat_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); + if (IS_ERR(cmem_lat_pmu->base[i])) { + dev_err(dev, "Failed map address for instance %d\n", i); + return PTR_ERR(cmem_lat_pmu->base[i]); + } + } + + ret = cmem_lat_pmu_get_cpus(cmem_lat_pmu, socket); + if (ret) + return ret; + + ret = cpuhp_state_add_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); + if (ret) { + dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); + return ret; + } + + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_ENABLE); + cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_CLR); + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_DISABLE); + + ret = perf_pmu_register(&cmem_lat_pmu->pmu, name, -1); + if (ret) { + dev_err(&pdev->dev, "Failed to register PMU: %d\n", ret); + cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); + return ret; + } + + dev_dbg(&pdev->dev, "Registered %s PMU\n", name); + + return 0; +} + +static void cmem_lat_pmu_device_remove(struct platform_device *pdev) +{ + struct cmem_lat_pmu *cmem_lat_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&cmem_lat_pmu->pmu); + cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); +} + +static const struct acpi_device_id cmem_lat_pmu_acpi_match[] = { + { "NVDA2021", }, + { } +}; +MODULE_DEVICE_TABLE(acpi, cmem_lat_pmu_acpi_match); + +static struct platform_driver cmem_lat_pmu_driver = { + .driver = { + .name = "nvidia-t410-cmem-latency-pmu", + .acpi_match_table = ACPI_PTR(cmem_lat_pmu_acpi_match), + .suppress_bind_attrs = true, + }, + .probe = cmem_lat_pmu_probe, + .remove = cmem_lat_pmu_device_remove, +}; + +static int __init cmem_lat_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/nvidia/cmem_latency:online", + cmem_lat_pmu_cpu_online, + cmem_lat_pmu_cpu_teardown); + if (ret < 0) + return ret; + + cmem_lat_pmu_cpuhp_state = ret; + + return platform_driver_register(&cmem_lat_pmu_driver); +} + +static void __exit cmem_lat_pmu_exit(void) +{ + platform_driver_unregister(&cmem_lat_pmu_driver); + cpuhp_remove_multi_state(cmem_lat_pmu_cpuhp_state); +} + +module_init(cmem_lat_pmu_init); +module_exit(cmem_lat_pmu_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("NVIDIA Tegra410 CPU Memory Latency PMU driver"); +MODULE_AUTHOR("Besar Wicaksono "); From 5b3b644c0479a8ae9741c2051445e6c7f05bc7f7 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:54 +0000 Subject: [PATCH 118/464] NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 C2C PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Adds NVIDIA C2C PMU support in Tegra410 SOC. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit 5d68165043f57f1fa62916a7ba02e5a2919f8c5e noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 151 +++ drivers/perf/Kconfig | 7 + drivers/perf/Makefile | 1 + drivers/perf/nvidia_t410_c2c_pmu.c | 1061 +++++++++++++++++ 4 files changed, 1220 insertions(+) create mode 100644 drivers/perf/nvidia_t410_c2c_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 11fc1c88346a0..f81f356debe1f 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -9,6 +9,9 @@ metrics like memory bandwidth, latency, and utilization: * PCIE * PCIE-TGT * CPU Memory (CMEM) Latency +* NVLink-C2C +* NV-CLink +* NV-DLink PMU Driver ---------- @@ -367,3 +370,151 @@ see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. Example usage:: perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' + +NVLink-C2C PMU +-------------- + +This PMU monitors latency events of memory read/write requests that pass through +the NVIDIA Chip-to-Chip (C2C) interface. Bandwidth events are not available +in this PMU, unlike the C2C PMU in Grace (Tegra241 SoC). + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. + * IN_RD_REQ: the number of incoming read requests. + * IN_WR_CUM_OUTS: accumulated outstanding request (in cycles) of incoming write requests. + * IN_WR_REQ: the number of incoming write requests. + * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. + * OUT_RD_REQ: the number of outgoing read requests. + * OUT_WR_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing write requests. + * OUT_WR_REQ: the number of outgoing write requests. + * CYCLES: NVLink-C2C interface cycle counts. + +The incoming events count the reads/writes from remote device to the SoC. +The outgoing events count the reads/writes from the SoC to remote device. + +The sysfs /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_/peer +contains the information about the connected device. + +When the C2C interface is connected to GPU(s), the user can use the +"gpu_mask" parameter to filter traffic to/from specific GPU(s). Each bit represents the GPU +index, e.g. "gpu_mask=0x1" corresponds to GPU 0 and "gpu_mask=0x3" is for GPU 0 and 1. +The PMU will monitor all GPUs by default if not specified. + +When connected to another SoC, only the read events are available. + +The events can be used to calculate the average latency of the read/write requests:: + + C2C_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + IN_WR_AVG_LATENCY_IN_CYCLES = IN_WR_CUM_OUTS / IN_WR_REQ + IN_WR_AVG_LATENCY_IN_NS = IN_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ + OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + OUT_WR_AVG_LATENCY_IN_CYCLES = OUT_WR_CUM_OUTS / OUT_WR_REQ + OUT_WR_AVG_LATENCY_IN_NS = OUT_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + +Example usage: + + * Count incoming traffic from all GPUs connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_req/ + + * Count incoming traffic from GPU 0 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x1/ + + * Count incoming traffic from GPU 1 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x2/ + + * Count outgoing traffic to all GPUs connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_req/ + + * Count outgoing traffic to GPU 0 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x1/ + + * Count outgoing traffic to GPU 1 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x2/ + +NV-CLink PMU +------------ + +This PMU monitors latency events of memory read requests that pass through +the NV-CLINK interface. Bandwidth events are not available in this PMU. +In Tegra410 SoC, the NV-CLink interface is used to connect to another Tegra410 +SoC and this PMU only counts read traffic. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvclink_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. + * IN_RD_REQ: the number of incoming read requests. + * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. + * OUT_RD_REQ: the number of outgoing read requests. + * CYCLES: NV-CLINK interface cycle counts. + +The incoming events count the reads from remote device to the SoC. +The outgoing events count the reads from the SoC to remote device. + +The events can be used to calculate the average latency of the read requests:: + + CLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ + + OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ + OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ + +Example usage: + + * Count incoming read traffic from remote SoC connected via NV-CLINK:: + + perf stat -a -e nvidia_nvclink_pmu_0/in_rd_req/ + + * Count outgoing read traffic to remote SoC connected via NV-CLINK:: + + perf stat -a -e nvidia_nvclink_pmu_0/out_rd_req/ + +NV-DLink PMU +------------ + +This PMU monitors latency events of memory read requests that pass through +the NV-DLINK interface. Bandwidth events are not available in this PMU. +In Tegra410 SoC, this PMU only counts CXL memory read traffic. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvdlink_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding read requests (in cycles) to CXL memory. + * IN_RD_REQ: the number of read requests to CXL memory. + * CYCLES: NV-DLINK interface cycle counts. + +The events can be used to calculate the average latency of the read requests:: + + DLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / DLINK_FREQ_IN_GHZ + +Example usage: + + * Count read events to CXL memory:: + + perf stat -a -e '{nvidia_nvdlink_pmu_0/in_rd_req/,nvidia_nvdlink_pmu_0/in_rd_cum_outs/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 9fed3c41d5ea0..7ee36efe6bc0f 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -318,4 +318,11 @@ config NVIDIA_TEGRA410_CMEM_LATENCY_PMU Enable perf support for CPU memory latency counters monitoring on NVIDIA Tegra410 SoC. +config NVIDIA_TEGRA410_C2C_PMU + tristate "NVIDIA Tegra410 C2C PMU" + depends on ARM64 && ACPI + help + Enable perf support for counters in NVIDIA C2C interface of NVIDIA + Tegra410 SoC. + endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index 4aa6aad393c2d..eb8a022dad9a7 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -36,3 +36,4 @@ obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o +obj-$(CONFIG_NVIDIA_TEGRA410_C2C_PMU) += nvidia_t410_c2c_pmu.o diff --git a/drivers/perf/nvidia_t410_c2c_pmu.c b/drivers/perf/nvidia_t410_c2c_pmu.c new file mode 100644 index 0000000000000..362e0e5f8b24c --- /dev/null +++ b/drivers/perf/nvidia_t410_c2c_pmu.c @@ -0,0 +1,1061 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * NVIDIA Tegra410 C2C PMU driver. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* The C2C interface types in Tegra410. */ +#define C2C_TYPE_NVLINK 0x0 +#define C2C_TYPE_NVCLINK 0x1 +#define C2C_TYPE_NVDLINK 0x2 +#define C2C_TYPE_COUNT 0x3 + +/* The type of the peer device connected to the C2C interface. */ +#define C2C_PEER_TYPE_CPU 0x0 +#define C2C_PEER_TYPE_GPU 0x1 +#define C2C_PEER_TYPE_CXLMEM 0x2 +#define C2C_PEER_TYPE_COUNT 0x3 + +/* The number of peer devices can be connected to the C2C interface. */ +#define C2C_NR_PEER_CPU 0x1 +#define C2C_NR_PEER_GPU 0x2 +#define C2C_NR_PEER_CXLMEM 0x1 +#define C2C_NR_PEER_MAX 0x2 + +/* Number of instances on each interface. */ +#define C2C_NR_INST_NVLINK 14 +#define C2C_NR_INST_NVCLINK 12 +#define C2C_NR_INST_NVDLINK 16 +#define C2C_NR_INST_MAX 16 + +/* Register offsets. */ +#define C2C_CTRL 0x864 +#define C2C_IN_STATUS 0x868 +#define C2C_CYCLE_CNTR 0x86c +#define C2C_IN_RD_CUM_OUTS_CNTR 0x874 +#define C2C_IN_RD_REQ_CNTR 0x87c +#define C2C_IN_WR_CUM_OUTS_CNTR 0x884 +#define C2C_IN_WR_REQ_CNTR 0x88c +#define C2C_OUT_STATUS 0x890 +#define C2C_OUT_RD_CUM_OUTS_CNTR 0x898 +#define C2C_OUT_RD_REQ_CNTR 0x8a0 +#define C2C_OUT_WR_CUM_OUTS_CNTR 0x8a8 +#define C2C_OUT_WR_REQ_CNTR 0x8b0 + +/* C2C_IN_STATUS register field. */ +#define C2C_IN_STATUS_CYCLE_OVF BIT(0) +#define C2C_IN_STATUS_IN_RD_CUM_OUTS_OVF BIT(1) +#define C2C_IN_STATUS_IN_RD_REQ_OVF BIT(2) +#define C2C_IN_STATUS_IN_WR_CUM_OUTS_OVF BIT(3) +#define C2C_IN_STATUS_IN_WR_REQ_OVF BIT(4) + +/* C2C_OUT_STATUS register field. */ +#define C2C_OUT_STATUS_OUT_RD_CUM_OUTS_OVF BIT(0) +#define C2C_OUT_STATUS_OUT_RD_REQ_OVF BIT(1) +#define C2C_OUT_STATUS_OUT_WR_CUM_OUTS_OVF BIT(2) +#define C2C_OUT_STATUS_OUT_WR_REQ_OVF BIT(3) + +/* Events. */ +#define C2C_EVENT_CYCLES 0x0 +#define C2C_EVENT_IN_RD_CUM_OUTS 0x1 +#define C2C_EVENT_IN_RD_REQ 0x2 +#define C2C_EVENT_IN_WR_CUM_OUTS 0x3 +#define C2C_EVENT_IN_WR_REQ 0x4 +#define C2C_EVENT_OUT_RD_CUM_OUTS 0x5 +#define C2C_EVENT_OUT_RD_REQ 0x6 +#define C2C_EVENT_OUT_WR_CUM_OUTS 0x7 +#define C2C_EVENT_OUT_WR_REQ 0x8 + +#define C2C_NUM_EVENTS 0x9 +#define C2C_MASK_EVENT 0xFF +#define C2C_MAX_ACTIVE_EVENTS 32 + +#define C2C_ACTIVE_CPU_MASK 0x0 +#define C2C_ASSOCIATED_CPU_MASK 0x1 + +/* + * Maximum poll count for reading counter value using high-low-high sequence. + */ +#define HILOHI_MAX_POLL 1000 + +static unsigned long nv_c2c_pmu_cpuhp_state; + +/* PMU descriptor. */ + +/* Tracks the events assigned to the PMU for a given logical index. */ +struct nv_c2c_pmu_hw_events { + /* The events that are active. */ + struct perf_event *events[C2C_MAX_ACTIVE_EVENTS]; + + /* + * Each bit indicates a logical counter is being used (or not) for an + * event. + */ + DECLARE_BITMAP(used_ctrs, C2C_MAX_ACTIVE_EVENTS); +}; + +struct nv_c2c_pmu { + struct pmu pmu; + struct device *dev; + struct acpi_device *acpi_dev; + + const char *name; + const char *identifier; + + unsigned int c2c_type; + unsigned int peer_type; + unsigned int socket; + unsigned int nr_inst; + unsigned int nr_peer; + unsigned long peer_insts[C2C_NR_PEER_MAX][BITS_TO_LONGS(C2C_NR_INST_MAX)]; + u32 filter_default; + + struct nv_c2c_pmu_hw_events hw_events; + + cpumask_t associated_cpus; + cpumask_t active_cpu; + + struct hlist_node cpuhp_node; + + struct attribute **formats; + const struct attribute_group *attr_groups[6]; + + void __iomem *base_broadcast; + void __iomem *base[C2C_NR_INST_MAX]; +}; + +#define to_c2c_pmu(p) (container_of(p, struct nv_c2c_pmu, pmu)) + +/* Get event type from perf_event. */ +static inline u32 get_event_type(struct perf_event *event) +{ + return (event->attr.config) & C2C_MASK_EVENT; +} + +static inline u32 get_filter_mask(struct perf_event *event) +{ + u32 filter; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + + filter = ((u32)event->attr.config1) & c2c_pmu->filter_default; + if (filter == 0) + filter = c2c_pmu->filter_default; + + return filter; +} + +/* PMU operations. */ + +static int nv_c2c_pmu_get_event_idx(struct nv_c2c_pmu_hw_events *hw_events, + struct perf_event *event) +{ + u32 idx; + + idx = find_first_zero_bit(hw_events->used_ctrs, C2C_MAX_ACTIVE_EVENTS); + if (idx >= C2C_MAX_ACTIVE_EVENTS) + return -EAGAIN; + + set_bit(idx, hw_events->used_ctrs); + + return idx; +} + +static bool +nv_c2c_pmu_validate_event(struct pmu *pmu, + struct nv_c2c_pmu_hw_events *hw_events, + struct perf_event *event) +{ + if (is_software_event(event)) + return true; + + /* Reject groups spanning multiple HW PMUs. */ + if (event->pmu != pmu) + return false; + + return nv_c2c_pmu_get_event_idx(hw_events, event) >= 0; +} + +/* + * Make sure the group of events can be scheduled at once + * on the PMU. + */ +static bool nv_c2c_pmu_validate_group(struct perf_event *event) +{ + struct perf_event *sibling, *leader = event->group_leader; + struct nv_c2c_pmu_hw_events fake_hw_events; + + if (event->group_leader == event) + return true; + + memset(&fake_hw_events, 0, sizeof(fake_hw_events)); + + if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, leader)) + return false; + + for_each_sibling_event(sibling, leader) { + if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, + sibling)) + return false; + } + + return nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, event); +} + +static int nv_c2c_pmu_event_init(struct perf_event *event) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u32 event_type = get_event_type(event); + + if (event->attr.type != event->pmu->type || + event_type >= C2C_NUM_EVENTS) + return -ENOENT; + + /* + * Following other "uncore" PMUs, we do not support sampling mode or + * attach to a task (per-process mode). + */ + if (is_sampling_event(event)) { + dev_dbg(c2c_pmu->pmu.dev, "Can't support sampling events\n"); + return -EOPNOTSUPP; + } + + if (event->cpu < 0 || event->attach_state & PERF_ATTACH_TASK) { + dev_dbg(c2c_pmu->pmu.dev, "Can't support per-task counters\n"); + return -EINVAL; + } + + /* + * Make sure the CPU assignment is on one of the CPUs associated with + * this PMU. + */ + if (!cpumask_test_cpu(event->cpu, &c2c_pmu->associated_cpus)) { + dev_dbg(c2c_pmu->pmu.dev, + "Requested cpu is not associated with the PMU\n"); + return -EINVAL; + } + + /* Enforce the current active CPU to handle the events in this PMU. */ + event->cpu = cpumask_first(&c2c_pmu->active_cpu); + if (event->cpu >= nr_cpu_ids) + return -EINVAL; + + if (!nv_c2c_pmu_validate_group(event)) + return -EINVAL; + + hwc->idx = -1; + hwc->config = event_type; + + return 0; +} + +/* + * Read 64-bit register as a pair of 32-bit registers using hi-lo-hi sequence. + */ +static u64 read_reg64_hilohi(const void __iomem *addr, u32 max_poll_count) +{ + u32 val_lo, val_hi; + u64 val; + + /* Use high-low-high sequence to avoid tearing */ + do { + if (max_poll_count-- == 0) { + pr_err("NV C2C PMU: timeout hi-low-high sequence\n"); + return 0; + } + + val_hi = readl(addr + 4); + val_lo = readl(addr); + } while (val_hi != readl(addr + 4)); + + val = (((u64)val_hi << 32) | val_lo); + + return val; +} + +static void nv_c2c_pmu_check_status(struct nv_c2c_pmu *c2c_pmu, u32 instance) +{ + u32 in_status, out_status; + + in_status = readl(c2c_pmu->base[instance] + C2C_IN_STATUS); + out_status = readl(c2c_pmu->base[instance] + C2C_OUT_STATUS); + + if (in_status || out_status) + dev_warn(c2c_pmu->dev, + "C2C PMU overflow in: 0x%x, out: 0x%x\n", + in_status, out_status); +} + +static u32 nv_c2c_ctr_offset[C2C_NUM_EVENTS] = { + [C2C_EVENT_CYCLES] = C2C_CYCLE_CNTR, + [C2C_EVENT_IN_RD_CUM_OUTS] = C2C_IN_RD_CUM_OUTS_CNTR, + [C2C_EVENT_IN_RD_REQ] = C2C_IN_RD_REQ_CNTR, + [C2C_EVENT_IN_WR_CUM_OUTS] = C2C_IN_WR_CUM_OUTS_CNTR, + [C2C_EVENT_IN_WR_REQ] = C2C_IN_WR_REQ_CNTR, + [C2C_EVENT_OUT_RD_CUM_OUTS] = C2C_OUT_RD_CUM_OUTS_CNTR, + [C2C_EVENT_OUT_RD_REQ] = C2C_OUT_RD_REQ_CNTR, + [C2C_EVENT_OUT_WR_CUM_OUTS] = C2C_OUT_WR_CUM_OUTS_CNTR, + [C2C_EVENT_OUT_WR_REQ] = C2C_OUT_WR_REQ_CNTR, +}; + +static u64 nv_c2c_pmu_read_counter(struct perf_event *event) +{ + u32 ctr_id, ctr_offset, filter_mask, filter_idx, inst_idx; + unsigned long *inst_mask; + DECLARE_BITMAP(filter_bitmap, C2C_NR_PEER_MAX); + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + u64 val = 0; + + filter_mask = get_filter_mask(event); + bitmap_from_arr32(filter_bitmap, &filter_mask, c2c_pmu->nr_peer); + + ctr_id = event->hw.config; + ctr_offset = nv_c2c_ctr_offset[ctr_id]; + + for_each_set_bit(filter_idx, filter_bitmap, c2c_pmu->nr_peer) { + inst_mask = c2c_pmu->peer_insts[filter_idx]; + for_each_set_bit(inst_idx, inst_mask, c2c_pmu->nr_inst) { + nv_c2c_pmu_check_status(c2c_pmu, inst_idx); + + /* + * Each instance share same clock and the driver always + * enables all instances. So we can use the counts from + * one instance for cycle counter. + */ + if (ctr_id == C2C_EVENT_CYCLES) + return read_reg64_hilohi( + c2c_pmu->base[inst_idx] + ctr_offset, + HILOHI_MAX_POLL); + + /* + * For other events, sum up the counts from all instances. + */ + val += read_reg64_hilohi( + c2c_pmu->base[inst_idx] + ctr_offset, + HILOHI_MAX_POLL); + } + } + + return val; +} + +static void nv_c2c_pmu_event_update(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 prev, now; + + do { + prev = local64_read(&hwc->prev_count); + now = nv_c2c_pmu_read_counter(event); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); + + local64_add(now - prev, &event->count); +} + +static void nv_c2c_pmu_start(struct perf_event *event, int pmu_flags) +{ + event->hw.state = 0; +} + +static void nv_c2c_pmu_stop(struct perf_event *event, int pmu_flags) +{ + event->hw.state |= PERF_HES_STOPPED; +} + +static int nv_c2c_pmu_add(struct perf_event *event, int flags) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx; + + if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &c2c_pmu->associated_cpus))) + return -ENOENT; + + idx = nv_c2c_pmu_get_event_idx(hw_events, event); + if (idx < 0) + return idx; + + hw_events->events[idx] = event; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + nv_c2c_pmu_start(event, PERF_EF_RELOAD); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void nv_c2c_pmu_del(struct perf_event *event, int flags) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + nv_c2c_pmu_stop(event, PERF_EF_UPDATE); + + hw_events->events[idx] = NULL; + + clear_bit(idx, hw_events->used_ctrs); + + perf_event_update_userpage(event); +} + +static void nv_c2c_pmu_read(struct perf_event *event) +{ + nv_c2c_pmu_event_update(event); +} + +static void nv_c2c_pmu_enable(struct pmu *pmu) +{ + void __iomem *bcast; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + + /* Check if any filter is enabled. */ + if (bitmap_empty(c2c_pmu->hw_events.used_ctrs, C2C_MAX_ACTIVE_EVENTS)) + return; + + /* Enable all the counters. */ + bcast = c2c_pmu->base_broadcast; + writel(0x1UL, bcast + C2C_CTRL); +} + +static void nv_c2c_pmu_disable(struct pmu *pmu) +{ + unsigned int idx; + void __iomem *bcast; + struct perf_event *event; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + + /* Disable all the counters. */ + bcast = c2c_pmu->base_broadcast; + writel(0x0UL, bcast + C2C_CTRL); + + /* + * The counters will start from 0 again on restart. + * Update the events immediately to avoid losing the counts. + */ + for_each_set_bit(idx, c2c_pmu->hw_events.used_ctrs, + C2C_MAX_ACTIVE_EVENTS) { + event = c2c_pmu->hw_events.events[idx]; + + if (!event) + continue; + + nv_c2c_pmu_event_update(event); + + local64_set(&event->hw.prev_count, 0ULL); + } +} + +/* PMU identifier attribute. */ + +static ssize_t nv_c2c_pmu_identifier_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); + + return sysfs_emit(page, "%s\n", c2c_pmu->identifier); +} + +static struct device_attribute nv_c2c_pmu_identifier_attr = + __ATTR(identifier, 0444, nv_c2c_pmu_identifier_show, NULL); + +static struct attribute *nv_c2c_pmu_identifier_attrs[] = { + &nv_c2c_pmu_identifier_attr.attr, + NULL, +}; + +static struct attribute_group nv_c2c_pmu_identifier_attr_group = { + .attrs = nv_c2c_pmu_identifier_attrs, +}; + +/* Peer attribute. */ + +static ssize_t nv_c2c_pmu_peer_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + const char *peer_type[C2C_PEER_TYPE_COUNT] = { + [C2C_PEER_TYPE_CPU] = "cpu", + [C2C_PEER_TYPE_GPU] = "gpu", + [C2C_PEER_TYPE_CXLMEM] = "cxlmem", + }; + + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); + return sysfs_emit(page, "nr_%s=%u\n", peer_type[c2c_pmu->peer_type], + c2c_pmu->nr_peer); +} + +static struct device_attribute nv_c2c_pmu_peer_attr = + __ATTR(peer, 0444, nv_c2c_pmu_peer_show, NULL); + +static struct attribute *nv_c2c_pmu_peer_attrs[] = { + &nv_c2c_pmu_peer_attr.attr, + NULL, +}; + +static struct attribute_group nv_c2c_pmu_peer_attr_group = { + .attrs = nv_c2c_pmu_peer_attrs, +}; + +/* Format attributes. */ + +#define NV_C2C_PMU_EXT_ATTR(_name, _func, _config) \ + (&((struct dev_ext_attribute[]){ \ + { \ + .attr = __ATTR(_name, 0444, _func, NULL), \ + .var = (void *)_config \ + } \ + })[0].attr.attr) + +#define NV_C2C_PMU_FORMAT_ATTR(_name, _config) \ + NV_C2C_PMU_EXT_ATTR(_name, device_show_string, _config) + +#define NV_C2C_PMU_FORMAT_EVENT_ATTR \ + NV_C2C_PMU_FORMAT_ATTR(event, "config:0-3") + +static struct attribute *nv_c2c_nvlink_pmu_formats[] = { + NV_C2C_PMU_FORMAT_EVENT_ATTR, + NV_C2C_PMU_FORMAT_ATTR(gpu_mask, "config1:0-1"), + NULL, +}; + +static struct attribute *nv_c2c_pmu_formats[] = { + NV_C2C_PMU_FORMAT_EVENT_ATTR, + NULL, +}; + +static struct attribute_group * +nv_c2c_pmu_alloc_format_attr_group(struct nv_c2c_pmu *c2c_pmu) +{ + struct attribute_group *format_group; + struct device *dev = c2c_pmu->dev; + + format_group = + devm_kzalloc(dev, sizeof(struct attribute_group), GFP_KERNEL); + if (!format_group) + return NULL; + + format_group->name = "format"; + format_group->attrs = c2c_pmu->formats; + + return format_group; +} + +/* Event attributes. */ + +static ssize_t nv_c2c_pmu_sysfs_event_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, typeof(*pmu_attr), attr); + return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); +} + +#define NV_C2C_PMU_EVENT_ATTR(_name, _config) \ + PMU_EVENT_ATTR_ID(_name, nv_c2c_pmu_sysfs_event_show, _config) + +static struct attribute *nv_c2c_pmu_events[] = { + NV_C2C_PMU_EVENT_ATTR(cycles, C2C_EVENT_CYCLES), + NV_C2C_PMU_EVENT_ATTR(in_rd_cum_outs, C2C_EVENT_IN_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_rd_req, C2C_EVENT_IN_RD_REQ), + NV_C2C_PMU_EVENT_ATTR(in_wr_cum_outs, C2C_EVENT_IN_WR_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_wr_req, C2C_EVENT_IN_WR_REQ), + NV_C2C_PMU_EVENT_ATTR(out_rd_cum_outs, C2C_EVENT_OUT_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(out_rd_req, C2C_EVENT_OUT_RD_REQ), + NV_C2C_PMU_EVENT_ATTR(out_wr_cum_outs, C2C_EVENT_OUT_WR_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(out_wr_req, C2C_EVENT_OUT_WR_REQ), + NULL +}; + +static umode_t +nv_c2c_pmu_event_attr_is_visible(struct kobject *kobj, struct attribute *attr, + int unused) +{ + struct device *dev = kobj_to_dev(kobj); + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); + struct perf_pmu_events_attr *eattr; + + eattr = container_of(attr, typeof(*eattr), attr.attr); + + if (c2c_pmu->c2c_type == C2C_TYPE_NVDLINK) { + /* Only incoming reads are available. */ + switch (eattr->id) { + case C2C_EVENT_IN_WR_CUM_OUTS: + case C2C_EVENT_IN_WR_REQ: + case C2C_EVENT_OUT_RD_CUM_OUTS: + case C2C_EVENT_OUT_RD_REQ: + case C2C_EVENT_OUT_WR_CUM_OUTS: + case C2C_EVENT_OUT_WR_REQ: + return 0; + default: + return attr->mode; + } + } else { + /* Hide the write events if C2C connected to another SoC. */ + if (c2c_pmu->peer_type == C2C_PEER_TYPE_CPU) { + switch (eattr->id) { + case C2C_EVENT_IN_WR_CUM_OUTS: + case C2C_EVENT_IN_WR_REQ: + case C2C_EVENT_OUT_WR_CUM_OUTS: + case C2C_EVENT_OUT_WR_REQ: + return 0; + default: + return attr->mode; + } + } + } + + return attr->mode; +} + +static const struct attribute_group nv_c2c_pmu_events_group = { + .name = "events", + .attrs = nv_c2c_pmu_events, + .is_visible = nv_c2c_pmu_event_attr_is_visible, +}; + +/* Cpumask attributes. */ + +static ssize_t nv_c2c_pmu_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pmu *pmu = dev_get_drvdata(dev); + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + struct dev_ext_attribute *eattr = + container_of(attr, struct dev_ext_attribute, attr); + unsigned long mask_id = (unsigned long)eattr->var; + const cpumask_t *cpumask; + + switch (mask_id) { + case C2C_ACTIVE_CPU_MASK: + cpumask = &c2c_pmu->active_cpu; + break; + case C2C_ASSOCIATED_CPU_MASK: + cpumask = &c2c_pmu->associated_cpus; + break; + default: + return 0; + } + return cpumap_print_to_pagebuf(true, buf, cpumask); +} + +#define NV_C2C_PMU_CPUMASK_ATTR(_name, _config) \ + NV_C2C_PMU_EXT_ATTR(_name, nv_c2c_pmu_cpumask_show, \ + (unsigned long)_config) + +static struct attribute *nv_c2c_pmu_cpumask_attrs[] = { + NV_C2C_PMU_CPUMASK_ATTR(cpumask, C2C_ACTIVE_CPU_MASK), + NV_C2C_PMU_CPUMASK_ATTR(associated_cpus, C2C_ASSOCIATED_CPU_MASK), + NULL, +}; + +static const struct attribute_group nv_c2c_pmu_cpumask_attr_group = { + .attrs = nv_c2c_pmu_cpumask_attrs, +}; + +/* Per PMU device attribute groups. */ + +static int nv_c2c_pmu_alloc_attr_groups(struct nv_c2c_pmu *c2c_pmu) +{ + const struct attribute_group **attr_groups = c2c_pmu->attr_groups; + + attr_groups[0] = nv_c2c_pmu_alloc_format_attr_group(c2c_pmu); + attr_groups[1] = &nv_c2c_pmu_events_group; + attr_groups[2] = &nv_c2c_pmu_cpumask_attr_group; + attr_groups[3] = &nv_c2c_pmu_identifier_attr_group; + attr_groups[4] = &nv_c2c_pmu_peer_attr_group; + + if (!attr_groups[0]) + return -ENOMEM; + + return 0; +} + +static int nv_c2c_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct nv_c2c_pmu *c2c_pmu = + hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); + + if (!cpumask_test_cpu(cpu, &c2c_pmu->associated_cpus)) + return 0; + + /* If the PMU is already managed, there is nothing to do */ + if (!cpumask_empty(&c2c_pmu->active_cpu)) + return 0; + + /* Use this CPU for event counting */ + cpumask_set_cpu(cpu, &c2c_pmu->active_cpu); + + return 0; +} + +static int nv_c2c_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) +{ + unsigned int dst; + + struct nv_c2c_pmu *c2c_pmu = + hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); + + /* Nothing to do if this CPU doesn't own the PMU */ + if (!cpumask_test_and_clear_cpu(cpu, &c2c_pmu->active_cpu)) + return 0; + + /* Choose a new CPU to migrate ownership of the PMU to */ + dst = cpumask_any_and_but(&c2c_pmu->associated_cpus, + cpu_online_mask, cpu); + if (dst >= nr_cpu_ids) + return 0; + + /* Use this CPU for event counting */ + perf_pmu_migrate_context(&c2c_pmu->pmu, cpu, dst); + cpumask_set_cpu(dst, &c2c_pmu->active_cpu); + + return 0; +} + +static int nv_c2c_pmu_get_cpus(struct nv_c2c_pmu *c2c_pmu) +{ + int ret = 0, socket = c2c_pmu->socket, cpu; + + for_each_possible_cpu(cpu) { + if (cpu_to_node(cpu) == socket) + cpumask_set_cpu(cpu, &c2c_pmu->associated_cpus); + } + + if (cpumask_empty(&c2c_pmu->associated_cpus)) { + dev_dbg(c2c_pmu->dev, + "No cpu associated with C2C PMU socket-%u\n", socket); + ret = -ENODEV; + } + + return ret; +} + +static int nv_c2c_pmu_init_socket(struct nv_c2c_pmu *c2c_pmu) +{ + const char *uid_str; + int ret, socket; + + uid_str = acpi_device_uid(c2c_pmu->acpi_dev); + if (!uid_str) { + ret = -ENODEV; + goto fail; + } + + ret = kstrtou32(uid_str, 0, &socket); + if (ret) + goto fail; + + c2c_pmu->socket = socket; + return 0; + +fail: + dev_err(c2c_pmu->dev, "Failed to initialize socket\n"); + return ret; +} + +static int nv_c2c_pmu_init_id(struct nv_c2c_pmu *c2c_pmu) +{ + const char *name_fmt[C2C_TYPE_COUNT] = { + [C2C_TYPE_NVLINK] = "nvidia_nvlink_c2c_pmu_%u", + [C2C_TYPE_NVCLINK] = "nvidia_nvclink_pmu_%u", + [C2C_TYPE_NVDLINK] = "nvidia_nvdlink_pmu_%u", + }; + + char *name; + int ret; + + name = devm_kasprintf(c2c_pmu->dev, GFP_KERNEL, + name_fmt[c2c_pmu->c2c_type], c2c_pmu->socket); + if (!name) { + ret = -ENOMEM; + goto fail; + } + + c2c_pmu->name = name; + + c2c_pmu->identifier = acpi_device_hid(c2c_pmu->acpi_dev); + + return 0; + +fail: + dev_err(c2c_pmu->dev, "Failed to initialize name\n"); + return ret; +} + +static int nv_c2c_pmu_init_filter(struct nv_c2c_pmu *c2c_pmu) +{ + u32 cpu_en = 0; + struct device *dev = c2c_pmu->dev; + + if (c2c_pmu->c2c_type == C2C_TYPE_NVDLINK) { + c2c_pmu->peer_type = C2C_PEER_TYPE_CXLMEM; + + c2c_pmu->nr_inst = C2C_NR_INST_NVDLINK; + c2c_pmu->peer_insts[0][0] = (1UL << c2c_pmu->nr_inst) - 1; + + c2c_pmu->nr_peer = C2C_NR_PEER_CXLMEM; + c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; + + c2c_pmu->formats = nv_c2c_pmu_formats; + + return 0; + } + + c2c_pmu->nr_inst = (c2c_pmu->c2c_type == C2C_TYPE_NVLINK) ? + C2C_NR_INST_NVLINK : C2C_NR_INST_NVCLINK; + + if (device_property_read_u32(dev, "cpu_en_mask", &cpu_en)) + dev_dbg(dev, "no cpu_en_mask property\n"); + + if (cpu_en) { + c2c_pmu->peer_type = C2C_PEER_TYPE_CPU; + + /* Fill peer_insts bitmap with instances connected to peer CPU. */ + bitmap_from_arr32(c2c_pmu->peer_insts[0], &cpu_en, + c2c_pmu->nr_inst); + + c2c_pmu->nr_peer = 1; + c2c_pmu->formats = nv_c2c_pmu_formats; + } else { + u32 i; + u32 gpu_en = 0; + const char *props[C2C_NR_PEER_MAX] = { + "gpu0_en_mask", "gpu1_en_mask" + }; + + for (i = 0; i < C2C_NR_PEER_MAX; i++) { + if (device_property_read_u32(dev, props[i], &gpu_en)) + dev_dbg(dev, "no %s property\n", props[i]); + + if (gpu_en) { + /* Fill peer_insts bitmap with instances connected to peer GPU. */ + bitmap_from_arr32(c2c_pmu->peer_insts[i], &gpu_en, + c2c_pmu->nr_inst); + + c2c_pmu->nr_peer++; + } + } + + if (c2c_pmu->nr_peer == 0) { + dev_err(dev, "No GPU is enabled\n"); + return -EINVAL; + } + + c2c_pmu->peer_type = C2C_PEER_TYPE_GPU; + c2c_pmu->formats = nv_c2c_nvlink_pmu_formats; + } + + c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; + + return 0; +} + +static void *nv_c2c_pmu_init_pmu(struct platform_device *pdev) +{ + int ret; + struct nv_c2c_pmu *c2c_pmu; + struct acpi_device *acpi_dev; + struct device *dev = &pdev->dev; + + acpi_dev = ACPI_COMPANION(dev); + if (!acpi_dev) + return ERR_PTR(-ENODEV); + + c2c_pmu = devm_kzalloc(dev, sizeof(*c2c_pmu), GFP_KERNEL); + if (!c2c_pmu) + return ERR_PTR(-ENOMEM); + + c2c_pmu->dev = dev; + c2c_pmu->acpi_dev = acpi_dev; + c2c_pmu->c2c_type = (unsigned int)(unsigned long)device_get_match_data(dev); + platform_set_drvdata(pdev, c2c_pmu); + + ret = nv_c2c_pmu_init_socket(c2c_pmu); + if (ret) + goto done; + + ret = nv_c2c_pmu_init_id(c2c_pmu); + if (ret) + goto done; + + ret = nv_c2c_pmu_init_filter(c2c_pmu); + if (ret) + goto done; + +done: + if (ret) + return ERR_PTR(ret); + + return c2c_pmu; +} + +static int nv_c2c_pmu_init_mmio(struct nv_c2c_pmu *c2c_pmu) +{ + int i; + struct device *dev = c2c_pmu->dev; + struct platform_device *pdev = to_platform_device(dev); + + /* Map the address of all the instances. */ + for (i = 0; i < c2c_pmu->nr_inst; i++) { + c2c_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); + if (IS_ERR(c2c_pmu->base[i])) { + dev_err(dev, "Failed map address for instance %d\n", i); + return PTR_ERR(c2c_pmu->base[i]); + } + } + + /* Map broadcast address. */ + c2c_pmu->base_broadcast = devm_platform_ioremap_resource(pdev, + c2c_pmu->nr_inst); + if (IS_ERR(c2c_pmu->base_broadcast)) { + dev_err(dev, "Failed map broadcast address\n"); + return PTR_ERR(c2c_pmu->base_broadcast); + } + + return 0; +} + +static int nv_c2c_pmu_register_pmu(struct nv_c2c_pmu *c2c_pmu) +{ + int ret; + + ret = cpuhp_state_add_instance(nv_c2c_pmu_cpuhp_state, + &c2c_pmu->cpuhp_node); + if (ret) { + dev_err(c2c_pmu->dev, "Error %d registering hotplug\n", ret); + return ret; + } + + c2c_pmu->pmu = (struct pmu) { + .parent = c2c_pmu->dev, + .task_ctx_nr = perf_invalid_context, + .pmu_enable = nv_c2c_pmu_enable, + .pmu_disable = nv_c2c_pmu_disable, + .event_init = nv_c2c_pmu_event_init, + .add = nv_c2c_pmu_add, + .del = nv_c2c_pmu_del, + .start = nv_c2c_pmu_start, + .stop = nv_c2c_pmu_stop, + .read = nv_c2c_pmu_read, + .attr_groups = c2c_pmu->attr_groups, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | + PERF_PMU_CAP_NO_INTERRUPT, + }; + + ret = perf_pmu_register(&c2c_pmu->pmu, c2c_pmu->name, -1); + if (ret) { + dev_err(c2c_pmu->dev, "Failed to register C2C PMU: %d\n", ret); + cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, + &c2c_pmu->cpuhp_node); + return ret; + } + + return 0; +} + +static int nv_c2c_pmu_probe(struct platform_device *pdev) +{ + int ret; + struct nv_c2c_pmu *c2c_pmu; + + c2c_pmu = nv_c2c_pmu_init_pmu(pdev); + if (IS_ERR(c2c_pmu)) + return PTR_ERR(c2c_pmu); + + ret = nv_c2c_pmu_init_mmio(c2c_pmu); + if (ret) + return ret; + + ret = nv_c2c_pmu_get_cpus(c2c_pmu); + if (ret) + return ret; + + ret = nv_c2c_pmu_alloc_attr_groups(c2c_pmu); + if (ret) + return ret; + + ret = nv_c2c_pmu_register_pmu(c2c_pmu); + if (ret) + return ret; + + dev_dbg(c2c_pmu->dev, "Registered %s PMU\n", c2c_pmu->name); + + return 0; +} + +static void nv_c2c_pmu_device_remove(struct platform_device *pdev) +{ + struct nv_c2c_pmu *c2c_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&c2c_pmu->pmu); + cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, &c2c_pmu->cpuhp_node); +} + +static const struct acpi_device_id nv_c2c_pmu_acpi_match[] = { + { "NVDA2023", (kernel_ulong_t)C2C_TYPE_NVLINK }, + { "NVDA2022", (kernel_ulong_t)C2C_TYPE_NVCLINK }, + { "NVDA2020", (kernel_ulong_t)C2C_TYPE_NVDLINK }, + { } +}; +MODULE_DEVICE_TABLE(acpi, nv_c2c_pmu_acpi_match); + +static struct platform_driver nv_c2c_pmu_driver = { + .driver = { + .name = "nvidia-t410-c2c-pmu", + .acpi_match_table = ACPI_PTR(nv_c2c_pmu_acpi_match), + .suppress_bind_attrs = true, + }, + .probe = nv_c2c_pmu_probe, + .remove = nv_c2c_pmu_device_remove, +}; + +static int __init nv_c2c_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/nvidia/c2c:online", + nv_c2c_pmu_online_cpu, + nv_c2c_pmu_cpu_teardown); + if (ret < 0) + return ret; + + nv_c2c_pmu_cpuhp_state = ret; + return platform_driver_register(&nv_c2c_pmu_driver); +} + +static void __exit nv_c2c_pmu_exit(void) +{ + platform_driver_unregister(&nv_c2c_pmu_driver); + cpuhp_remove_multi_state(nv_c2c_pmu_cpuhp_state); +} + +module_init(nv_c2c_pmu_init); +module_exit(nv_c2c_pmu_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("NVIDIA Tegra410 C2C PMU driver"); +MODULE_AUTHOR("Besar Wicaksono "); From 67c136cacb28dba27bc492545cbf9ba060cc5e17 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 26 Jan 2026 18:11:55 +0000 Subject: [PATCH 119/464] NVIDIA: VR: SAUCE: arm64: defconfig: Enable NVIDIA TEGRA410 PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Enable driver for NVIDIA TEGRA410 CMEM Latency and C2C PMU device. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260126181155.2776097-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit 6ff891383036db92d47a1a76598564c6ee0e348d noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- arch/arm64/configs/defconfig | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 269e811132f9b..b99d72bd959a8 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1801,6 +1801,8 @@ CONFIG_APPLE_M1_CPU_PMU=y CONFIG_HISI_PMU=y CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU=m CONFIG_NVIDIA_CORESIGHT_PMU_ARCH_SYSTEM_PMU=m +CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU=m +CONFIG_NVIDIA_TEGRA410_C2C_PMU=m CONFIG_MESON_DDR_PMU=m CONFIG_NVMEM_LAYOUT_SL28_VPD=m CONFIG_NVMEM_APPLE_EFUSES=m From 2a12c77919baf2c26c2c96f89c34013d5b5ceb5e Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 27 Jan 2026 22:59:09 +0000 Subject: [PATCH 120/464] NVIDIA: VR: SAUCE: perf vendor events arm64: Add Tegra410 Olympus PMU events BugLink: https://bugs.launchpad.net/bugs/2139315 Add JSON files for NVIDIA Tegra410 Olympus core PMU events. Also updated the common-and-microarch.json. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260127225909.3296202-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit f1f980c9661e73f839331d381cd046188aa02430 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- .../arch/arm64/common-and-microarch.json | 90 +++ tools/perf/pmu-events/arch/arm64/mapfile.csv | 1 + .../arch/arm64/nvidia/t410/branch.json | 45 ++ .../arch/arm64/nvidia/t410/brbe.json | 6 + .../arch/arm64/nvidia/t410/bus.json | 48 ++ .../arch/arm64/nvidia/t410/exception.json | 62 ++ .../arch/arm64/nvidia/t410/fp_operation.json | 78 ++ .../arch/arm64/nvidia/t410/general.json | 15 + .../arch/arm64/nvidia/t410/l1d_cache.json | 122 +++ .../arch/arm64/nvidia/t410/l1i_cache.json | 114 +++ .../arch/arm64/nvidia/t410/l2d_cache.json | 134 ++++ .../arch/arm64/nvidia/t410/ll_cache.json | 107 +++ .../arch/arm64/nvidia/t410/memory.json | 46 ++ .../arch/arm64/nvidia/t410/metrics.json | 722 ++++++++++++++++++ .../arch/arm64/nvidia/t410/misc.json | 646 ++++++++++++++++ .../arch/arm64/nvidia/t410/retired.json | 94 +++ .../arch/arm64/nvidia/t410/spe.json | 42 + .../arm64/nvidia/t410/spec_operation.json | 230 ++++++ .../arch/arm64/nvidia/t410/stall.json | 145 ++++ .../arch/arm64/nvidia/t410/tlb.json | 158 ++++ 20 files changed, 2905 insertions(+) create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json diff --git a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json index 468cb085d8796..6af15776ff170 100644 --- a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json +++ b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json @@ -179,6 +179,11 @@ "EventName": "BUS_CYCLES", "BriefDescription": "Bus cycle" }, + { + "EventCode": "0x001E", + "EventName": "CHAIN", + "BriefDescription": "Chain a pair of event counters." + }, { "PublicDescription": "Level 1 data cache allocation without refill", "EventCode": "0x1F", @@ -1512,11 +1517,26 @@ "EventName": "L2D_CACHE_REFILL_PRFM", "BriefDescription": "Level 2 data cache refill, software preload" }, + { + "EventCode": "0x8150", + "EventName": "L3D_CACHE_RW", + "BriefDescription": "Level 3 data cache demand access." + }, + { + "EventCode": "0x8151", + "EventName": "L3D_CACHE_PRFM", + "BriefDescription": "Level 3 data cache software prefetch" + }, { "EventCode": "0x8152", "EventName": "L3D_CACHE_MISS", "BriefDescription": "Level 3 data cache demand access miss" }, + { + "EventCode": "0x8153", + "EventName": "L3D_CACHE_REFILL_PRFM", + "BriefDescription": "Level 3 data cache refill, software prefetch." + }, { "EventCode": "0x8154", "EventName": "L1D_CACHE_HWPRF", @@ -1527,6 +1547,11 @@ "EventName": "L2D_CACHE_HWPRF", "BriefDescription": "Level 2 data cache hardware prefetch." }, + { + "EventCode": "0x8156", + "EventName": "L3D_CACHE_HWPRF", + "BriefDescription": "Level 3 data cache hardware prefetch." + }, { "EventCode": "0x8158", "EventName": "STALL_FRONTEND_MEMBOUND", @@ -1682,6 +1707,11 @@ "EventName": "L2D_CACHE_REFILL_HWPRF", "BriefDescription": "Level 2 data cache refill, hardware prefetch." }, + { + "EventCode": "0x81BE", + "EventName": "L3D_CACHE_REFILL_HWPRF", + "BriefDescription": "Level 3 data cache refill, hardware prefetch." + }, { "EventCode": "0x81C0", "EventName": "L1I_CACHE_HIT_RD", @@ -1712,11 +1742,31 @@ "EventName": "L1I_CACHE_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by software preload" }, + { + "EventCode": "0x81DC", + "EventName": "L1D_CACHE_HIT_RW_FPRFM", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by software prefetch." + }, { "EventCode": "0x81E0", "EventName": "L1I_CACHE_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by hardware prefetcher" }, + { + "EventCode": "0x81EC", + "EventName": "L1D_CACHE_HIT_RW_FHWPRF", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by hardware prefetcher." + }, + { + "EventCode": "0x81F0", + "EventName": "L1I_CACHE_HIT_RD_FPRF", + "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by prefetch." + }, + { + "EventCode": "0x81FC", + "EventName": "L1D_CACHE_HIT_RW_FPRF", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by prefetch." + }, { "EventCode": "0x8200", "EventName": "L1I_CACHE_HIT", @@ -1767,11 +1817,26 @@ "EventName": "L1I_LFB_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by software preload" }, + { + "EventCode": "0x825C", + "EventName": "L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by software prefetch." + }, { "EventCode": "0x8260", "EventName": "L1I_LFB_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by hardware prefetcher" }, + { + "EventCode": "0x826C", + "EventName": "L1D_LFB_HIT_RW_FHWPRF", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by hardware prefetcher." + }, + { + "EventCode": "0x827C", + "EventName": "L1D_LFB_HIT_RW_FPRF", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by prefetch." + }, { "EventCode": "0x8280", "EventName": "L1I_CACHE_PRF", @@ -1807,6 +1872,11 @@ "EventName": "LL_CACHE_REFILL", "BriefDescription": "Last level cache refill" }, + { + "EventCode": "0x828E", + "EventName": "L3D_CACHE_REFILL_PRF", + "BriefDescription": "Level 3 data cache refill, prefetch." + }, { "EventCode": "0x8320", "EventName": "L1D_CACHE_REFILL_PERCYC", @@ -1872,6 +1942,16 @@ "EventName": "FP_FP8_MIN_SPEC", "BriefDescription": "Floating-point operation speculatively_executed, smallest type is 8-bit floating-point." }, + { + "EventCode": "0x8480", + "EventName": "FP_SP_FIXED_MIN_OPS_SPEC", + "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." + }, + { + "EventCode": "0x8482", + "EventName": "FP_HP_FIXED_MIN_OPS_SPEC", + "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." + }, { "EventCode": "0x8483", "EventName": "FP_BF16_FIXED_MIN_OPS_SPEC", @@ -1882,6 +1962,16 @@ "EventName": "FP_FP8_FIXED_MIN_OPS_SPEC", "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is 8-bit floating-point." }, + { + "EventCode": "0x8488", + "EventName": "FP_SP_SCALE_MIN_OPS_SPEC", + "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." + }, + { + "EventCode": "0x848A", + "EventName": "FP_HP_SCALE_MIN_OPS_SPEC", + "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." + }, { "EventCode": "0x848B", "EventName": "FP_BF16_SCALE_MIN_OPS_SPEC", diff --git a/tools/perf/pmu-events/arch/arm64/mapfile.csv b/tools/perf/pmu-events/arch/arm64/mapfile.csv index bb3fa8a33496a..7f0eaa7020485 100644 --- a/tools/perf/pmu-events/arch/arm64/mapfile.csv +++ b/tools/perf/pmu-events/arch/arm64/mapfile.csv @@ -46,3 +46,4 @@ 0x00000000500f0000,v1,ampere/emag,core 0x00000000c00fac30,v1,ampere/ampereone,core 0x00000000c00fac40,v1,ampere/ampereonex,core +0x000000004e0f0100,v1,nvidia/t410,core diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json new file mode 100644 index 0000000000000..532bc59dc573e --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json @@ -0,0 +1,45 @@ +[ + { + "ArchStdEvent": "BR_MIS_PRED", + "PublicDescription": "The Event counts Branches which are speculatively executed and mis-predicted." + }, + { + "ArchStdEvent": "BR_PRED", + "PublicDescription": "The Event counts all speculatively executed Branches." + }, + { + "EventCode": "0x017e", + "EventName": "BR_PRED_BTB_CTX_UPDATE", + "PublicDescription": "Branch context table update." + }, + { + "EventCode": "0x0188", + "EventName": "BR_MIS_PRED_DIR_RESOLVED", + "PublicDescription": "Number of Branch misprediction due to direction misprediction." + }, + { + "EventCode": "0x0189", + "EventName": "BR_MIS_PRED_DIR_UNCOND_RESOLVED", + "PublicDescription": "Number of Branch misprediction due to direction misprediction for unconditional Branches." + }, + { + "EventCode": "0x018a", + "EventName": "BR_MIS_PRED_DIR_UNCOND_DIRECT_RESOLVED", + "PublicDescription": "Number of Branch misprediction due to direction misprediction for unconditional direct Branches." + }, + { + "EventCode": "0x018b", + "EventName": "BR_PRED_MULTI_RESOLVED", + "PublicDescription": "Number of resolved branch which made prediction by polymorphic indirect predictor." + }, + { + "EventCode": "0x018c", + "EventName": "BR_MIS_PRED_MULTI_RESOLVED", + "PublicDescription": "Number of branch misprediction which made prediction by polymorphic indirect predictor." + }, + { + "EventCode": "0x01e4", + "EventName": "BR_RGN_RECLAIM", + "PublicDescription": "The Event counts the Indirect predictor entries flushed by region reclamation." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json new file mode 100644 index 0000000000000..7c43a01c9707d --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json @@ -0,0 +1,6 @@ +[ + { + "ArchStdEvent": "BRB_FILTRATE", + "PublicDescription": "The Event counts each valid Branch record captured in the Branch record buffer. Branch records that are not captured because they are removed by filtering are not counted." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json new file mode 100644 index 0000000000000..c4cee0be1242e --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json @@ -0,0 +1,48 @@ +[ + { + "ArchStdEvent": "BUS_ACCESS", + "PublicDescription": "The Event counts the number of Data-beat access between the CPU and the external bus. This count includes access due to Read, Write, and Snoop. Each beat of Data is counted individually." + }, + { + "ArchStdEvent": "BUS_CYCLES", + "PublicDescription": "The Event counts bus cycles in the CPU. Bus cycles represent a clock cycle in which a transaction could be sent or received on the interface from the CPU to the external bus. Since that interface is driven at the same clock speed as the CPU, this Event increments at the rate of CPU clock. Regardless of the WFE/WFI state of the PE, this Event increment on each processor clock." + }, + { + "ArchStdEvent": "BUS_ACCESS_RD", + "PublicDescription": "The Event counts memory Read transactions seen on the external bus. Each beat of Data is counted individually." + }, + { + "ArchStdEvent": "BUS_ACCESS_WR", + "PublicDescription": "The Event counts memory Write transactions seen on the external bus. Each beat of Data is counted individually." + }, + { + "EventCode": "0x0154", + "EventName": "BUS_REQUEST_REQ", + "PublicDescription": "Bus request, request." + }, + { + "EventCode": "0x0155", + "EventName": "BUS_REQUEST_RETRY", + "PublicDescription": "Bus request, retry." + }, + { + "EventCode": "0x0198", + "EventName": "L2_CHI_CBUSY0", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 0." + }, + { + "EventCode": "0x0199", + "EventName": "L2_CHI_CBUSY1", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 1." + }, + { + "EventCode": "0x019a", + "EventName": "L2_CHI_CBUSY2", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 2." + }, + { + "EventCode": "0x019b", + "EventName": "L2_CHI_CBUSY3", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 3." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json new file mode 100644 index 0000000000000..2f31fb2e67a25 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json @@ -0,0 +1,62 @@ +[ + { + "ArchStdEvent": "EXC_TAKEN", + "PublicDescription": "The Event counts any taken architecturally visible exceptions such as IRQ, FIQ, SError, and other synchronous exceptions. Exceptions are counted whether or not they are taken locally." + }, + { + "ArchStdEvent": "EXC_RETURN", + "PublicDescription": "The Event counts any architecturally executed exception return Instructions. For example: AArch64: ERET." + }, + { + "ArchStdEvent": "EXC_UNDEF", + "PublicDescription": "The Event counts the number of synchronous exceptions which are taken locally that are due to attempting to execute an Instruction that is UNDEFINED. Attempting to execute Instruction bit patterns that have not been allocated. Attempting to execute Instructions when they are disabled. Attempting to execute Instructions at an inappropriate Exception level. Attempting to execute an Instruction when the value of PSTATE.IL is 1." + }, + { + "ArchStdEvent": "EXC_SVC", + "PublicDescription": "The Event counts SVC exceptions taken locally." + }, + { + "ArchStdEvent": "EXC_PABORT", + "PublicDescription": "The Event counts synchronous exceptions that are taken locally and caused by Instruction Aborts." + }, + { + "ArchStdEvent": "EXC_DABORT", + "PublicDescription": "The Event counts exceptions that are taken locally and are caused by Data aborts or SErrors. Conditions that could cause those exceptions are attempting to read or write memory where the MMU generates a fault, attempting to read or write memory with a misaligned address, Interrupts from the nSEI inputs and internally generated SErrors." + }, + { + "ArchStdEvent": "EXC_IRQ", + "PublicDescription": "The Event counts IRQ exceptions including the virtual IRQs that are taken locally." + }, + { + "ArchStdEvent": "EXC_FIQ", + "PublicDescription": "The Event counts FIQ exceptions including the virtual FIQs that are taken locally." + }, + { + "ArchStdEvent": "EXC_SMC", + "PublicDescription": "The Event counts SMC exceptions take to EL3." + }, + { + "ArchStdEvent": "EXC_HVC", + "PublicDescription": "The Event counts HVC exceptions taken to EL2." + }, + { + "ArchStdEvent": "EXC_TRAP_PABORT", + "PublicDescription": "The Event counts exceptions which are traps not taken locally and are caused by Instruction Aborts. For example, attempting to execute an Instruction with a misaligned PC." + }, + { + "ArchStdEvent": "EXC_TRAP_DABORT", + "PublicDescription": "The Event counts exceptions which are traps not taken locally and are caused by Data Aborts or SError Interrupts. Conditions that could cause those exceptions are: * Attempting to read or write memory where the MMU generates a fault, * Attempting to read or write memory with a misaligned address, * Interrupts from the SEI input. * internally generated SErrors." + }, + { + "ArchStdEvent": "EXC_TRAP_OTHER", + "PublicDescription": "The Event counts the number of synchronous trap exceptions which are not taken locally and are not SVC, SMC, HVC, Data aborts, Instruction Aborts, or Interrupts." + }, + { + "ArchStdEvent": "EXC_TRAP_IRQ", + "PublicDescription": "The Event counts IRQ exceptions including the virtual IRQs that are not taken locally." + }, + { + "ArchStdEvent": "EXC_TRAP_FIQ", + "PublicDescription": "The Event counts FIQs which are not taken locally but taken from EL0, EL1, or EL2 to EL3 (which would be the normal behavior for FIQs when not executing in EL3)." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json new file mode 100644 index 0000000000000..0b6d047207518 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json @@ -0,0 +1,78 @@ +[ + { + "ArchStdEvent": "FP_HP_SPEC", + "PublicDescription": "The Event counts speculatively executed half precision floating point operations." + }, + { + "ArchStdEvent": "FP_SP_SPEC", + "PublicDescription": "The Event counts speculatively executed single precision floating point operations." + }, + { + "ArchStdEvent": "FP_DP_SPEC", + "PublicDescription": "The Event counts speculatively executed double precision floating point operations." + }, + { + "ArchStdEvent": "FP_SCALE_OPS_SPEC", + "PublicDescription": "The Event counts speculatively executed scalable single precision floating point operations." + }, + { + "ArchStdEvent": "FP_FIXED_OPS_SPEC", + "PublicDescription": "The Event counts speculatively executed non-scalable single precision floating point operations." + }, + { + "ArchStdEvent": "FP_HP_SCALE_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes the counter to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_FIXED_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The even to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_SCALE_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_FIXED_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_DP_SCALE_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was double-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_DP_FIXED_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was double-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_BF16_FIXED_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was BFloat16 floating-point. Where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_BF16_SCALE_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was BFloat16 floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_FP8_FIXED_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was 8-bit floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_FP8_SCALE_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was 8-bit floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_FIXED_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_SCALE_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_FIXED_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_SCALE_MIN_OPS_SPEC", + "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json new file mode 100644 index 0000000000000..bbeb4a75d2618 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json @@ -0,0 +1,15 @@ +[ + { + "ArchStdEvent": "CPU_CYCLES", + "PublicDescription": "The Event counts CPU clock cycles, when the PE is not in WFE/WFI. The clock measured by this Event is defined as the physical clock driving the CPU logic." + }, + { + "ArchStdEvent": "CNT_CYCLES", + "PublicDescription": "The Event increments at a constant frequency equal to the rate of increment of the System Counter, CNTPCT_EL0. This Event does not increment when the PE is in WFE/WFI." + }, + { + "EventCode": "0x01e1", + "EventName": "CPU_SLOT", + "PublicDescription": "Entitled CPU slots. The Event counts the number of slots. When in ST mode, this Event shall increment by PMMIR_EL1.SLOTS quantities, and when in SMT partitioned resource mode (regardless of in WFI state or otherwise), this Event is incremented by PMMIR_EL1.SLOTS/2 quantities." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json new file mode 100644 index 0000000000000..7c9b2fc4b38cb --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json @@ -0,0 +1,122 @@ +[ + { + "ArchStdEvent": "L1D_CACHE_REFILL", + "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Load or Store operations or preload Instructions or hardware cache prefetching that missed in the L1 D-cache. This Event only counts one Event per cache line. Since the caches are Write-back only for this processor, there are no Write-through cache accesses." + }, + { + "ArchStdEvent": "L1D_CACHE", + "PublicDescription": "The Event counts L1 D-cache accesses from any Load/Store operations, software preload or hardware prefetch operation. Atomic operations that resolve in the CPUs caches (near atomic operations) counts as both a Write access and Read access. Each access to a cache line is counted including the multiple accesses caused by single Instructions such as LDM or STM. Each access to other L1 Data or unified memory structures, for example refill buffers, Write buffers, and Write-back buffers, are also counted. This Event counts the sum of L1D_CACHE_RD, L1D_CACHE_WR, L1D_CACHE_PRFM and L1D_CACHE_HWPRF." + }, + { + "ArchStdEvent": "L1D_CACHE_WB", + "PublicDescription": "The Event counts Write-backs of dirty Data from the L1 D-cache to the L2 cache. This occurs when either a dirty cache line is evicted from L1 D-cache and allocated in the L2 cache or dirty Data is written to the L2 and possibly to the next level of cache. This Event counts both victim cache line evictions and cache Write-backs from snoops or cache maintenance operations. The following cache operations are not counted: * Invalidations which do not result in Data being transferred out of the L1 (such as evictions of clean Data), * Full line Writes which write to L2 without writing L1, such as Write streaming mode. This Event is the sum of the L1D_CACHE_WB_CLEAN and L1D_CACHE_WB_VICTIM Events." + }, + { + "ArchStdEvent": "L1D_CACHE_LMISS_RD", + "PublicDescription": "The Event counts cache line refills into the L1 D-cache from any memory Read operations, that incurred additional latency. Counts same as L1D_CACHE_REFILL_RD on this CPU." + }, + { + "ArchStdEvent": "L1D_CACHE_RD", + "PublicDescription": "The Event counts L1 D-cache accesses from any Load operation. Atomic Load operations that resolve in the CPUs caches counts as both a Write access and Read access." + }, + { + "ArchStdEvent": "L1D_CACHE_WR", + "PublicDescription": "The Event counts L1 D-cache accesses generated by Store operations. This Event also counts accesses caused by a DC ZVA (D-cache zero, specified by virtual address) Instruction. Near atomic operations that resolve in the CPUs caches count as a Write access and Read access. This Event is a subset of the L1D_CACHE Event, except this Event only counts memory Write operations." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_RD", + "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Load Instructions where the memory Read operation misses in the L1 D-cache. This Event only counts one Event per cache line. This Event is a subset of the L1D_CACHE_REFILL Event, but only counts memory Read operations. This Event does not count reads caused by cache maintenance operations or preload Instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_WR", + "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Store Instructions where the memory Write operation misses in the L1 D-cache. This Event only counts one Event per cache line. This Event is a subset of the L1D_CACHE_REFILL Event, but only counts memory Write operations." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_INNER", + "PublicDescription": "The Event counts L1 D-cache refills (L1D_CACHE_REFILL) where the cache line Data came from caches inside the immediate Cluster of the Core (L2 cache)." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_OUTER", + "PublicDescription": "The Event counts L1 D-cache refills (L1D_CACHE_REFILL) for which the cache line Data came from outside the immediate Cluster of the Core, like an SLC in the system interconnect or DRAM or remote socket." + }, + { + "ArchStdEvent": "L1D_CACHE_WB_VICTIM", + "PublicDescription": "The Event counts dirty cache line evictions from the L1 D-cache caused by a new cache line allocation. This Event does not count evictions caused by cache maintenance operations. This Event is a subset of the L1D_CACHE_WB Event, but the Event only counts Write-backs that are a result of the line being allocated for an access made by the CPU." + }, + { + "ArchStdEvent": "L1D_CACHE_WB_CLEAN", + "PublicDescription": "The Event counts Write-backs from the L1 D-cache that are a result of a coherency operation made by another CPU. Event count includes cache maintenance operations. This Event is a subset of the L1D_CACHE_WB Event." + }, + { + "ArchStdEvent": "L1D_CACHE_INVAL", + "PublicDescription": "The Event counts each explicit invalidation of a cache line in the L1 D-cache caused by: * Cache Maintenance Operations (CMO) that operate by a virtual address. * Broadcast cache coherency operations from another CPU in the system. This Event does not count for the following conditions: * A cache refill invalidates a cache line. * A CMO which is executed on that CPU and invalidates a cache line specified by Set/Way. Note that CMOs that operate by Set/Way cannot be broadcast from one CPU to another." + }, + { + "ArchStdEvent": "L1D_CACHE_RW", + "PublicDescription": "The Event counts L1 Data demand cache accesses from any Load or Store operation. Near atomic operations that resolve in the CPUs caches counts as both a Write access and Read access. This Event is implemented as L1D_CACHE_RD + L1D_CACHE_WR" + }, + { + "ArchStdEvent": "L1D_CACHE_PRFM", + "PublicDescription": "The Event counts L1 D-cache accesses from software preload or prefetch Instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_PRFM", + "PublicDescription": "The Event counts L1 D-cache refills where the cache line access was generated by software preload or prefetch Instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_HWPRF", + "PublicDescription": "The Event counts L1 D-cache accesses from any Load/Store operations generated by the hardware prefetcher." + }, + { + "ArchStdEvent": "L1D_CACHE_MISS", + "PublicDescription": "The Event counts each demand access counted by L1D_CACHE_RW that misses in the L1 Data or unified cache, causing an access to outside of the L1 caches of this PE." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch access counted by L1D_CACHE_HWPRF that causes a refill of the L1 D-cache from outside of the L1 D-cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRFM", + "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched in response to a prefetch Instruction. That is, the L1D_CACHE_REFILL_PRFM Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FHWPRF", + "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched by a hardware prefetcher. That is, the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRF", + "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW where the cache line was fetched in response to a prefetch Instruction or by a hardware prefetcher. That is, the L1D_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FHWPRF", + "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF, where the cache line was fetched by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FPRFM", + "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF where the cache line was fetched in response to a prefetch Instruction. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and theL1D_CACHE_REFILL_PRFM Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FPRF", + "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW where the cache line was fetched in response to a prefetch Instruction or by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0204", + "EventName": "L1D_CACHE_REFILL_OUTER_LLC", + "PublicDescription": "The Event counts L1D_CACHE_REFILL from L3 D-cache." + }, + { + "EventCode": "0x0205", + "EventName": "L1D_CACHE_REFILL_OUTER_DRAM", + "PublicDescription": "The Event counts L1D_CACHE_REFILL from local memory." + }, + { + "EventCode": "0x0206", + "EventName": "L1D_CACHE_REFILL_OUTER_REMOTE", + "PublicDescription": "The Event counts L1D_CACHE_REFILL from a remote memory." + }, + { + "EventCode": "0x01f5", + "EventName": "L1D_CACHE_REFILL_RW", + "PublicDescription": "L1 D-cache refill, demand Read and Write. The Event counts demand Read and Write accesses that causes a refill of the L1 D-cache of this PE, from outside of this cache." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json new file mode 100644 index 0000000000000..a1faa284b80f6 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json @@ -0,0 +1,114 @@ +[ + { + "ArchStdEvent": "L1I_CACHE_REFILL", + "PublicDescription": "The Event counts cache line refills in the L1 I-cache caused by a missed Instruction fetch (Demand, hardware prefetch and software preload accesses). Instruction fetches may include accessing multiple Instructions, but the single cache line allocation is counted once." + }, + { + "ArchStdEvent": "L1I_CACHE", + "PublicDescription": "The Event counts Instruction fetches (Demand, hardware prefetch and software preload accesses) which access the L1 Instruction Cache. Instruction Cache accesses caused by cache maintenance operations are not counted." + }, + { + "ArchStdEvent": "L1I_CACHE_LMISS", + "PublicDescription": "The Event counts cache line refills into the L1 I-cache, that incurred additional latency. Counts the same as L1I_CACHE_REFILL in this CPU." + }, + { + "ArchStdEvent": "L1I_CACHE_RD", + "PublicDescription": "The Event counts demand Instruction fetches which access the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_PRFM", + "PublicDescription": "The Event counts Instruction fetches generated by software preload or prefetch Instructions which access the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HWPRF", + "PublicDescription": "The Event counts Instruction fetches which access the L1 I-cache generated by the hardware prefetcher." + }, + { + "ArchStdEvent": "L1I_CACHE_REFILL_PRFM", + "PublicDescription": "The Event counts cache line refills in the L1 I-cache caused by a missed Instruction fetch generated by software preload or prefetch Instructions. Instruction fetches may include accessing multiple Instructions, but the single cache line allocation is counted once." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_RD", + "PublicDescription": "The Event counts demand Instruction fetches that access the L1 I-cache and hit in the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT", + "PublicDescription": "The Event counts Instruction fetches that access the L1 I-cache (Demand, hardware prefetch and software preload accesses) and hit in the L1 I-cache. I-cache accesses caused by cache maintenance operations are not counted." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_PRFM", + "PublicDescription": "The Event counts Instruction fetches generated by software preload or prefetch Instructions that access the L1 I-cache and hit in the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_LFB_HIT_RD", + "PublicDescription": "The Event counts demand Instruction fetches that access the L1 I-cache and hit in a line that is in the process of being loaded into the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_REFILL_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch access counted by L1I_CACHE_HWPRF that causes a refill of the Level 1I-cache from outside of the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_RD_FPRF", + "PublicDescription": "The Event counts each demand fetch first hit counted by L1I_CACHE_HIT_RD where the cache line was fetched in response to a software preload or by a hardware prefetcher. That is, the L1I_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0174", + "EventName": "L1I_HWPRF_REQ_DROP", + "PublicDescription": "L1 I-cache hardware prefetch dropped." + }, + { + "EventCode": "0x01ea", + "EventName": "L1I_CFC_ENTRIES", + "PublicDescription": "The Event counts the CFC (Cache Fill Control) entries. The CFC is the fill buffer for I-cache." + }, + { + "EventCode": "0x0228", + "EventName": "L1I_CACHE_HIT_PRFM_FPRF", + "PublicDescription": "L1 I-cache software prefetch access first hit, fetched by hardware or software prefetch. The Event counts each software preload access first hit where the cache line was fetched in response to a hadware prefetcher or software preload Instruction. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0212", + "EventName": "L1I_CACHE_HIT_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch access that hits an L1 I-cache." + }, + { + "EventCode": "0x022a", + "EventName": "L1I_CACHE_HIT_HWPRF_FPRF", + "PublicDescription": "L1 I-cache hardware prefetch access first hit, fetched by hardware or software prefetch. The Event counts each hardware prefetch access first hit where the cache line was fetched in response to a hardware or prefetch Instruction. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0215", + "EventName": "L1I_LFB_HIT", + "PublicDescription": "L1 Line fill buffer hit. The Event counts each Demand or software preload or hardware prefetch induced Instruction fetch that hits an L1 I-cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0216", + "EventName": "L1I_LFB_HIT_PRFM", + "PublicDescription": "The Event counts each software prefetch access that hits a cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0219", + "EventName": "L1I_LFB_HIT_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch access that hits a cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0221", + "EventName": "L1I_PRFM_REQ", + "PublicDescription": "L1 I-cache software prefetch requests." + }, + { + "EventCode": "0x0222", + "EventName": "L1I_HWPRF_REQ", + "PublicDescription": "L1 I-cache hardware prefetch requests." + }, + { + "EventCode": "0x01e3", + "EventName": "L1I_CACHE_REFILL_RD", + "PublicDescription": "L1 I-cache refill, Read. The Event counts demand Instruction fetch that causes a refill of the L1 I-cache of this PE, from outside of this cache." + }, + { + "EventCode": "0x01ef", + "EventName": "L1I_CACHE_INVAL", + "PublicDescription": "L1 I-cache invalidate. The Event counts each explicit invalidation of a cache line in the L1 I-cache caused by: * Broadcast cache coherency operations from another CPU in the system. * Invalidation dues to capacity eviction in L2 D-cache. This Event does not count for the following conditions: * A cache refill invalidates a cache line. * A CMO which is executed on that CPU Core and invalidates a cache line specified by Set/Way. * Cache Maintenance Operations (CMO) that operate by a virtual address. Note that * CMOs that operate by Set/Way cannot be broadcast from one CPU Core to another. * The CMO is treated as No-op for the purposes of L1 I-cache line invalidation, as this Core implements fully coherent I-cache." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json new file mode 100644 index 0000000000000..ac3f8095a9979 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json @@ -0,0 +1,134 @@ +[ + { + "ArchStdEvent": "L2D_CACHE", + "PublicDescription": "The Event counts accesses to the L2 cache due to Data accesses. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This Event also counts Write back of dirty Data from L1 D-cache to the L2 cache. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is the sum of the L2D_CACHE_RD, L2D_CACHE_WR, L2D_CACHE_PRFM, and L2D_CACHE_HWPRF Events." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL", + "PublicDescription": "The Event counts cache line refills into the L2 cache. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented. This Event is the sum of L2D_CACHE_REFILL_RD, L2D_CACHE_REFILL_WR, L2D_CACHE_REFILL_HWPRF, and L2D_CACHE_REFILL_PRFM." + }, + { + "ArchStdEvent": "L2D_CACHE_WB", + "PublicDescription": "The Event counts Write-backs of Data from the L2 cache to outside the CPU. This includes snoops to the L2 (from other CPUs) which return Data even if the snoops cause an invalidation. L2 cache line invalidations which do not write Data outside the CPU and snoops which return Data from an L1 cache are not counted. Data would not be written outside the cache when invalidating a clean cache line. This Event is the sum of the L2D_CACHE_WB_VICTIM and L2D_CACHE_WB_CLEAN Events." + }, + { + "ArchStdEvent": "L2D_CACHE_RD", + "PublicDescription": "The Event counts L2 D-cache accesses due to memory Read operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is a subset of the L2D_CACHE Event, but this Event only counts memory Read operations." + }, + { + "ArchStdEvent": "L2D_CACHE_WR", + "PublicDescription": "The Event counts L2 cache accesses due to memory Write operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This Event is a subset of the L2D_CACHE Event, but this Event only counts memory Write operations." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_RD", + "PublicDescription": "The Event counts refills for memory accesses due to memory Read operation counted by L2D_CACHE_RD. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented. This Event is a subset of the L2D_CACHE_REFILL Event. This Event does not count L2 refills caused by stashes into L2. This count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_WR", + "PublicDescription": "The Event counts refills for memory accesses due to memory Write operation counted by L2D_CACHE_WR. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." + }, + { + "ArchStdEvent": "L2D_CACHE_WB_VICTIM", + "PublicDescription": "The Event counts evictions from the L2 cache because of a line being allocated into the L2 cache. This Event is a subset of the L2D_CACHE_WB Event." + }, + { + "ArchStdEvent": "L2D_CACHE_WB_CLEAN", + "PublicDescription": "The Event counts Write-backs from the L2 cache that are a result of any of the following: * Cache maintenance operations, * Snoop responses or, * Direct cache transfers to another CPU due to a forwarding snoop request. This Event is a subset of the L2D_CACHE_WB Event." + }, + { + "ArchStdEvent": "L2D_CACHE_INVAL", + "PublicDescription": "The Event counts each explicit invalidation of a cache line in the L2 cache by cache maintenance operations that operate by a virtual address, or by external coherency operations. This Event does not count if either: * A cache refill invalidates a cache line, or * A cache Maintenance Operation (CMO), which invalidates a cache line specified by Set/Way, is executed on that CPU. CMOs that operate by Set/Way cannot be broadcast from one CPU to another." + }, + { + "ArchStdEvent": "L2D_CACHE_LMISS_RD", + "PublicDescription": "The Event counts cache line refills into the L2 unified cache from any memory Read operations that incurred additional latency. Counts the same as L2D_CACHE_REFILL_RD in this CPU" + }, + { + "ArchStdEvent": "L2D_CACHE_RW", + "PublicDescription": "The Event counts L2 cache demand accesses from any Load/Store operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is the sum of the L2D_CACHE_RD and L2D_CACHE_WR Events." + }, + { + "ArchStdEvent": "L2D_CACHE_PRFM", + "PublicDescription": "The Event counts L2 D-cache accesses generated by software preload or prefetch Instructions with target = L1/L2/L3 cache. Note that a software preload or prefetch Instructions with (target = L1/L2/L3) that hits in L1D will not result in an L2 D-cache access. Therefore, such a software preload or prefetch Instructions will not be counted by this Event." + }, + { + "ArchStdEvent": "L2D_CACHE_MISS", + "PublicDescription": "The Event counts cache line misses in the L2 cache. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. Counts same as L2D_CACHE_REFILL_RD in this CPU" + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_PRFM", + "PublicDescription": "The Event counts refills due to accesses generated as a result of software preload or prefetch Instructions as counted by L2D_CACHE_PRFM. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch access counted by L2D_CACHE_HWPRF that causes a refill of the L2 cache, or any L1 Data, or Instruction of this PE, from outside of those caches. This does not include prefetch requests pending waiting for a refill in LFB and a new demand request to the same cache line hitting the LFB entry. All such refills are counted as L2D_LFB_HIT_RWL1PRF_FHWPRF." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_PRF", + "PublicDescription": "The Event counts each access to L2 Cache due to a prefetch Instruction, or hardware prefetch that causes a refill of the L2 or any Level 1, from outside of those caches." + }, + { + "ArchStdEvent": "L2D_CACHE_HWPRF", + "PublicDescription": "The Event counts the L2 D-cache access caused by L1 or L2 hardware prefetcher." + }, + { + "EventCode": "0x0108", + "EventName": "L2D_CACHE_IF_REFILL", + "PublicDescription": "L2 D-cache refill, Instruction fetch. The Event counts demand Instruction fetch that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0109", + "EventName": "L2D_CACHE_TBW_REFILL", + "PublicDescription": "L2 D-cache refill, Page table walk. The Event counts demand translation table walk that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x010a", + "EventName": "L2D_CACHE_PF_REFILL", + "PublicDescription": "L2 D-cache refill, prefetch. The Event counts L1 or L2 hardware or software prefetch accesses that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0201", + "EventName": "L2D_CACHE_BACKSNOOP_L1D_VIRT_ALIASING", + "PublicDescription": "The Event counts when the L2 D-cache sends an invalidating back-snoop to the L1 D for an access initiated by the L1 D, where the corresponding line is already present in the L1 D-cache. The L2 D-cache line tags the PE that refilled the line. It also retains specific bits of the VA to identify virtually aliased addresses. The L1 D request requiring a back-snoop can originate either from the same PE that refilled the L2 D line or from a different PE. In either case, this Event only counts those back snoop where the requested VA mismatch the VA stored in the L2 D tag. This Event is counted only by PE that initiated the original request necessitating a back-snoop. Note : The L1 D is VIPT, it identifies this access as a miss. Conversely, as L2 is PIPT, it identifies this as a hit. L2 D utilizes the back-snoop mechanism to refill L1 D with the snooped Data." + }, + { + "EventCode": "0x0179", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FHWPRF", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by hardware prefetch.. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x020c", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRFM", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software prefetch. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x020e", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRF", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software or hardware prefetch. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch or software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x010b", + "EventName": "L2D_LFB_HIT_RWL1PRF_FHWPRF", + "PublicDescription": "L2 line fill buffer demand Read, demand Write or L1 prefetch first hit, fetched by hardware prefetch. The Event counts each of the following access that hit the line-fill buffer when the same cache line is already being fetched due to an L2 hardware prefetcher. * Demand Read or Write * L1I-HWPRF * L1D-HWPRF * L1I PRFM * L1D PRFM These accesses hit a cache line that is currently being loaded into the L2 cache as a result of a hardware prefetcher to the same line. Consequently, this access does not initiate a new refill but waits for the completion of the previous refill. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x01b9", + "EventName": "L2D_CACHE_REFILL_L1PRF", + "PublicDescription": "L2 D-cache refill, L1 hardware or software prefetch. The Event counts each access counted by L2D_CACHE_L1PRF that causes a refill of the L2 cache or any L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x020a", + "EventName": "L2D_CACHE_REFILL_RWL1PRF", + "PublicDescription": "L2 D-cache refill, demand Read, demand Write or L1 hardware or software prefetch. The Event counts each access counted by L2D_CACHE_RWL1PRF that causes a refill of the L2 cache, or any L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x01b8", + "EventName": "L2D_CACHE_L1PRF", + "PublicDescription": "L2 D-cache access, L1 hardware or software prefetch. The Event counts L1 Hardware or software prefetch access to L2 D-cache." + }, + { + "EventCode": "0x0208", + "EventName": "L2D_CACHE_RWL1PRF", + "PublicDescription": "L2 D-cache access, demand Read, demand Write or L1 hardware or software prefetch. The Event counts each access to L2 D-cache due to the following: * Demand Read or Write. * L1 Hardware or software prefetch." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json new file mode 100644 index 0000000000000..661cbed4ee347 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json @@ -0,0 +1,107 @@ +[ + { + "ArchStdEvent": "LL_CACHE_RD", + "PublicDescription": "This is an alias to the Event L3D_CACHE_RD (0x00a0)." + }, + { + "ArchStdEvent": "LL_CACHE_MISS_RD", + "PublicDescription": "This is an alias to the Event L3D_CACHE_REFILL_RD (0x00a2)." + }, + { + "ArchStdEvent": "L3D_CACHE_ALLOCATE", + "PublicDescription": "The Event counts each memory Write operation that writes an entire line into the L3 Data without fetching Data from outside the L3 Data. These are allocations of cache lines in the L3 Data that are not refills counted by L3D_CACHE_REFILL. For example: A Write-back of an entire cache line from an L2 cache to the L3 D-cache. * A Write of an entire cache line from a coalescing Write buffer. * An operation such as DC ZVA. This counter does not count that writes an entire line to beyond level 3. Thus this counter does not count the streaming Writes to beyond L3 cache." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL", + "PublicDescription": "The Event counts each access counted by L3D_CACHE that causes a refill of the L3 Data,or any L1 Data, Instruction or L2 cache of this PE, from outside of those caches. This includes the refill due to hardware prefetch and software prefetch accesses. This Event is a sum of L3D_CACHE_MISS, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL_HWPRF Event. A refill includes any access that causes Data to be fetched from outside of the L1 to L3 caches, even if the Data is ultimately not allocated into the L3 D-cache." + }, + { + "ArchStdEvent": "L3D_CACHE", + "PublicDescription": "The Event counts each memory Read operation or memory Write operation that causes a cache access to the Level 3. This Event is a sum of the following Events: * L3D_CACHE_RD(0x00a0) * L3D_CACHE_ALLOCATE(0x0029) * L3D_CACHE_PRFM(0x8151) * L3D_CACHE_HWPRF(0x8156) * L2D_CACHE_WB(0x0018)" + }, + { + "ArchStdEvent": "L3D_CACHE_RD", + "PublicDescription": "The Event counts each Memory Read operation to L3 D-cache from Instruction Fetch, Load/Store, and MMU translation table accesses. This does not include HWPRF or PRFM accesses. This include L1 and L2 prefetcher accesses to L3 D-cache." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_RD", + "PublicDescription": "The Event counts each access counted by both L3D_CACHE_RD and L3D_CACHE_REFILL. That is, every refill of the L3 cache counted by L3D_CACHE_REFILL that is caused by a Memory Read operation. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_LMISS_RD", + "PublicDescription": "The Event counts each memory Read operation to the L3 cache counted by L3D_CACHE that incurs additional latency because it returns Data from outside of the L1 to L3 caches. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_RW", + "PublicDescription": "The Event counts each access counted by L3D_CACHE that is due to a demand memory Read operation or demand memory Write operation. This Event is a sum of L3D_CACHE_RD(0x00a0), L3D_CACHE_ALLOCATE(0x0029) and L2D_CACHE_WB(0x0018). Note that this counter does not count that writes an entire line to beyond level 3. Thus this counter does not count the streaming Writes to beyond L3 cache." + }, + { + "ArchStdEvent": "L3D_CACHE_PRFM", + "PublicDescription": "The Event counts each access counted by L3D_CACHE that is due to a prefetch Instruction. This includes L3 Data accesses due to the L1, L2, or L3 prefetch Instruction." + }, + { + "ArchStdEvent": "L3D_CACHE_MISS", + "PublicDescription": "The Event counts each demand Read access counted by L3D_CACHE_RD that misses in the L1 to L3 Data, causing an access to outside of the L3 cache. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_PRFM", + "PublicDescription": "The Event counts each access counted by L3D_CACHE_PRFM that causes a refill of the L3 cache, or any L1 or L2 Data, from outside of those caches." + }, + { + "ArchStdEvent": "L3D_CACHE_HWPRF", + "PublicDescription": "The Event counts each access to L3 cache that is due to a hardware prefetcher. This includes L3D accesses due to the Level-1 or Level-2 or Level-3 hardware prefetcher." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_HWPRF", + "PublicDescription": "The Event counts each hardware prefetch counted by L3D_CACHE_HWPRF that causes a refill of the L3 Data or unified cache, or any L1 or L2 Data, Instruction, or unified cache of this PE, from outside of those caches." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_PRF", + "PublicDescription": "The Event counts each access to L3 cache due to a prefetch Instruction, or hardware prefetch that causes a refill of the L3 Data, or any L1 or L2 Data, from outside of those caches." + }, + { + "EventCode": "0x01f7", + "EventName": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", + "PublicDescription": "L3 cache demand Read, demand Write , L1 prefetch L2 prefetch first hit, fetched by software or hardware prefetch. The Event counts each demand Read, demand Write , L1 hardware or software prefetch request and L2 hardware or software prefetch that hit an L3 D-cache line that was refilled into L3 D-cache in response to an L3 hardware prefetch or software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x01e9", + "EventName": "L3D_CACHE_REFILL_RWL1PRFL2PRF", + "PublicDescription": "L3 cache refill, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch. The Event counts each access counted by L3D_CACHE_RWL1PRFL2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x01e8", + "EventName": "L3D_CACHE_RWL1PRFL2PRF", + "PublicDescription": "L3 cache access, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch. The Event counts each access to L3 D-cache due to the following: * Demand Read or Write. * L1 Hardware or software prefetch. * L2 Hardware or software prefetch." + }, + { + "EventCode": "0x0225", + "EventName": "L3D_CACHE_REFILL_IF", + "PublicDescription": "L3 cache refill, Instruction fetch. The Event counts demand Instruction fetch that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0226", + "EventName": "L3D_CACHE_REFILL_MM", + "PublicDescription": "L3 cache refill, translation table walk access. The Event counts demand translation table access that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0227", + "EventName": "L3D_CACHE_REFILL_L1PRF", + "PublicDescription": "The Event counts each access counted by L3D_CACHE_L1PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x01f6", + "EventName": "L3D_CACHE_REFILL_L2PRF", + "PublicDescription": "The Event counts each access counted by L3D_CACHE_L2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x022c", + "EventName": "L3D_CACHE_L1PRF", + "PublicDescription": "The Event counts the L3 D-cache access due to L1 hardware prefetch of software prefetch request. The L1 hardware prefetch or software prefetch request that miss the L1I, L1D and L2 D-cache are counted by this counter" + }, + { + "EventCode": "0x022d", + "EventName": "L3D_CACHE_L2PRF", + "PublicDescription": "The Event counts the L3 D-cache access due to L2 hardware prefetch of software prefetch request. The L2 hardware prefetch or software prefetch request that miss the L2 D-cache are counted by this counter" + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json new file mode 100644 index 0000000000000..64fced85a9881 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json @@ -0,0 +1,46 @@ +[ + { + "ArchStdEvent": "MEM_ACCESS", + "PublicDescription": "The Event counts memory accesses issued by the CPU Load/Store unit, where those accesses are issued due to Load or Store operations. This Event counts memory accesses no matter whether the Data is received from any level of cache hierarchy or external memory. If memory accesses are broken up into smaller transactions than what were specified in the Load or Store Instructions, then the Event counts those smaller memory transactions. Memory accesses generated by the following Instructions or activity are not counted: Instruction fetches, cache maintenance Instructions, Translation table walks or prefetches, Memory prefetch operations. This Event counts the sum of the MEM_ACCESS_RD and MEM_ACCESS_WR Events." + }, + { + "ArchStdEvent": "MEMORY_ERROR", + "PublicDescription": "The Event counts any detected correctable or uncorrectable physical memory errors (ECC or parity) in protected CPUs RAMs. On the Core, this Event counts errors in the caches (including Data and tag rams). Any detected memory error (from either a speculative and abandoned access, or an architecturally executed access) is counted. Note that errors are only detected when the actual protected memory is accessed by an operation." + }, + { + "ArchStdEvent": "REMOTE_ACCESS", + "PublicDescription": "Counter counts each external bus Read access that causes an access to a remote device. That is, a socket that does not contain the PE." + }, + { + "ArchStdEvent": "MEM_ACCESS_RD", + "PublicDescription": "The Event counts memory accesses issued by the CPU due to Load operations. The Event counts any memory Load access, no matter whether the Data is received from any level of cache hierarchy or external memory. The Event also counts atomic Load operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the Event counts those smaller transactions. The following Instructions are not counted: 1) Instruction fetches, 2) Cache maintenance Instructions, 3) Translation table walks or prefetches, 4) Memory prefetch operations. This Event is a subset of the MEM_ACCESS Event but the Event only counts memory-Read operations." + }, + { + "ArchStdEvent": "MEM_ACCESS_WR", + "PublicDescription": "The Event counts memory accesses issued by the CPU due to Store operations. The Event counts any memory Store access, no matter whether the Data is located in any level of cache or external memory. The Event also counts atomic Load and Store operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the Event counts those smaller transactions." + }, + { + "ArchStdEvent": "LDST_ALIGN_LAT", + "PublicDescription": "The Event counts the number of memory Read and Write accesses in a cycle that incurred additional latency, due to the alignment of the address and the size of Data being accessed, which results in Store crossing a single cache line. This Event is implemented as the sum of LD_ALIGN_LAT and ST_ALIGN_LAT on this CPU." + }, + { + "ArchStdEvent": "LD_ALIGN_LAT", + "PublicDescription": "The Event counts the number of memory Read accesses in a cycle that incurred additional latency, due to the alignment of the address and size of Data being accessed, which results in Load crossing a single cache line." + }, + { + "ArchStdEvent": "ST_ALIGN_LAT", + "PublicDescription": "The Event counts the number of memory Write access in a cycle that incurred additional latency, due to the alignment of the address and size of Data being accessed incurred additional latency." + }, + { + "ArchStdEvent": "INST_FETCH_PERCYC", + "PublicDescription": "The Event counts number of Instruction fetches outstanding per cycle, which will provide an average latency of Instruction fetch." + }, + { + "ArchStdEvent": "MEM_ACCESS_RD_PERCYC", + "PublicDescription": "The Event counts the number of outstanding Loads or memory Read accesses per cycle." + }, + { + "ArchStdEvent": "INST_FETCH", + "PublicDescription": "The Event counts Instruction memory accesses that the PE makes." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json new file mode 100644 index 0000000000000..18c2fd58ee9ec --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json @@ -0,0 +1,722 @@ +[ + { + "MetricName": "backend_bound", + "MetricExpr": "100 * (STALL_SLOT_BACKEND / CPU_SLOT)", + "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the backend of the processor.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "backend_busy_bound", + "MetricExpr": "100 * (STALL_BACKEND_BUSY / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to issue queues being full to accept operations for execution.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_cache_l1d_bound", + "MetricExpr": "100 * (STALL_BACKEND_L1D / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L1 D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_cache_l2d_bound", + "MetricExpr": "100 * (STALL_BACKEND_MEM / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L2 D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_core_bound", + "MetricExpr": "100 * (STALL_BACKEND_CPUBOUND / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints not related to Instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_core_rename_bound", + "MetricExpr": "100 * (STALL_BACKEND_RENAME / STALL_BACKEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend as the rename unit registers are unavailable.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_bound", + "MetricExpr": "100 * (STALL_BACKEND_MEMBOUND / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints related to memory access latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_cache_bound", + "MetricExpr": "100 * ((STALL_BACKEND_L1D + STALL_BACKEND_MEM) / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory latency issues caused by D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_store_bound", + "MetricExpr": "100 * (STALL_BACKEND_ST / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory Write pending caused by Stores stalled in the pre-commit stage.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_tlb_bound", + "MetricExpr": "100 * (STALL_BACKEND_TLB / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by Data TLB misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_stalled_cycles", + "MetricExpr": "100 * (STALL_BACKEND / CPU_CYCLES)", + "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the backend unit of the processor.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Cycle_Accounting" + }, + { + "MetricName": "bad_speculation", + "MetricExpr": "100 - (frontend_bound + retiring + backend_bound)", + "BriefDescription": "This metric is the percentage of total slots that executed operations and didn't retire due to a pipeline flush. This indicates cycles that were utilized but inefficiently.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "bus_bandwidth", + "MetricExpr": "BUS_ACCESS * 32 / duration_time ", + "BriefDescription": "This metrics measures the bus-bandwidth of the Data transferred between this PE's L2 with unCore in the system", + "ScaleUnit": "1Bytes/sec" + }, + { + "MetricName": "barrier_percentage", + "MetricExpr": "100 * ((ISB_SPEC + DSB_SPEC + DMB_SPEC) / INST_SPEC)", + "BriefDescription": "This metric measures Instruction and Data barrier operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "branch_direct_ratio", + "MetricExpr": "BR_IMMED_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of direct branches retired to the total number of Branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "branch_indirect_ratio", + "MetricExpr": "BR_IND_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of indirect Branches retired, including function returns, to the total number of Branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "branch_misprediction_ratio", + "MetricExpr": "BR_MIS_PRED_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of Branches mispredicted to the total number of Branches architecturally executed. This gives an indication of the effectiveness of the Branch prediction unit.", + "ScaleUnit": "1per branch", + "MetricGroup": "Miss_Ratio;Branch_Effectiveness" + }, + { + "MetricName": "branch_mpki", + "MetricExpr": "1000 * (BR_MIS_PRED_RETIRED / INST_RETIRED)", + "BriefDescription": "This metric measures the number of Branch mispredictions per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;Branch_Effectiveness" + }, + { + "MetricName": "branch_percentage", + "MetricExpr": "100 * ((BR_IMMED_SPEC + BR_INDIRECT_SPEC) / INST_SPEC)", + "BriefDescription": "This metric measures Branch operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "branch_return_ratio", + "MetricExpr": "BR_RETURN_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of Branches retired that are function returns to the total number of Branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "cpu_cycles_fraction_in_st_mode", + "MetricExpr": "((CPU_SLOT/CPU_CYCLES) - 5) / 5", + "BriefDescription": "This metric counts fraction of the CPU cycles spent in ST mode during program execution.", + "ScaleUnit": "1fraction of cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "cpu_cycles_in_smt_mode", + "MetricExpr": "(1 - cpu_cycles_fraction_in_st_mode) * CPU_CYCLES", + "BriefDescription": "This metric counts CPU cycles in SMT mode during program execution.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "cpu_cycles_in_st_mode", + "MetricExpr": "cpu_cycles_fraction_in_st_mode * CPU_CYCLES", + "BriefDescription": "This metric counts CPU cycles in ST mode during program execution.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "crypto_percentage", + "MetricExpr": "100 * (CRYPTO_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures crypto operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "dtlb_mpki", + "MetricExpr": "1000 * (DTLB_WALK / INST_RETIRED)", + "BriefDescription": "This metric measures the number of Data TLB Walks per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;DTLB_Effectiveness" + }, + { + "MetricName": "dtlb_walk_average_latency", + "MetricExpr": "DTLB_WALK_PERCYC / DTLB_WALK", + "BriefDescription": "This metric measures the average latency of Data TLB walks in CPU cycles", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "dtlb_walk_ratio", + "MetricExpr": "DTLB_WALK / L1D_TLB", + "BriefDescription": "This metric measures the ratio of Data TLB Walks to the total number of Data TLB accesses. This gives an indication of the effectiveness of the Data TLB accesses.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" + }, + { + "MetricName": "fp_ops_per_cycle", + "MetricExpr": "(FP_SCALE_OPS_SPEC + FP_FIXED_OPS_SPEC) / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by any Instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "fp16_percentage", + "MetricExpr": "100 * (FP_HP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures half-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "fp32_percentage", + "MetricExpr": "100 * (FP_SP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures single-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "fp64_percentage", + "MetricExpr": "100 * (FP_DP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures double-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "frontend_bound", + "MetricExpr": "100 * (STALL_SLOT_FRONTEND_WITHOUT_MISPRED / CPU_SLOT)", + "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the frontend of the processor.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "frontend_cache_l1i_bound", + "MetricExpr": "100 * (STALL_FRONTEND_L1I / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L1 I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_cache_l2i_bound", + "MetricExpr": "100 * (STALL_FRONTEND_MEM / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L2 I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_bound", + "MetricExpr": "100 * (STALL_FRONTEND_CPUBOUND / STALL_FRONTEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints not related to Instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_flow_bound", + "MetricExpr": "100 * (STALL_FRONTEND_FLOW / STALL_FRONTEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the decode unit is awaiting input from the Branch prediction unit.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_flush_bound", + "MetricExpr": "100 * (STALL_FRONTEND_FLUSH / STALL_FRONTEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the processor is recovering from a pipeline flush caused by bad speculation or other machine resteers.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_bound", + "MetricExpr": "100 * (STALL_FRONTEND_MEMBOUND / STALL_FRONTEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints related to the Instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_cache_bound", + "MetricExpr": "100 * ((STALL_FRONTEND_L1I + STALL_FRONTEND_MEM) / STALL_FRONTEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to Instruction fetch latency issues caused by I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_tlb_bound", + "MetricExpr": "100 * (STALL_FRONTEND_TLB / STALL_FRONTEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to Instruction fetch latency issues caused by Instruction TLB misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_stalled_cycles", + "MetricExpr": "100 * (STALL_FRONTEND / CPU_CYCLES)", + "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the frontend unit of the processor.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Cycle_Accounting" + }, + { + "MetricName": "instruction_fetch_average_latency", + "MetricExpr": "INST_FETCH_PERCYC / INST_FETCH", + "BriefDescription": "This metric measures the average latency of Instruction fetches in CPU cycles", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "integer_dp_percentage", + "MetricExpr": "100 * (DP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalar integer operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "ipc", + "MetricExpr": "INST_RETIRED / CPU_CYCLES", + "BriefDescription": "This metric measures the number of Instructions retired per cycle.", + "ScaleUnit": "1per cycle", + "MetricGroup": "General" + }, + { + "MetricName": "itlb_mpki", + "MetricExpr": "1000 * (ITLB_WALK / INST_RETIRED)", + "BriefDescription": "This metric measures the number of Instruction TLB Walks per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness" + }, + { + "MetricName": "itlb_walk_average_latency", + "MetricExpr": "ITLB_WALK_PERCYC / ITLB_WALK", + "BriefDescription": "This metric measures the average latency of Instruction TLB walks in CPU cycles", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "itlb_walk_ratio", + "MetricExpr": "ITLB_WALK / L1I_TLB", + "BriefDescription": "This metric measures the ratio of Instruction TLB Walks to the total number of Instruction TLB accesses. This gives an indication of the effectiveness of the Instruction TLB accesses.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" + }, + { + "MetricName": "l1d_cache_miss_ratio", + "MetricExpr": "L1D_CACHE_REFILL / L1D_CACHE", + "BriefDescription": "This metric measures the ratio of L1 D-cache accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L1D_Cache_Effectiveness" + }, + { + "MetricName": "l1d_cache_mpki", + "MetricExpr": "1000 * (L1D_CACHE_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 D-cache accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L1D_Cache_Effectiveness" + }, + { + "MetricName": "l1d_cache_rw_miss_ratio", + "MetricExpr": "l1d_demand_misses / l1d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L1 D-cache Read accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache for demand Load or Store traffic.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_demand_accesses", + "MetricExpr": "L1D_CACHE_RW", + "BriefDescription": "This metric measures the count of L1 D-cache accesses incurred on Load or Store by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_demand_misses", + "MetricExpr": "L1D_CACHE_REFILL_RW", + "BriefDescription": "This metric measures the count of L1 D-cache misses incurred on a Load or Store by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_prf_accuracy", + "MetricExpr": "100 * (l1d_useful_prf / l1d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_prf_coverage", + "MetricExpr": "100 * (l1d_useful_prf / (l1d_demand_misses + l1d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_refilled_prf", + "MetricExpr": "L1D_CACHE_REFILL_HWPRF + L1D_CACHE_REFILL_PRFM + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 Data prefetcher (hardware prefetches or software preload) into L1 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_tlb_miss_ratio", + "MetricExpr": "L1D_TLB_REFILL / L1D_TLB", + "BriefDescription": "This metric measures the ratio of L1 Data TLB accesses missed to the total number of L1 Data TLB accesses. This gives an indication of the effectiveness of the L1 Data TLB.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" + }, + { + "MetricName": "l1d_tlb_mpki", + "MetricExpr": "1000 * (L1D_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 Data TLB accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;DTLB_Effectiveness" + }, + { + "MetricName": "l1d_useful_prf", + "MetricExpr": "L1D_CACHE_HIT_RW_FPRF + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 Data prefetcher (hardware prefetches or software preload) into L1 D-cache which are further used by Load or Store from the Instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_cache_miss_ratio", + "MetricExpr": "L1I_CACHE_REFILL / L1I_CACHE", + "BriefDescription": "This metric measures the ratio of L1 I-cache accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L1I_Cache_Effectiveness" + }, + { + "MetricName": "l1i_cache_mpki", + "MetricExpr": "1000 * (L1I_CACHE_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 I-cache accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L1I_Cache_Effectiveness" + }, + { + "MetricName": "l1i_cache_rd_miss_ratio", + "MetricExpr": "l1i_demand_misses / l1i_demand_accesses", + "BriefDescription": "This metric measures the ratio of L1 I-cache Read accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache for demand Instruction fetch traffic. Note that cache accesses in this cache is demand Instruction fetch.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_demand_accesses", + "MetricExpr": "L1I_CACHE_RD", + "BriefDescription": "This metric measures the count of L1 I-cache accesses caused by an Instruction fetch by the Instructions stream of the program", + "ScaleUnit": "1Count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_demand_misses", + "MetricExpr": "L1I_CACHE_REFILL_RD", + "BriefDescription": "This metric measures the count of L1 I-cache misses caused by an Instruction fetch by the Instructions stream of the program", + "ScaleUnit": "1Count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_prf_accuracy", + "MetricExpr": "100 * (l1i_useful_prf / l1i_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_prf_coverage", + "MetricExpr": "100 * (l1i_useful_prf / (l1i_demand_misses + l1i_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_refilled_prf", + "MetricExpr": "L1I_CACHE_REFILL_HWPRF + L1I_CACHE_REFILL_PRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 Instruction prefetcher (hardware prefetches or software preload) into L1 I-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_tlb_miss_ratio", + "MetricExpr": "L1I_TLB_REFILL / L1I_TLB", + "BriefDescription": "This metric measures the ratio of L1 Instruction TLB accesses missed to the total number of L1 Instruction TLB accesses. This gives an indication of the effectiveness of the L1 Instruction TLB.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" + }, + { + "MetricName": "l1i_tlb_mpki", + "MetricExpr": "1000 * (L1I_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 Instruction TLB accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness" + }, + { + "MetricName": "l1i_useful_prf", + "MetricExpr": "L1I_CACHE_HIT_RD_FPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 Instruction prefetcher (hardware prefetches or software preload) into L1 I-cache which are further used by Instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2_cache_miss_ratio", + "MetricExpr": "L2D_CACHE_REFILL / L2D_CACHE", + "BriefDescription": "This metric measures the ratio of L2 cache accesses missed to the total number of L2 cache accesses. This gives an indication of the effectiveness of the L2 cache, which is a unified cache that stores both Data and Instruction.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L2_Cache_Effectiveness" + }, + { + "MetricName": "l2_cache_mpki", + "MetricExpr": "1000 * (L2D_CACHE_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L2 unified cache accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L2_Cache_Effectiveness" + }, + { + "MetricName": "l2_tlb_miss_ratio", + "MetricExpr": "L2D_TLB_REFILL / L2D_TLB", + "BriefDescription": "This metric measures the ratio of L2 unified TLB accesses missed to the total number of L2 unified TLB accesses.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness;DTLB_Effectiveness" + }, + { + "MetricName": "l2_tlb_mpki", + "MetricExpr": "1000 * (L2D_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L2 unified TLB accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness;DTLB_Effectiveness" + }, + { + "MetricName": "l2d_cache_rwl1prf_miss_ratio", + "MetricExpr": "l2d_demand_misses / l2d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L2 D-cache Read accesses missed to the total number of L2 D-cache accesses.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_demand_accesses", + "MetricExpr": "L2D_CACHE_RD + L2D_CACHE_WR + L2D_CACHE_L1PRF", + "BriefDescription": "This metric measures the count of L2 D-cache accesses incurred on an Instruction Fetch, Load, Store, or L1 prefetcher accesses by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_demand_misses", + "MetricExpr": "L2D_CACHE_REFILL_RD + L2D_CACHE_REFILL_WR + L2D_CACHE_REFILL_L1PRF", + "BriefDescription": "This metric measures the count of L2 D-cache misses incurred on an Instruction Fetch, Load, Store, or L1 prefetcher accesses by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_prf_accuracy", + "MetricExpr": "100 * (l2d_useful_prf / l2d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_prf_coverage", + "MetricExpr": "100 * (l2d_useful_prf / (l2d_demand_misses + l2d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_refilled_prf", + "MetricExpr": "(L2D_CACHE_REFILL_PRF - L2D_CACHE_REFILL_L1PRF) + L2D_LFB_HIT_RWL1PRF_FHWPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L2 Data prefetcher (hardware prefetches or software preload) into L2 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_useful_prf", + "MetricExpr": "L2D_CACHE_HIT_RWL1PRF_FPRF + L2D_LFB_HIT_RWL1PRF_FHWPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L2 Data prefetcher (hardware prefetches or software preload) into L2 D-cache which are further used by Instruction Fetch, Load, Store, or L1 prefetcher accesses from the Instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_cache_rwl1prfl2prf_miss_ratio", + "MetricExpr": "l3d_demand_misses / l3d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L3 D-cache Read accesses missed to the total number of L3 D-cache accesses. This gives an indication of the effectiveness of the L2 D-cache for demand Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses traffic.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_demand_accesses", + "MetricExpr": "L3D_CACHE_RWL1PRFL2PRF", + "BriefDescription": "This metric measures the count of L3 D-cache accesses incurred on an Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_demand_misses", + "MetricExpr": "L3D_CACHE_REFILL_RWL1PRFL2PRF", + "BriefDescription": "This metric measures the count of L3 D-cache misses incurred on an Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the Instructions stream of the program", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_prf_accuracy", + "MetricExpr": "100 * (l3d_useful_prf / l3d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_prf_coverage", + "MetricExpr": "100 * (l3d_useful_prf / (l3d_demand_misses + l3d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_refilled_prf", + "MetricExpr": "L3D_CACHE_REFILL_HWPRF + L3D_CACHE_REFILL_PRFM - L3D_CACHE_REFILL_L1PRF - L3D_CACHE_REFILL_L2PRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L3 Data prefetcher (hardware prefetches or software preload) into L3 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_useful_prf", + "MetricExpr": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L3 Data prefetcher (hardware prefetches or software preload) into L3 D-cache which are further used by Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses from the Instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "ll_cache_read_hit_ratio", + "MetricExpr": "(LL_CACHE_RD - LL_CACHE_MISS_RD) / LL_CACHE_RD", + "BriefDescription": "This metric measures the ratio of last level cache Read accesses hit in the cache to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either Data memory access or Instruction fetch as this is a system level cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "LL_Cache_Effectiveness" + }, + { + "MetricName": "ll_cache_read_miss_ratio", + "MetricExpr": "LL_CACHE_MISS_RD / LL_CACHE_RD", + "BriefDescription": "This metric measures the ratio of last level cache Read accesses missed to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either Data memory access or Instruction fetch as this is a system level cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;LL_Cache_Effectiveness" + }, + { + "MetricName": "ll_cache_read_mpki", + "MetricExpr": "1000 * (LL_CACHE_MISS_RD / INST_RETIRED)", + "BriefDescription": "This metric measures the number of last level cache Read accesses missed per thousand Instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;LL_Cache_Effectiveness" + }, + { + "MetricName": "load_average_latency", + "MetricExpr": "MEM_ACCESS_RD_PERCYC / MEM_ACCESS", + "BriefDescription": "This metric measures the average latency of Load operations in CPU cycles", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "load_percentage", + "MetricExpr": "100 * (LD_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures Load operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "nonsve_fp_ops_per_cycle", + "MetricExpr": "FP_FIXED_OPS_SPEC / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by an Instruction that is not an SVE Instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "retiring", + "MetricExpr": "100 * ((OP_RETIRED/OP_SPEC) * (1 - (STALL_SLOT/CPU_SLOT)))", + "BriefDescription": "This metric is the percentage of total slots that retired operations, which indicates cycles that were utilized efficiently.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "scalar_fp_percentage", + "MetricExpr": "100 * (VFP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalar floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "simd_percentage", + "MetricExpr": "100 * (ASE_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures advanced SIMD operations as a percentage of total operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "store_percentage", + "MetricExpr": "100 * (ST_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures Store operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "sve_all_percentage", + "MetricExpr": "100 * (SVE_INST_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalable vector operations, including Loads and Stores, as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "sve_fp_ops_per_cycle", + "MetricExpr": "FP_SCALE_OPS_SPEC / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by SVE Instructions. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "sve_predicate_empty_percentage", + "MetricExpr": "100 * (SVE_PRED_EMPTY_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with no active predicates as a percentage of sve predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_full_percentage", + "MetricExpr": "100 * (SVE_PRED_FULL_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with all active predicates as a percentage of sve predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_partial_percentage", + "MetricExpr": "100 * (SVE_PRED_PARTIAL_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with at least one active predicates as a percentage of sve predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_percentage", + "MetricExpr": "100 * (SVE_PRED_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with predicates as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "SVE_Effectiveness" + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json new file mode 100644 index 0000000000000..0d657789ee482 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json @@ -0,0 +1,646 @@ +[ + { + "ArchStdEvent": "SW_INCR", + "PublicDescription": "The Event counts software writes to the PMSWINC_EL0 (software PMU increment) register. The PMSWINC_EL0 register is a manually updated counter for use by application software. This Event could be used to measure any user program Event, such as accesses to a particular Data structure (by writing to the PMSWINC_EL0 register each time the Data structure is accessed). To use the PMSWINC_EL0 register and Event, developers must insert Instructions that write to the PMSWINC_EL0 register into the source code. Since the SW_INCR Event records Writes to the PMSWINC_EL0 register, there is no need to do a Read/Increment/Write sequence to the PMSWINC_EL0 register." + }, + { + "ArchStdEvent": "CHAIN", + "PublicDescription": "For odd-numbered counters, this Event increments the count by one for each overflow of the preceding even-numbered counter. For even-numbered counters, there is no increment. This Event is used when the even/odd pairs of registers are used as a single counter." + }, + { + "ArchStdEvent": "TRB_WRAP", + "PublicDescription": "The Event is generated each time the trace buffer current Write pointer is wrapped to the trace buffer base pointer." + }, + { + "ArchStdEvent": "TRCEXTOUT0", + "PublicDescription": "Trace unit external output 0." + }, + { + "ArchStdEvent": "TRCEXTOUT1", + "PublicDescription": "Trace unit external output 1." + }, + { + "ArchStdEvent": "TRCEXTOUT2", + "PublicDescription": "Trace unit external output 2." + }, + { + "ArchStdEvent": "TRCEXTOUT3", + "PublicDescription": "Trace unit external output 3." + }, + { + "ArchStdEvent": "CTI_TRIGOUT4", + "PublicDescription": "Cross-trigger Interface output trigger 4." + }, + { + "ArchStdEvent": "CTI_TRIGOUT5", + "PublicDescription": "Cross-trigger Interface output trigger 5." + }, + { + "ArchStdEvent": "CTI_TRIGOUT6", + "PublicDescription": "Cross-trigger Interface output trigger 6." + }, + { + "ArchStdEvent": "CTI_TRIGOUT7", + "PublicDescription": "Cross-trigger Interface output trigger 7." + }, + { + "EventCode": "0x00e1", + "EventName": "L1I_PRFM_REQ_DROP", + "PublicDescription": "L1 I-cache software prefetch dropped." + }, + { + "EventCode": "0x0100", + "EventName": "L1_PF_REFILL", + "PublicDescription": "L1 prefetch requests, refilled to L1 cache." + }, + { + "EventCode": "0x0120", + "EventName": "FLUSH", + "PublicDescription": "The Event counts both the CT flush and BX flush. The BR_MIS_PRED counts the BX flushes. So the FLUSH-BR_MIS_PRED gives the CT flushes." + }, + { + "EventCode": "0x0121", + "EventName": "FLUSH_MEM", + "PublicDescription": "Flushes due to memory hazards. This only includes CT flushes." + }, + { + "EventCode": "0x0122", + "EventName": "FLUSH_BAD_BRANCH", + "PublicDescription": "Flushes due to bad predicted Branch. This only includes CT flushes." + }, + { + "EventCode": "0x0124", + "EventName": "FLUSH_ISB", + "PublicDescription": "Flushes due to ISB or similar side-effects. This only includes CT flushes." + }, + { + "EventCode": "0x0125", + "EventName": "FLUSH_OTHER", + "PublicDescription": "Flushes due to other hazards. This only includes CT flushes." + }, + { + "EventCode": "0x0126", + "EventName": "STORE_STREAM", + "PublicDescription": "Stored lines in streaming no-Write-allocate mode." + }, + { + "EventCode": "0x0127", + "EventName": "NUKE_RAR", + "PublicDescription": "Load/Store nuke due to Read-after-Read ordering hazard." + }, + { + "EventCode": "0x0128", + "EventName": "NUKE_RAW", + "PublicDescription": "Load/Store nuke due to Read-after-Write ordering hazard." + }, + { + "EventCode": "0x0129", + "EventName": "L1_PF_GEN_PAGE", + "PublicDescription": "Load/Store prefetch to L1 generated, Page mode." + }, + { + "EventCode": "0x012a", + "EventName": "L1_PF_GEN_STRIDE", + "PublicDescription": "Load/Store prefetch to L1 generated, stride mode." + }, + { + "EventCode": "0x012b", + "EventName": "L2_PF_GEN_LD", + "PublicDescription": "Load prefetch to L2 generated." + }, + { + "EventCode": "0x012d", + "EventName": "LS_PF_TRAIN_TABLE_ALLOC", + "PublicDescription": "LS prefetch train table entry allocated." + }, + { + "EventCode": "0x0130", + "EventName": "LS_PF_GEN_TABLE_ALLOC", + "PublicDescription": "The Event counts the number of cycles with at least one table allocation, for L2 hardware prefetches (including the SW PRFM that are converted into hardware prefetches due to D-TLB miss). LS prefetch gen table allocation (for L2 prefetches)." + }, + { + "EventCode": "0x0131", + "EventName": "LS_PF_GEN_TABLE_ALLOC_PF_PEND", + "PublicDescription": "The Event counts the number of cycles in which at least one hardware prefetch is dropped due to the inability to identify a victim when the generation table is full. The hardware prefetch considered here includes the software PRFM that is converted into hardware prefetches due to D-TLB miss." + }, + { + "EventCode": "0x0132", + "EventName": "TBW", + "PublicDescription": "Tablewalks." + }, + { + "EventCode": "0x0134", + "EventName": "S1L2_HIT", + "PublicDescription": "Translation cache hit on S1L2 walk cache entry." + }, + { + "EventCode": "0x0135", + "EventName": "S1L1_HIT", + "PublicDescription": "Translation cache hit on S1L1 walk cache entry." + }, + { + "EventCode": "0x0136", + "EventName": "S1L0_HIT", + "PublicDescription": "Translation cache hit on S1L0 walk cache entry." + }, + { + "EventCode": "0x0137", + "EventName": "S2L2_HIT", + "PublicDescription": "Translation cache hit for S2L2 IPA walk cache entry." + }, + { + "EventCode": "0x0138", + "EventName": "IPA_REQ", + "PublicDescription": "Translation cache lookups for IPA to PA entries." + }, + { + "EventCode": "0x0139", + "EventName": "IPA_REFILL", + "PublicDescription": "Translation cache refills for IPA to PA entries." + }, + { + "EventCode": "0x013a", + "EventName": "S1_FLT", + "PublicDescription": "Stage1 tablewalk fault." + }, + { + "EventCode": "0x013b", + "EventName": "S2_FLT", + "PublicDescription": "Stage2 tablewalk fault." + }, + { + "EventCode": "0x013c", + "EventName": "COLT_REFILL", + "PublicDescription": "Aggregated page refill." + }, + { + "EventCode": "0x0145", + "EventName": "L1_PF_HIT", + "PublicDescription": "L1 prefetch requests, hitting in L1 cache." + }, + { + "EventCode": "0x0146", + "EventName": "L1_PF", + "PublicDescription": "L1 prefetch requests." + }, + { + "EventCode": "0x0147", + "EventName": "CACHE_LS_REFILL", + "PublicDescription": "L2 D-cache refill, Load/Store." + }, + { + "EventCode": "0x0148", + "EventName": "CACHE_PF", + "PublicDescription": "L2 prefetch requests." + }, + { + "EventCode": "0x0149", + "EventName": "CACHE_PF_HIT", + "PublicDescription": "L2 prefetch requests, hitting in L2 cache." + }, + { + "EventCode": "0x0150", + "EventName": "UNUSED_PF", + "PublicDescription": "L2 unused prefetch." + }, + { + "EventCode": "0x0151", + "EventName": "PFT_SENT", + "PublicDescription": "L2 prefetch TGT sent. Note that PFT_SENT != PFT_USEFUL + PFT_DROP. There may be PFT_SENT for which the accesses resulted in a SLC hit." + }, + { + "EventCode": "0x0152", + "EventName": "PFT_USEFUL", + "PublicDescription": "L2 prefetch TGT useful." + }, + { + "EventCode": "0x0153", + "EventName": "PFT_DROP", + "PublicDescription": "L2 prefetch TGT dropped." + }, + { + "EventCode": "0x0162", + "EventName": "LRQ_FULL", + "PublicDescription": "The Event counts the number of cycles the LRQ is full." + }, + { + "EventCode": "0x0163", + "EventName": "FETCH_FQ_EMPTY", + "PublicDescription": "Fetch Queue empty cycles." + }, + { + "EventCode": "0x0164", + "EventName": "FPG2", + "PublicDescription": "Forward progress guarantee. Medium range livelock triggered." + }, + { + "EventCode": "0x0165", + "EventName": "FPG", + "PublicDescription": "Forward progress guarantee. Tofu global livelock buster is triggered." + }, + { + "EventCode": "0x0172", + "EventName": "DEADBLOCK", + "PublicDescription": "Write-back evictions converted to Dataless EVICT. The victim line is deemed deadblock if the likeliness of a reuse is low. The Core uses Dataless evict to evict a deadblock; And it uses a evict with Data to evict an L2 line that is not a deadblock." + }, + { + "EventCode": "0x0173", + "EventName": "PF_PRQ_ALLOC_PF_PEND", + "PublicDescription": "L1 prefetch prq allocation (replacing pending)." + }, + { + "EventCode": "0x0178", + "EventName": "FETCH_ICACHE_INSTR", + "PublicDescription": "Instructions fetched from I-cache." + }, + { + "EventCode": "0x017b", + "EventName": "NEAR_CAS", + "PublicDescription": "Near atomics: compare and swap." + }, + { + "EventCode": "0x017c", + "EventName": "NEAR_CAS_PASS", + "PublicDescription": "Near atomics: compare and swap pass." + }, + { + "EventCode": "0x017d", + "EventName": "FAR_CAS", + "PublicDescription": "Far atomics: compare and swap." + }, + { + "EventCode": "0x0186", + "EventName": "L2_BTB_RELOAD_MAIN_BTB", + "PublicDescription": "Number of completed L1 BTB update initiated by L2 BTB hit which swap Branch information between L1 BTB and L2 BTB." + }, + { + "EventCode": "0x0190", + "EventName": "PF_MODE_0_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most aggressive mode." + }, + { + "EventCode": "0x0191", + "EventName": "PF_MODE_1_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the more aggressive mode." + }, + { + "EventCode": "0x0192", + "EventName": "PF_MODE_2_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the less aggressive mode." + }, + { + "EventCode": "0x0193", + "EventName": "PF_MODE_3_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most conservative mode." + }, + { + "EventCode": "0x0194", + "EventName": "TXREQ_LIMIT_MAX_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is the L2_TQ_SIZE." + }, + { + "EventCode": "0x0195", + "EventName": "TXREQ_LIMIT_3QUARTER_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 3/4 of the L2_TQ_SIZE and the L2_TQ_SIZE-1." + }, + { + "EventCode": "0x0196", + "EventName": "TXREQ_LIMIT_HALF_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/2 of the L2_TQ_SIZE and 3/4 of the L2_TQ_SIZE." + }, + { + "EventCode": "0x0197", + "EventName": "TXREQ_LIMIT_1QUARTER_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/4 of the L2_TQ_SIZE and 1/2 of the L2_TQ_SIZE." + }, + { + "EventCode": "0x019d", + "EventName": "PREFETCH_LATE_CMC", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by CMC prefetch request." + }, + { + "EventCode": "0x019e", + "EventName": "PREFETCH_LATE_BO", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by BO prefetch request." + }, + { + "EventCode": "0x019f", + "EventName": "PREFETCH_LATE_STRIDE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by STRIDE prefetch request." + }, + { + "EventCode": "0x01a0", + "EventName": "PREFETCH_LATE_SPATIAL", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SPATIAL prefetch request." + }, + { + "EventCode": "0x01a2", + "EventName": "PREFETCH_LATE_TBW", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by TBW prefetch request." + }, + { + "EventCode": "0x01a3", + "EventName": "PREFETCH_LATE_PAGE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by PAGE prefetch request." + }, + { + "EventCode": "0x01a4", + "EventName": "PREFETCH_LATE_GSMS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by GSMS prefetch request." + }, + { + "EventCode": "0x01a5", + "EventName": "PREFETCH_LATE_SIP_CONS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SIP_CONS prefetch request." + }, + { + "EventCode": "0x01a6", + "EventName": "PREFETCH_REFILL_CMC", + "PublicDescription": "PF/prefetch or PF/readclean request from CMC pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a7", + "EventName": "PREFETCH_REFILL_BO", + "PublicDescription": "PF/prefetch or PF/readclean request from BO pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a8", + "EventName": "PREFETCH_REFILL_STRIDE", + "PublicDescription": "PF/prefetch or PF/readclean request from STRIDE pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a9", + "EventName": "PREFETCH_REFILL_SPATIAL", + "PublicDescription": "PF/prefetch or PF/readclean request from SPATIAL pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ab", + "EventName": "PREFETCH_REFILL_TBW", + "PublicDescription": "PF/prefetch or PF/readclean request from TBW pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ac", + "EventName": "PREFETCH_REFILL_PAGE", + "PublicDescription": "PF/prefetch or PF/readclean request from PAGE pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ad", + "EventName": "PREFETCH_REFILL_GSMS", + "PublicDescription": "PF/prefetch or PF/readclean request from GSMS pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ae", + "EventName": "PREFETCH_REFILL_SIP_CONS", + "PublicDescription": "PF/prefetch or PF/readclean request from SIP_CONS pf engine filled the L2 cache." + }, + { + "EventCode": "0x01af", + "EventName": "CACHE_HIT_LINE_PF_CMC", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by CMC prefetch request." + }, + { + "EventCode": "0x01b0", + "EventName": "CACHE_HIT_LINE_PF_BO", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by BO prefetch request." + }, + { + "EventCode": "0x01b1", + "EventName": "CACHE_HIT_LINE_PF_STRIDE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by STRIDE prefetch request." + }, + { + "EventCode": "0x01b2", + "EventName": "CACHE_HIT_LINE_PF_SPATIAL", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SPATIAL prefetch request." + }, + { + "EventCode": "0x01b4", + "EventName": "CACHE_HIT_LINE_PF_TBW", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by TBW prefetch request." + }, + { + "EventCode": "0x01b5", + "EventName": "CACHE_HIT_LINE_PF_PAGE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by PAGE prefetch request." + }, + { + "EventCode": "0x01b6", + "EventName": "CACHE_HIT_LINE_PF_GSMS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by GSMS prefetch request." + }, + { + "EventCode": "0x01b7", + "EventName": "CACHE_HIT_LINE_PF_SIP_CONS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SIP_CONS prefetch request." + }, + { + "EventCode": "0x01cb", + "EventName": "L2_TQ_OUTSTANDING", + "PublicDescription": "Outstanding tracker count, per cycle. The Event increments by the number of valid entries pertaining to this thread in the L2TQ, in each cycle. The Event can be used to calculate the occupancy of L2TQ by dividing this by the CPU_CYCLES Event. The L2TQ queue tracks the outstanding Read, Write ,and Snoop transactions. The Read transaction and the Write transaction entries are attributable to PE, whereas the Snoop transactions are not always attributable to PE." + }, + { + "EventCode": "0x01cc", + "EventName": "TXREQ_LIMIT_COUNT_CYCLES", + "PublicDescription": "The Event increments by the dynamic TXREQ value, in each cycle. This is a companion Event of TXREQ_LIMIT_MAX_CYCLES, TXREQ_LIMIT_3QUARTER_CYCLES, TXREQ_LIMIT_HALF_CYCLES, and TXREQ_LIMIT_1QUARTER_CYCLES." + }, + { + "EventCode": "0x01d2", + "EventName": "DVM_TLBI_RCVD", + "PublicDescription": "The Event counts the number of TLBI DVM message received over CHI interface, for *this* Core." + }, + { + "EventCode": "0x01d6", + "EventName": "DSB_COMMITING_LOCAL_TLBI", + "PublicDescription": "The Event counts the number of DSB that are retired and committed at least one local TLBI Instruction. This Event increments no more than once (in a cycle) even if the DSB commits multiple local TLBI Instruction." + }, + { + "EventCode": "0x01d7", + "EventName": "DSB_COMMITING_BROADCAST_TLBI", + "PublicDescription": "The Event counts the number of DSB that are retired and committed at least one broadcast TLBI Instruction. This Event increments no more than once (in a cycle) even if the DSB commits multiple broadcast TLBI Instruction." + }, + { + "EventCode": "0x01f0", + "EventName": "TMS_ST_TO_SMT_LATENCY", + "PublicDescription": "The Event counts the number of CPU cycles spent on TMS for ST-to-SMT switch. This Event is counted by both the threads - The Event in both threads increment during TMS for ST-to-SMT switch." + }, + { + "EventCode": "0x01f1", + "EventName": "TMS_SMT_TO_ST_LATENCY", + "PublicDescription": "The Event counts the number of CPU cycles spent on TMS for SMT-to-ST switch. The count also includes the CPU cycles spend due to an aborted SMT-to-ST TMS attempt. This Event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x01f2", + "EventName": "TMS_ST_TO_SMT_COUNT", + "PublicDescription": "The Event counts the number of completed TMS from ST-to-SMT. This Event is counted only by the active thread (the one that is not in WFI). Note: When an active thread enters the Debug state in ST-Full resource mode, it is switched to SMT mode. This is because the inactive thread cannot wake up while the other thread remains in the Debug state. To prEvent this issue, threads operating in ST-Full resource mode are transitioned to SMT mode upon entering Debug state. The Event count will also reflect such switches from ST to SMT mode. (Also see the (NV_CPUACTLR14_EL1.chka_prEvent_st_tx_to_smt_when_tx_in_debug_state bit to disable this behavior.)" + }, + { + "EventCode": "0x01f3", + "EventName": "TMS_SMT_TO_ST_COUNT", + "PublicDescription": "The Event counts the number of completed TMS from SMT-to-ST. This Event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x01f4", + "EventName": "TMS_SMT_TO_ST_COUNT_ABRT", + "PublicDescription": "The Event counts the number of aborted TMS from SMT-to-ST. This Event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x021c", + "EventName": "CWT_ALLOC_ENTRY", + "PublicDescription": "Cache Way Tracker Allocate entry." + }, + { + "EventCode": "0x021d", + "EventName": "CWT_ALLOC_LINE", + "PublicDescription": "Cache Way Tracker Allocate line." + }, + { + "EventCode": "0x021e", + "EventName": "CWT_HIT", + "PublicDescription": "Cache Way Tracker hit." + }, + { + "EventCode": "0x021f", + "EventName": "CWT_HIT_TAG", + "PublicDescription": "Cache Way Tracker hit when ITAG lookup suppressed." + }, + { + "EventCode": "0x0220", + "EventName": "CWT_REPLAY_TAG", + "PublicDescription": "Cache Way Tracker causes ITAG replay due to miss when ITAG lookup suppressed." + }, + { + "EventCode": "0x0250", + "EventName": "GPT_REQ", + "PublicDescription": "GPT lookup." + }, + { + "EventCode": "0x0251", + "EventName": "GPT_WC_HIT", + "PublicDescription": "GPT lookup hit in Walk cache." + }, + { + "EventCode": "0x0252", + "EventName": "GPT_PG_HIT", + "PublicDescription": "GPT lookup hit in TLB." + }, + { + "EventCode": "0x01ba", + "EventName": "PREFETCH_LATE_STORE_ISSUE", + "PublicDescription": "The Event counts the number of demand requests that matches a Store-issue prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bb", + "EventName": "PREFETCH_LATE_STORE_STRIDE", + "PublicDescription": "The Event counts the number of demand requests that matches a Store-stride prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bc", + "EventName": "PREFETCH_LATE_PC_OFFSET", + "PublicDescription": "The Event counts the number of demand requests that matches a PC-offset prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bd", + "EventName": "PREFETCH_LATE_IFUPF", + "PublicDescription": "The Event counts the number of demand requests that matches a IFU prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01be", + "EventName": "PREFETCH_REFILL_STORE_ISSUE", + "PublicDescription": "The Event counts the number of cache refills due to Store-Issue prefetcher." + }, + { + "EventCode": "0x01bf", + "EventName": "PREFETCH_REFILL_STORE_STRIDE", + "PublicDescription": "The Event counts the number of cache refills due to Store-stride prefetcher." + }, + { + "EventCode": "0x01c0", + "EventName": "PREFETCH_REFILL_PC_OFFSET", + "PublicDescription": "The Event counts the number of cache refills due to PC-offset prefetcher." + }, + { + "EventCode": "0x01c1", + "EventName": "PREFETCH_REFILL_IFUPF", + "PublicDescription": "The Event counts the number of cache refills due to IFU prefetcher." + }, + { + "EventCode": "0x01c2", + "EventName": "CACHE_HIT_LINE_PF_STORE_ISSUE", + "PublicDescription": "The Event counts the number of first hit to a cache line filled by Store-issue prefetcher." + }, + { + "EventCode": "0x01c3", + "EventName": "CACHE_HIT_LINE_PF_STORE_STRIDE", + "PublicDescription": "The Event counts the number of first hit to a cache line filled by Store-stride prefetcher." + }, + { + "EventCode": "0x01c4", + "EventName": "CACHE_HIT_LINE_PF_PC_OFFSET", + "PublicDescription": "The Event counts the number of first hit to a cache line filled by PC-offset prefetcher." + }, + { + "EventCode": "0x01c5", + "EventName": "CACHE_HIT_LINE_PF_IFUPF", + "PublicDescription": "The Event counts the number of first hit to a cache line filled by IFU prefetcher." + }, + { + "EventCode": "0x01c6", + "EventName": "L2_PF_GEN_ST_ISSUE", + "PublicDescription": "Store-issue prefetch to L2 generated." + }, + { + "EventCode": "0x01c7", + "EventName": "L2_PF_GEN_ST_STRIDE", + "PublicDescription": "Store-stride prefetch to L2 generated" + }, + { + "EventCode": "0x01ee", + "EventName": "CACHE_HIT_LINE_PF_CONVERTED_PRFM", + "PublicDescription": "The Event counts the number of first hit to a cache line filled by Converted-L1D-PRFM or Converted-L2D-PRFM. Note that L2D_CACHE_HIT_RWL1PRF_FPRFM is inclusive of CACHE_HIT_LINE_PF_CONVERTED_PRFM, where both the CACHE_HIT_LINE_PF_CONVERTED_PRFM and the L2D_CACHE_HIT_RWL1PRF_FPRFM increment on a first hit to L2 D-cache filled by Converted-L1D-PRFM or Converted-L2D-PRFM." + }, + { + "EventCode": "0x01ec", + "EventName": "PREFETCH_LATE_CONVERTED_PRFM", + "PublicDescription": "The Event counts the number of demand requests that matches a Converted-L1D-PRFM or Converted-L2D-PRFM pending refill request at L2 D-cache. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements. Note that this Event is not counted by the L2D_CACHE_HIT_RWL1PRF_LATE_HWPRF, though the Converted-L1D-PRFM or Converted-L2D-PRFM are replayed by the L2PRQ." + }, + { + "EventCode": "0x01ed", + "EventName": "PREFETCH_REFILL_CONVERTED_PRFM", + "PublicDescription": "The Event counts the number of L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM. Note : L2D_CACHE_REFILL_PRFM is inclusive of PREFETCH_REFILL_PRFM_CONVERTED, where both the PREFETCH_REFILL_PRFM_CONVERTED and the L2D_CACHE_REFILL_PRFM increment when L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM." + }, + { + "EventCode": "0x01eb", + "EventName": "L1DPRFM_L2DPRFM_TO_L2PRQ_CONVERTED", + "PublicDescription": "The Event counts the number of Converted-L1D-PRFMs and Converted-L2D-PRFM. Activities involving the Converted-L1D-PRFM are counted by the L1D_CACHE_PRFM. However they are *not* counted by the L1D_CACHE_REFILL_PRFM, and L1D_CACHE_REFILL, as these Converted-L1D-PRFM are treated as L2 D hardware prefetches. Activities around the Converted-L1D-PRFMs and Converted-L2D-PRFMs are counted by the L2D_CACHE_PRFM, L2D_CACHE_REFILL_PRFM and L2D_CACHE_REFILL Events." + }, + { + "EventCode": "0x01ce", + "EventName": "L3DPRFM_TO_L2PRQ_CONVERTED", + "PublicDescription": "The Event counts the number of Converted-L3D-PRFMs. These are indeed L3D PRFM and activities around these PRFM are counted by the L3D_CACHE_PRFM, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL Events." + }, + { + "EventCode": "0x0202", + "EventName": "L0I_CACHE_RD", + "PublicDescription": "The Event counts the number of predict blocks serviced out of L0 I-cache. Note: The L0 I-cache performs at most 4 L0 I look-up in a cycle. Two of which are to service PB from L0 I. And the other two to refill L0 I-cache from L1 I. This Event count only the L0 I-cache lookup pertaining to servicing the PB from L0 I." + }, + { + "EventCode": "0x0203", + "EventName": "L0I_CACHE_REFILL", + "PublicDescription": "The Event counts the number of L0I cache refill from L1 I-cache." + }, + { + "EventCode": "0x0207", + "EventName": "INTR_LATENCY", + "PublicDescription": "The Event counts the number of cycles elapsed between when an Interrupt is recognized (after masking) to when a uop associated with the first Instruction in the destination exception level is allocated. If there is some other flush condition that pre-empts the Interrupt, then the cycles counted terminates early at the first Instruction executed after that flush. In the Event of dropped Interrupts (when an Interrupt is deasserted before it is taken), this counter measures the number of cycles that elapse from the moment an Interrupt is recognized (post-masking) until the Interrupt is dropped or deasserted." + }, + { + "EventCode": "0x018f", + "EventName": "L1_PF_GEN_MCMC", + "PublicDescription": "Load/Store prefetch to L1 generated, MCMC." + }, + { + "EventCode": "0x0123", + "EventName": "FLUSH_STDBYPASS", + "PublicDescription": "Flushes due to bad predecode. This only includes CT flushes." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json new file mode 100644 index 0000000000000..ab384239541b8 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json @@ -0,0 +1,94 @@ +[ + { + "ArchStdEvent": "INST_RETIRED", + "PublicDescription": "The Event counts Instructions that have been architecturally executed." + }, + { + "ArchStdEvent": "CID_WRITE_RETIRED", + "PublicDescription": "The Event counts architecturally executed Writes to the CONTEXTIDR_EL1 register, which usually contain the kernel PID and can be output with hardware trace." + }, + { + "ArchStdEvent": "BR_IMMED_RETIRED", + "PublicDescription": "The Event counts architecturally executed direct Branches." + }, + { + "ArchStdEvent": "BR_RETURN_RETIRED", + "PublicDescription": "The Event counts architecturally executed procedure returns." + }, + { + "ArchStdEvent": "TTBR_WRITE_RETIRED", + "PublicDescription": "The Event counts architectural Writes to TTBR0/1_EL1. If virtualization host extensions are enabled (by setting the HCR_EL2.E2H bit to 1), then accesses to TTBR0/1_EL1 that are redirected to TTBR0/1_EL2, or accesses to TTBR0/1_EL12, are counted. TTBRn registers are typically updated when the kernel is swapping user-space threads or applications." + }, + { + "ArchStdEvent": "BR_RETIRED", + "PublicDescription": "The Event counts architecturally executed Branches, whether the Branch is taken or not. Instructions that explicitly write to the PC are also counted. Note that exception generating Instructions, exception return Instructions and context synchronization Instructions are not counted." + }, + { + "ArchStdEvent": "BR_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts Branches counted by BR_RETIRED which were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "OP_RETIRED", + "PublicDescription": "The Event counts micro-operations that are architecturally executed. This is a count of number of micro-operations retired from the commit queue in a single cycle." + }, + { + "ArchStdEvent": "BR_INDNR_TAKEN_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were taken." + }, + { + "ArchStdEvent": "BR_IMMED_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed direct Branches that were correctly predicted." + }, + { + "ArchStdEvent": "BR_IMMED_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed direct Branches that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_IND_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_IND_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_RETURN_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_RETURN_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_INDNR_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_INDNR_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_TAKEN_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed Branches that were taken and were correctly predicted." + }, + { + "ArchStdEvent": "BR_TAKEN_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed branches that were taken and were mispredicted causing a pipeline flush." + }, + { + "ArchStdEvent": "BR_SKIP_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed Branches that were not taken and were correctly predicted." + }, + { + "ArchStdEvent": "BR_SKIP_MIS_PRED_RETIRED", + "PublicDescription": "The Event counts architecturally executed Branches that were not taken and were mispredicted causing a pipeline flush." + }, + { + "ArchStdEvent": "BR_PRED_RETIRED", + "PublicDescription": "The Event counts Branch Instructions counted by BR_RETIRED which were correctly predicted." + }, + { + "ArchStdEvent": "BR_IND_RETIRED", + "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json new file mode 100644 index 0000000000000..8d1fe2a8b161f --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json @@ -0,0 +1,42 @@ +[ + { + "ArchStdEvent": "SAMPLE_POP", + "PublicDescription": "The Event counts statistical profiling sample population, the count of all operations that could be sampled but may or may not be chosen for sampling." + }, + { + "ArchStdEvent": "SAMPLE_FEED", + "PublicDescription": "The Event counts statistical profiling samples taken for sampling." + }, + { + "ArchStdEvent": "SAMPLE_FILTRATE", + "PublicDescription": "The Event counts statistical profiling samples taken which are not removed by filtering." + }, + { + "ArchStdEvent": "SAMPLE_COLLISION", + "PublicDescription": "The Event counts statistical profiling samples that have collided with a previous sample and so therefore not taken." + }, + { + "ArchStdEvent": "SAMPLE_FEED_BR", + "PublicDescription": "The Event counts statistical profiling samples taken which are Branches." + }, + { + "ArchStdEvent": "SAMPLE_FEED_LD", + "PublicDescription": "The Event counts statistical profiling samples taken which are Loads or Load atomic operations." + }, + { + "ArchStdEvent": "SAMPLE_FEED_ST", + "PublicDescription": "The Event counts statistical profiling samples taken which are Stores or Store atomic operations." + }, + { + "ArchStdEvent": "SAMPLE_FEED_OP", + "PublicDescription": "The Event counts statistical profiling samples taken which are matching any operation type filters supported." + }, + { + "ArchStdEvent": "SAMPLE_FEED_EVENT", + "PublicDescription": "The Event counts statistical profiling samples taken which are matching Event packet filter constraints." + }, + { + "ArchStdEvent": "SAMPLE_FEED_LAT", + "PublicDescription": "The Event counts statistical profiling samples taken which are exceeding minimum latency set by operation latency filter constraints." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json new file mode 100644 index 0000000000000..18df3f7011150 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json @@ -0,0 +1,230 @@ +[ + { + "ArchStdEvent": "INST_SPEC", + "PublicDescription": "The Event counts operations that have been speculatively executed." + }, + { + "ArchStdEvent": "OP_SPEC", + "PublicDescription": "The Event counts micro-operations speculatively executed. This is the count of the number of micro-operations dispatched in a cycle." + }, + { + "ArchStdEvent": "UNALIGNED_LD_SPEC", + "PublicDescription": "The Event counts unaligned memory Read operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. The Event does not count preload operations (PLD, PLI). This Event is a subset of the UNALIGNED_LDST_SPEC Event." + }, + { + "ArchStdEvent": "UNALIGNED_ST_SPEC", + "PublicDescription": "The Event counts unaligned memory Write operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. This Event is a subset of the UNALIGNED_LDST_SPEC Event." + }, + { + "ArchStdEvent": "UNALIGNED_LDST_SPEC", + "PublicDescription": "The Event counts unaligned memory operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. This Event is the sum of the UNALIGNED_ST_SPEC and UNALIGNED_LD_SPEC Events." + }, + { + "ArchStdEvent": "LDREX_SPEC", + "PublicDescription": "The Event counts Load-Exclusive operations that have been speculatively executed. For example: LDREX, LDX" + }, + { + "ArchStdEvent": "STREX_PASS_SPEC", + "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed and have successfully completed the Store operation." + }, + { + "ArchStdEvent": "STREX_FAIL_SPEC", + "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed and have not successfully completed the Store operation." + }, + { + "ArchStdEvent": "STREX_SPEC", + "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed. This Event is the sum of STREX_PASS_SPEC and STREX_FAIL_SPEC Events." + }, + { + "ArchStdEvent": "LD_SPEC", + "PublicDescription": "The Event counts speculatively executed Load operations including Single Instruction Multiple Data (SIMD) Load operations." + }, + { + "ArchStdEvent": "ST_SPEC", + "PublicDescription": "The Event counts speculatively executed Store operations including Single Instruction Multiple Data (SIMD) Store operations." + }, + { + "ArchStdEvent": "LDST_SPEC", + "PublicDescription": "The Event counts Load and Store operations that have been speculatively executed." + }, + { + "ArchStdEvent": "DP_SPEC", + "PublicDescription": "The Event counts speculatively executed logical or arithmetic Instructions such as MOV/MVN operations." + }, + { + "ArchStdEvent": "ASE_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD operations excluding Load, Store, and Move micro-operations that move Data to or from SIMD (vector) registers." + }, + { + "ArchStdEvent": "VFP_SPEC", + "PublicDescription": "The Event counts speculatively executed floating point operations. This Event does not count operations that move Data to or from floating point (vector) registers." + }, + { + "ArchStdEvent": "PC_WRITE_SPEC", + "PublicDescription": "The Event counts speculatively executed operations which cause software changes of the PC. Those operations include all taken Branch operations." + }, + { + "ArchStdEvent": "CRYPTO_SPEC", + "PublicDescription": "The Event counts speculatively executed cryptographic operations except for PMULL and VMULL operations." + }, + { + "ArchStdEvent": "BR_IMMED_SPEC", + "PublicDescription": "The Event counts direct Branch operations which are speculatively executed." + }, + { + "ArchStdEvent": "BR_RETURN_SPEC", + "PublicDescription": "The Event counts procedure return operations (RET, RETAA and RETAB) which are speculatively executed." + }, + { + "ArchStdEvent": "BR_INDIRECT_SPEC", + "PublicDescription": "The Event counts indirect branch operations including procedure returns, which are speculatively executed. This includes operations that force a software change of the PC, other than exception-generating operations and direct Branch Instructions. Some examples of the Instructions counted by this Event include BR Xn, RET, etc." + }, + { + "ArchStdEvent": "ISB_SPEC", + "PublicDescription": "The Event counts ISB operations that are executed." + }, + { + "ArchStdEvent": "DSB_SPEC", + "PublicDescription": "The Event counts DSB operations that are speculatively issued to Load/Store unit in the CPU." + }, + { + "ArchStdEvent": "DMB_SPEC", + "PublicDescription": "The Event counts DMB operations that are speculatively issued to the Load/Store unit in the CPU. This Event does not count implied barriers from Load-acquire/Store-release operations." + }, + { + "ArchStdEvent": "CSDB_SPEC", + "PublicDescription": "The Event counts CSDB operations that are speculatively issued to the Load/Store unit in the CPU. This Event does not count implied barriers from Load-acquire/Store-release operations." + }, + { + "ArchStdEvent": "RC_LD_SPEC", + "PublicDescription": "The Event counts any Load acquire operations that are speculatively executed. For example: LDAR, LDARH, LDARB" + }, + { + "ArchStdEvent": "RC_ST_SPEC", + "PublicDescription": "The Event counts any Store release operations that are speculatively executed. For example: STLR, STLRH, STLRB" + }, + { + "ArchStdEvent": "SIMD_INST_SPEC", + "PublicDescription": "The Event counts speculatively executed operations that are SIMD or SVE vector operations or Advanced SIMD non-scalar operations." + }, + { + "ArchStdEvent": "ASE_INST_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD operations." + }, + { + "ArchStdEvent": "SVE_INST_SPEC", + "PublicDescription": "The Event counts speculatively executed operations that are SVE operations." + }, + { + "ArchStdEvent": "INT_SPEC", + "PublicDescription": "The Event counts speculatively executed integer arithmetic operations." + }, + { + "ArchStdEvent": "SVE_PRED_SPEC", + "PublicDescription": "The Event counts speculatively executed predicated SVE operations. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." + }, + { + "ArchStdEvent": "SVE_PRED_EMPTY_SPEC", + "PublicDescription": "The Event counts speculatively executed predicated SVE operations with no active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." + }, + { + "ArchStdEvent": "SVE_PRED_FULL_SPEC", + "PublicDescription": "The Event counts speculatively executed predicated SVE operations with all predicate elements active. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." + }, + { + "ArchStdEvent": "SVE_PRED_PARTIAL_SPEC", + "PublicDescription": "The Event counts speculatively executed predicated SVE operations with at least one but not all active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." + }, + { + "ArchStdEvent": "SVE_PRED_NOT_FULL_SPEC", + "PublicDescription": "The Event counts speculatively executed predicated SVE operations with at least one non active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." + }, + { + "ArchStdEvent": "PRF_SPEC", + "PublicDescription": "The Event counts speculatively executed operations that prefetch memory. For example, Scalar: PRFM, SVE: PRFB, PRFD, PRFH, or PRFW." + }, + { + "ArchStdEvent": "SVE_LDFF_SPEC", + "PublicDescription": "The Event counts speculatively executed SVE first fault or non-fault Load operations." + }, + { + "ArchStdEvent": "SVE_LDFF_FAULT_SPEC", + "PublicDescription": "The Event counts speculatively executed SVE first fault or non-fault Load operations that clear at least one bit in the FFR." + }, + { + "ArchStdEvent": "ASE_SVE_INT8_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type an 8-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT16_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 16-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT32_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 32-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT64_SPEC", + "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 64-bit integer." + }, + { + "EventCode": "0x011d", + "EventName": "SPEC_RET_STACK_FULL", + "PublicDescription": "The Event counts predict pipe stalls due to speculative return address predictor full." + }, + { + "EventCode": "0x011f", + "EventName": "MOPS_SPEC", + "PublicDescription": "Macro-ops speculatively decoded." + }, + { + "EventCode": "0x0180", + "EventName": "BR_SPEC_PRED_TAKEN", + "PublicDescription": "Number of Predicted Taken from Branch Predictor." + }, + { + "EventCode": "0x0181", + "EventName": "BR_SPEC_PRED_TAKEN_FROM_L2BTB", + "PublicDescription": "Number of Predicted Taken Branch from L2 BTB." + }, + { + "EventCode": "0x0182", + "EventName": "BR_SPEC_PRED_TAKEN_MULTI", + "PublicDescription": "Number of Predicted Taken for Polymorphic Branch." + }, + { + "EventCode": "0x0185", + "EventName": "BR_SPEC_PRED_STATIC", + "PublicDescription": "Number of post fetch prediction." + }, + { + "EventCode": "0x01d0", + "EventName": "TLBI_LOCAL_SPEC", + "PublicDescription": "A non-broadcast TLBI Instruction executed (Speculatively or otherwise) on *this* PE." + }, + { + "EventCode": "0x01d1", + "EventName": "TLBI_BROADCAST_SPEC", + "PublicDescription": "A broadcast TLBI Instruction executed (Speculatively or otherwise) on *this* PE." + }, + { + "EventCode": "0x0200", + "EventName": "SIMD_CRYPTO_INST_SPEC", + "PublicDescription": "SIMD, SVE, and CRYPTO Instructions speculatively decoded." + }, + { + "EventCode": "0x01e7", + "EventName": "BR_SPEC_PRED_ALN_REDIR", + "PublicDescription": "BPU predict pipe align redirect (either AL-APQ hit/miss)." + }, + { + "EventCode": "0x022e", + "EventName": "VPRED_LD_SPEC", + "PublicDescription": "The Event counts the number of Speculatively-executed-Load operations with addresses produced by the value-prediction mechanism. The loaded Data might be discarded if the predicted address differs from the actual address." + }, + { + "EventCode": "0x022f", + "EventName": "VPRED_LD_SPEC_MISMATCH", + "PublicDescription": "The Event counts a subset of VPRED_LD_SPEC where the predicted Load address and the actual address mismatched." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json new file mode 100644 index 0000000000000..652c1e3305d08 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json @@ -0,0 +1,145 @@ +[ + { + "ArchStdEvent": "STALL_FRONTEND", + "PublicDescription": "The Event counts cycles when frontend could not send any micro-operations to the rename stage because of frontend resource stalls caused by fetch memory latency or Branch prediction flow stalls. STALL_FRONTEND_SLOTS counts SLOTS during the cycle when this Event counts. STALL_SLOT_FRONTEND will count SLOTS when this Event is counted on this CPU." + }, + { + "ArchStdEvent": "STALL_BACKEND", + "PublicDescription": "The Event counts cycles whenever the rename unit is unable to send any micro-operations to the backend of the pipeline because of backend resource constraints. Backend resource constraints can include issue stage fullness, execution stage fullness, or other internal pipeline resource fullness. All the backend slots were empty during the cycle when this Event counts." + }, + { + "ArchStdEvent": "STALL", + "PublicDescription": "The Event counts cycles when no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall). This Event is the sum of STALL_FRONTEND and STALL_BACKEND." + }, + { + "ArchStdEvent": "STALL_SLOT_BACKEND", + "PublicDescription": "The Event counts slots per cycle in which no operations are sent from the rename unit to the backend due to backend resource constraints. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND counts at least 1. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND is SLOTS." + }, + { + "ArchStdEvent": "STALL_SLOT_FRONTEND", + "PublicDescription": "The Event counts slots per cycle in which no operations are sent to the rename unit from the frontend due to frontend resource constraints. STALL_FRONTEND counts during the cycle when STALL_SLOT_FRONTEND is SLOTS." + }, + { + "ArchStdEvent": "STALL_SLOT", + "PublicDescription": "The Event counts slots per cycle in which no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall). STALL_SLOT is the sum of STALL_SLOT_FRONTEND and STALL_SLOT_BACKEND." + }, + { + "ArchStdEvent": "STALL_BACKEND_MEM", + "PublicDescription": "The Event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the last level Core cache. Last level cache in this CPU is Level 2, hence this Event counts same as STALL_BACKEND_L2D." + }, + { + "ArchStdEvent": "STALL_FRONTEND_MEMBOUND", + "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the memory resources." + }, + { + "ArchStdEvent": "STALL_FRONTEND_L1I", + "PublicDescription": "The Event counts cycles when the frontend is stalled because there is an Instruction fetch request pending in the L1 I-cache." + }, + { + "ArchStdEvent": "STALL_FRONTEND_MEM", + "PublicDescription": "The Event counts cycles when the frontend is stalled because there is an Instruction fetch request pending in the last level Core cache. Last level cache in this CPU is Level 2, hence this Event counts rather than STALL_FRONTEND_L2I." + }, + { + "ArchStdEvent": "STALL_FRONTEND_TLB", + "PublicDescription": "The Event counts when the frontend is stalled on any TLB misses being handled. This Event also counts the TLB accesses made by hardware prefetches." + }, + { + "ArchStdEvent": "STALL_FRONTEND_CPUBOUND", + "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the CPU resources excluding memory resources." + }, + { + "ArchStdEvent": "STALL_FRONTEND_FLOW", + "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the Branch prediction unit." + }, + { + "ArchStdEvent": "STALL_FRONTEND_FLUSH", + "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage as the frontend is recovering from a machine flush or resteer. Example scenarios that cause a flush include Branch mispredictions, taken exceptions, microarchitectural flush etc." + }, + { + "ArchStdEvent": "STALL_BACKEND_MEMBOUND", + "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to resource constraints in the memory resources." + }, + { + "ArchStdEvent": "STALL_BACKEND_L1D", + "PublicDescription": "The Event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the L1 D-cache." + }, + { + "ArchStdEvent": "STALL_BACKEND_TLB", + "PublicDescription": "The Event counts cycles when the backend is stalled on any demand TLB misses being handled." + }, + { + "ArchStdEvent": "STALL_BACKEND_ST", + "PublicDescription": "The Event counts cycles when the backend is stalled and there is a Store that has not reached the pre-commit stage." + }, + { + "ArchStdEvent": "STALL_BACKEND_CPUBOUND", + "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to any resource constraints in the CPU excluding memory resources." + }, + { + "ArchStdEvent": "STALL_BACKEND_BUSY", + "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations because the issue queues are full to take any operations for execution." + }, + { + "ArchStdEvent": "STALL_BACKEND_ILOCK", + "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to resource constraints imposed by input dependency." + }, + { + "ArchStdEvent": "STALL_BACKEND_RENAME", + "PublicDescription": "The Event counts cycles when backend is stalled even when operations are available from the frontend but at least one is not ready to be sent to the backend because no rename register is available." + }, + { + "EventCode": "0x0158", + "EventName": "FLAG_DISP_STALL", + "PublicDescription": "Rename stalled due to FRF(Flag register file) full." + }, + { + "EventCode": "0x0159", + "EventName": "GEN_DISP_STALL", + "PublicDescription": "Rename stalled due to GRF (General-purpose register file) full." + }, + { + "EventCode": "0x015a", + "EventName": "VEC_DISP_STALL", + "PublicDescription": "Rename stalled due to VRF (Vector register file) full." + }, + { + "EventCode": "0x015c", + "EventName": "SX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, SX." + }, + { + "EventCode": "0x015d", + "EventName": "MX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, MX." + }, + { + "EventCode": "0x015e", + "EventName": "LS_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, LS." + }, + { + "EventCode": "0x015f", + "EventName": "VX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, VX." + }, + { + "EventCode": "0x0160", + "EventName": "MCQ_FULL_STALL", + "PublicDescription": "Dispatch stalled due to MCQ full." + }, + { + "EventCode": "0x01cf", + "EventName": "PRD_DISP_STALL", + "PublicDescription": "Rename stalled due to predicate registers (physical) are full." + }, + { + "EventCode": "0x01e0", + "EventName": "CSDB_STALL", + "PublicDescription": "Rename stalled due to CSDB." + }, + { + "EventCode": "0x01e2", + "EventName": "STALL_SLOT_FRONTEND_WITHOUT_MISPRED", + "PublicDescription": "Stall slot frontend during non-mispredicted branch. The Event counts the STALL_STOT_FRONTEND Events, except for the 4 cycles following a mispredicted branch Event or 4 cycles following a commit flush&restart Event." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json new file mode 100644 index 0000000000000..9a81a62a26462 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json @@ -0,0 +1,158 @@ +[ + { + "ArchStdEvent": "L1I_TLB_REFILL", + "PublicDescription": "The Event counts L1 Instruction TLB refills from any Instruction fetch (Demand, hardware prefetch and software preload accesses). If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL", + "PublicDescription": "The Event counts L1 Data TLB accesses that resulted in TLB refills. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count on an access from an AT(address translation) Instruction. This Event is the sum of the L1D_TLB_REFILL_RD and L1D_TLB_REFILL_WR Events." + }, + { + "ArchStdEvent": "L1D_TLB", + "PublicDescription": "The Event counts L1 Data TLB accesses caused by any memory Load or Store operation. Note that Load or Store Instructions can be broken up into multiple memory operations. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB", + "PublicDescription": "The Event counts L1 Instruction TLB accesses (caused by Demand or hardware prefetch or software preload accesses), whether the access hits or misses in the TLB. This Event counts both demand accesses and prefetch or preload generated accesses. This Event is a superset of the L1I_TLB_REFILL Event." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL", + "PublicDescription": "The Event counts L2 TLB refills caused by memory operations from both Data and Instruction fetch, except for those caused by TLB maintenance operations and hardware prefetches. This Event is the sum of the L2D_TLB_REFILL_RD and L2D_TLB_REFILL_WR Events." + }, + { + "ArchStdEvent": "L2D_TLB", + "PublicDescription": "The Event counts L2 TLB accesses except those caused by TLB maintenance operations. This Event is the sum of the L2D_TLB_RD and L2D_TLB_WR Events." + }, + { + "ArchStdEvent": "DTLB_WALK", + "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations. This Event does not include prefetches." + }, + { + "ArchStdEvent": "ITLB_WALK", + "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations. This Event does not include prefetches." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL_RD", + "PublicDescription": "The Event counts L1 Data TLB refills caused by memory Read operations. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count on an access from an Address Translation (AT) Instruction. This Event is a subset of the L1D_TLB_REFILL Event." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL_WR", + "PublicDescription": "The Event counts L1 Data TLB refills caused by Data side memory Write operations. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count with an access from an Address Translation (AT) Instruction. This Event is a subset of the L1D_TLB_REFILL Event." + }, + { + "ArchStdEvent": "L1D_TLB_RD", + "PublicDescription": "The Event counts L1 Data TLB accesses caused by memory Read operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1D_TLB_WR", + "PublicDescription": "The Event counts any L1 Data side TLB accesses caused by memory Write operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL_RD", + "PublicDescription": "The Event counts L2 TLB refills caused by memory Read operations from both Data and Instruction fetch except for those caused by TLB maintenance operations or hardware prefetches. This Event is a subset of the L2D_TLB_REFILL Event." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL_WR", + "PublicDescription": "The Event counts L2 TLB refills caused by memory Write operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB_REFILL Event." + }, + { + "ArchStdEvent": "L2D_TLB_RD", + "PublicDescription": "The Event counts L2 TLB accesses caused by memory Read operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB Event." + }, + { + "ArchStdEvent": "L2D_TLB_WR", + "PublicDescription": "The Event counts L2 TLB accesses caused by memory Write operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB Event." + }, + { + "ArchStdEvent": "DTLB_WALK_PERCYC", + "PublicDescription": "The Event counts the number of Data translation table walks in progress per cycle." + }, + { + "ArchStdEvent": "ITLB_WALK_PERCYC", + "PublicDescription": "The Event counts the number of Instruction translation table walks in progress per cycle." + }, + { + "ArchStdEvent": "L1D_TLB_RW", + "PublicDescription": "The Event counts L1 Data TLB demand accesses caused by memory Read or Write operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB_RD", + "PublicDescription": "The Event counts L1 Instruction TLB demand accesses whether the access hits or misses in the TLB." + }, + { + "ArchStdEvent": "L1D_TLB_PRFM", + "PublicDescription": "The Event counts L1 Data TLB accesses generated by software prefetch or preload memory accesses. Load or Store Instructions can be broken into multiple memory operations. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB_PRFM", + "PublicDescription": "The Event counts L1 Instruction TLB accesses generated by software preload or prefetch Instructions. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_HWUPD", + "PublicDescription": "The Event counts number of memory accesses triggered by a Data translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that this Event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." + }, + { + "ArchStdEvent": "ITLB_HWUPD", + "PublicDescription": "The Event counts number of memory accesses triggered by an Instruction translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." + }, + { + "ArchStdEvent": "DTLB_STEP", + "PublicDescription": "The Event counts number of memory accesses triggered by a demand Data translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that this Event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." + }, + { + "ArchStdEvent": "ITLB_STEP", + "PublicDescription": "The Event counts number of memory accesses triggered by an Instruction translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." + }, + { + "ArchStdEvent": "DTLB_WALK_LARGE", + "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_BLOCK is implemented, then it is an alias for this Event in this family. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_LARGE", + "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_BLOCK Event. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_SMALL", + "PublicDescription": "The Event counts number of Data translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_PAGE Event is implemented, then it is an alias for this Event in this family. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_SMALL", + "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_PAGE Event. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_RW", + "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_RD", + "PublicDescription": "The Event counts number of demand Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_PRFM", + "PublicDescription": "The Event counts number of software prefetches or preloads generated Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_PRFM", + "PublicDescription": "The Event counts number of software prefetches or preloads generated Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." + }, + { + "EventCode": "0x010e", + "EventName": "L1D_TLB_REFILL_RD_PF", + "PublicDescription": "L1 Data TLB refill, Read, prefetch." + }, + { + "EventCode": "0x010f", + "EventName": "L2TLB_PF_REFILL", + "PublicDescription": "L2 Data TLB refill, Read, prefetch. The Event counts MMU refills due to internal PFStream requests." + }, + { + "EventCode": "0x0223", + "EventName": "L1I_TLB_REFILL_RD", + "PublicDescription": "L1 Instruction TLB refills due to Demand miss." + }, + { + "EventCode": "0x0224", + "EventName": "L1I_TLB_REFILL_PRFM", + "PublicDescription": "L1 Instruction TLB refills due to Software prefetch miss." + } +] From e43be12531de778bb47cb1c5472a54b5cd5d10ee Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Mon, 26 Jan 2026 16:54:54 -0800 Subject: [PATCH 121/464] NVIDIA: VR: SAUCE: [Config] nvidia: enable TEGRA410_C2C_PMU and TEGRA410_CMEM_LATENCY_PMU BugLink: https://bugs.launchpad.net/bugs/2139315 Set the following kconfigs to enable these PMUs on T410: CONFIG_NVIDIA_TEGRA410_C2C_PMU=m CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU=m Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Jacob Martin Acked-by: Abdur Rahman Signed-off-by: Brad Figg (cherry picked from commit ea54294bb42c2d498c848ea147d1e3b4f8bcfb64 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index d4eff215ec49b..4d7e2164e3de0 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -183,6 +183,12 @@ CONFIG_NVGRACE_EGM note<'LP: #2119656'> CONFIG_NVIDIA_FFA_EC policy<{'arm64': 'y'}> CONFIG_NVIDIA_FFA_EC note<'LP: #2114230'> +CONFIG_NVIDIA_TEGRA410_C2C_PMU policy<{'arm64': 'm'}> +CONFIG_NVIDIA_TEGRA410_C2C_PMU note<'LP: #2139315'> + +CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU policy<{'arm64': 'm'}> +CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU note<'LP: #2139315'> + CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> From 8d0498847f49a1f7ed6918dee832f5864804f04a Mon Sep 17 00:00:00 2001 From: Jeremy Szu Date: Tue, 3 Feb 2026 18:49:43 +0800 Subject: [PATCH 122/464] UBUNTU: [Packaging] Enable coresight in Perf if arm64 BugLink: https://bugs.launchpad.net/bugs/2093957 Signed-off-by: Jeremy Szu Acked-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Abdur Rahman Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit a15be6bb7b879f3060e315a69c8ea649d8f3d28d noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian/rules.d/2-binary-arch.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/rules.d/2-binary-arch.mk b/debian/rules.d/2-binary-arch.mk index 7a8bb73afd78b..564923f55af31 100644 --- a/debian/rules.d/2-binary-arch.mk +++ b/debian/rules.d/2-binary-arch.mk @@ -625,7 +625,7 @@ ifeq ($(do_tools_cpupower),true) endif ifeq ($(do_tools_perf),true) cd $(builddirpa)/tools/perf && \ - LLVM_CONFIG=llvm-config-$(LLVM_VERSION) $(kmake) prefix=/usr HAVE_CPLUS_DEMANGLE_SUPPORT=1 CROSS_COMPILE=$(CROSS_COMPILE) NO_LIBPERL=1 WERROR=0 + LLVM_CONFIG=llvm-config-$(LLVM_VERSION) $(kmake) prefix=/usr HAVE_CPLUS_DEMANGLE_SUPPORT=1 CROSS_COMPILE=$(CROSS_COMPILE) NO_LIBPERL=1 WERROR=0 $(if $(filter arm64,$(build_arch)),CORESIGHT=1) endif ifeq ($(do_tools_bpftool),true) $(kmake) CROSS_COMPILE=$(CROSS_COMPILE) -C $(builddirpa)/tools/bpf/bpftool From ae5b639fa64bd219cd7e0556b209899ad53120d8 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 5 Feb 2026 08:26:16 +0000 Subject: [PATCH 123/464] NVIDIA: SAUCE: vfio: Remove vfio_device_from_file() declaration BugLink: https://bugs.launchpad.net/bugs/2138892 Remove this declaration which is now used within the file after merging upstream "vfio/nvgrace-gpu: register device memory for poison handling". Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Signed-off-by: Brad Figg (cherry picked from commit e78ec36bfb5fa7de13a49e2bb588d6ed633eb7c7 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- include/linux/vfio.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 313fca4b5636b..ef02a4996d451 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -335,7 +335,6 @@ static inline bool vfio_file_has_dev(struct file *file, struct vfio_device *devi return false; } #endif -struct vfio_device *vfio_device_from_file(struct file *file); bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); From 92320238adb16536ae81de3bad08cf27d8daccde Mon Sep 17 00:00:00 2001 From: Salman Nabi Date: Mon, 19 Jan 2026 12:27:29 +0000 Subject: [PATCH 124/464] NVIDIA: VR: SAUCE: firmware: smccc: add support for Live Firmware Activation (LFA) BugLink: https://bugs.launchpad.net/bugs/2138342 The Arm Live Firmware Activation (LFA) is a specification [1] to describe activating firmware components without a reboot. Those components (like TF-A's BL31, EDK-II, TF-RMM, secure paylods) would be updated the usual way: via fwupd, FF-A or other secure storage methods, or via some IMPDEF Out-Of-Bound method. The user can then activate this new firmware, at system runtime, without requiring a reboot. The specification covers the SMCCC interface to list and query available components and eventually trigger the activation. Add a new directory under /sys/firmware to present firmware components capable of live activation. Each of them is a directory under lfa/, and is identified via its GUID. The activation will be triggered by echoing "1" into the "activate" file: ========================================== /sys/firmware/lfa # ls -l . 6c* .: total 0 drwxr-xr-x 2 0 0 0 Jan 19 11:33 47d4086d-4cfe-9846-9b95-2950cbbd5a00 drwxr-xr-x 2 0 0 0 Jan 19 11:33 6c0762a6-12f2-4b56-92cb-ba8f633606d9 drwxr-xr-x 2 0 0 0 Jan 19 11:33 d6d0eea7-fcea-d54b-9782-9934f234b6e4 6c0762a6-12f2-4b56-92cb-ba8f633606d9: total 0 --w------- 1 0 0 4096 Jan 19 11:33 activate -r--r--r-- 1 0 0 4096 Jan 19 11:33 activation_capable -r--r--r-- 1 0 0 4096 Jan 19 11:33 activation_pending --w------- 1 0 0 4096 Jan 19 11:33 cancel -r--r--r-- 1 0 0 4096 Jan 19 11:33 cpu_rendezvous -r--r--r-- 1 0 0 4096 Jan 19 11:33 current_version -rw-r--r-- 1 0 0 4096 Jan 19 11:33 force_cpu_rendezvous -r--r--r-- 1 0 0 4096 Jan 19 11:33 may_reset_cpu -r--r--r-- 1 0 0 4096 Jan 19 11:33 name -r--r--r-- 1 0 0 4096 Jan 19 11:33 pending_version /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # grep . * grep: activate: Permission denied activation_capable:1 activation_pending:1 grep: cancel: Permission denied cpu_rendezvous:1 current_version:0.0 force_cpu_rendezvous:1 may_reset_cpu:0 name:TF-RMM pending_version:0.0 /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # echo 1 > activate [ 2825.797871] Arm LFA: firmware activation succeeded. /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # ========================================== [1] https://developer.arm.com/documentation/den0147/latest/ Signed-off-by: Salman Nabi Signed-off-by: Vedashree Vidwans (backported from https://lore.kernel.org/all/20260119122729.287522-2-salman.nabi@arm.com/) [nirmoyd: Added image_name fallback to fw_uuid in update_fw_image_node()] Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 347bebd6528e5f52494904234d9364b6279f7166 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/firmware/smccc/Kconfig | 8 + drivers/firmware/smccc/Makefile | 1 + drivers/firmware/smccc/lfa_fw.c | 670 ++++++++++++++++++++++++++++++++ 3 files changed, 679 insertions(+) create mode 100644 drivers/firmware/smccc/lfa_fw.c diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig index 15e7466179a62..ff7ca49486b05 100644 --- a/drivers/firmware/smccc/Kconfig +++ b/drivers/firmware/smccc/Kconfig @@ -23,3 +23,11 @@ config ARM_SMCCC_SOC_ID help Include support for the SoC bus on the ARM SMCCC firmware based platforms providing some sysfs information about the SoC variant. + +config ARM_LFA + tristate "Arm Live Firmware activation support" + depends on HAVE_ARM_SMCCC_DISCOVERY + default y + help + Include support for triggering Live Firmware Activation, which + allows to upgrade certain firmware components without a reboot. diff --git a/drivers/firmware/smccc/Makefile b/drivers/firmware/smccc/Makefile index 40d19144a8607..a6dd01558a94a 100644 --- a/drivers/firmware/smccc/Makefile +++ b/drivers/firmware/smccc/Makefile @@ -2,3 +2,4 @@ # obj-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smccc.o kvm_guest.o obj-$(CONFIG_ARM_SMCCC_SOC_ID) += soc_id.o +obj-$(CONFIG_ARM_LFA) += lfa_fw.o diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c new file mode 100644 index 0000000000000..adf4171066437 --- /dev/null +++ b/drivers/firmware/smccc/lfa_fw.c @@ -0,0 +1,670 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 Arm Limited + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#undef pr_fmt +#define pr_fmt(fmt) "Arm LFA: " fmt + +/* LFA v1.0b0 specification */ +#define LFA_1_0_FN_BASE 0xc40002e0 +#define LFA_1_0_FN(n) (LFA_1_0_FN_BASE + (n)) + +#define LFA_1_0_FN_GET_VERSION LFA_1_0_FN(0) +#define LFA_1_0_FN_CHECK_FEATURE LFA_1_0_FN(1) +#define LFA_1_0_FN_GET_INFO LFA_1_0_FN(2) +#define LFA_1_0_FN_GET_INVENTORY LFA_1_0_FN(3) +#define LFA_1_0_FN_PRIME LFA_1_0_FN(4) +#define LFA_1_0_FN_ACTIVATE LFA_1_0_FN(5) +#define LFA_1_0_FN_CANCEL LFA_1_0_FN(6) + +/* CALL_AGAIN flags (returned by SMC) */ +#define LFA_PRIME_CALL_AGAIN BIT(0) +#define LFA_ACTIVATE_CALL_AGAIN BIT(0) + +/* LFA return values */ +#define LFA_SUCCESS 0 +#define LFA_NOT_SUPPORTED 1 +#define LFA_BUSY 2 +#define LFA_AUTH_ERROR 3 +#define LFA_NO_MEMORY 4 +#define LFA_CRITICAL_ERROR 5 +#define LFA_DEVICE_ERROR 6 +#define LFA_WRONG_STATE 7 +#define LFA_INVALID_PARAMETERS 8 +#define LFA_COMPONENT_WRONG_STATE 9 +#define LFA_INVALID_ADDRESS 10 +#define LFA_ACTIVATION_FAILED 11 + +#define LFA_ERROR_STRING(name) \ + [name] = #name + +static const char * const lfa_error_strings[] = { + LFA_ERROR_STRING(LFA_SUCCESS), + LFA_ERROR_STRING(LFA_NOT_SUPPORTED), + LFA_ERROR_STRING(LFA_BUSY), + LFA_ERROR_STRING(LFA_AUTH_ERROR), + LFA_ERROR_STRING(LFA_NO_MEMORY), + LFA_ERROR_STRING(LFA_CRITICAL_ERROR), + LFA_ERROR_STRING(LFA_DEVICE_ERROR), + LFA_ERROR_STRING(LFA_WRONG_STATE), + LFA_ERROR_STRING(LFA_INVALID_PARAMETERS), + LFA_ERROR_STRING(LFA_COMPONENT_WRONG_STATE), + LFA_ERROR_STRING(LFA_INVALID_ADDRESS), + LFA_ERROR_STRING(LFA_ACTIVATION_FAILED) +}; + +enum image_attr_names { + LFA_ATTR_NAME, + LFA_ATTR_CURRENT_VERSION, + LFA_ATTR_PENDING_VERSION, + LFA_ATTR_ACT_CAPABLE, + LFA_ATTR_ACT_PENDING, + LFA_ATTR_MAY_RESET_CPU, + LFA_ATTR_CPU_RENDEZVOUS, + LFA_ATTR_FORCE_CPU_RENDEZVOUS, + LFA_ATTR_ACTIVATE, + LFA_ATTR_CANCEL, + LFA_ATTR_NR_IMAGES +}; + +struct image_props { + struct list_head image_node; + const char *image_name; + int fw_seq_id; + u64 current_version; + u64 pending_version; + bool activation_capable; + bool activation_pending; + bool may_reset_cpu; + bool cpu_rendezvous; + bool cpu_rendezvous_forced; + struct kobject *image_dir; + struct kobj_attribute image_attrs[LFA_ATTR_NR_IMAGES]; +}; +static LIST_HEAD(lfa_fw_images); + +/* A UUID split over two 64-bit registers */ +struct uuid_regs { + u64 uuid_lo; + u64 uuid_hi; +}; + +static const struct fw_image_uuid { + const char *name; + const char *uuid; +} fw_images_uuids[] = { + { + .name = "TF-A BL31 runtime", + .uuid = "47d4086d-4cfe-9846-9b95-2950cbbd5a00", + }, + { + .name = "BL33 non-secure payload", + .uuid = "d6d0eea7-fcea-d54b-9782-9934f234b6e4", + }, + { + .name = "TF-RMM", + .uuid = "6c0762a6-12f2-4b56-92cb-ba8f633606d9", + }, +}; + +static struct kobject *lfa_dir; +static DEFINE_MUTEX(lfa_lock); +static struct workqueue_struct *fw_images_update_wq; +static struct work_struct fw_images_update_work; + +static int update_fw_images_tree(void); + +static void delete_fw_image_node(struct image_props *attrs) +{ + int i; + + for (i = 0; i < LFA_ATTR_NR_IMAGES; i++) + sysfs_remove_file(attrs->image_dir, &attrs->image_attrs[i].attr); + + kobject_put(attrs->image_dir); + list_del(&attrs->image_node); + kfree(attrs); +} + +static void remove_invalid_fw_images(struct work_struct *work) +{ + struct image_props *attrs, *tmp; + + mutex_lock(&lfa_lock); + + /* + * Remove firmware images including directories that are no longer + * present in the LFA agent after updating the existing ones. + */ + list_for_each_entry_safe(attrs, tmp, &lfa_fw_images, image_node) { + if (attrs->fw_seq_id == -1) + delete_fw_image_node(attrs); + } + + mutex_unlock(&lfa_lock); +} + +static void set_image_flags(struct image_props *attrs, int seq_id, + u32 image_flags, u64 reg_current_ver, + u64 reg_pending_ver) +{ + attrs->fw_seq_id = seq_id; + attrs->current_version = reg_current_ver; + attrs->pending_version = reg_pending_ver; + attrs->activation_capable = !!(image_flags & BIT(0)); + attrs->activation_pending = !!(image_flags & BIT(1)); + attrs->may_reset_cpu = !!(image_flags & BIT(2)); + /* cpu_rendezvous_optional bit has inverse logic in the spec */ + attrs->cpu_rendezvous = !(image_flags & BIT(3)); +} + +static unsigned long get_nr_lfa_components(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_GET_INFO; + reg.a1 = 0; /* lfa_info_selector = 0 */ + + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 != LFA_SUCCESS) + return reg.a0; + + return reg.a1; +} + +static int lfa_cancel(void *data) +{ + struct image_props *attrs = data; + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_CANCEL; + reg.a1 = attrs->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + + /* + * When firmware activation is called with "skip_cpu_rendezvous=1", + * LFA_CANCEL can fail with LFA_BUSY if the activation could not be + * cancelled. + */ + if (reg.a0 == LFA_SUCCESS) { + pr_info("Activation cancelled for image %s\n", + attrs->image_name); + } else { + pr_err("Firmware activation could not be cancelled: %s\n", + lfa_error_strings[-reg.a0]); + return -EINVAL; + } + + return reg.a0; +} + +static int call_lfa_activate(void *data) +{ + struct image_props *attrs = data; + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_ACTIVATE; + reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + /* + * As we do not support updates requiring a CPU reset (yet), + * we pass 0 in reg.a3 and reg.a4, holding the entry point and context + * ID respectively. + * cpu_rendezvous_forced is set by the administrator, via sysfs, + * cpu_rendezvous is dictated by each firmware component. + */ + reg.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); + + for (;;) { + arm_smccc_1_2_invoke(®, ®); + + if ((long)reg.a0 < 0) { + pr_err("ACTIVATE for image %s failed: %s\n", + attrs->image_name, lfa_error_strings[-reg.a0]); + return reg.a0; + } + if (!(reg.a1 & LFA_ACTIVATE_CALL_AGAIN)) + break; /* ACTIVATE successful */ + } + + return reg.a0; +} + +static int activate_fw_image(struct image_props *attrs) +{ + int ret; + + mutex_lock(&lfa_lock); + if (attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous) + ret = stop_machine(call_lfa_activate, attrs, cpu_online_mask); + else + ret = call_lfa_activate(attrs); + + if (ret != 0) { + mutex_unlock(&lfa_lock); + return lfa_cancel(attrs); + } + + /* + * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the + * number of firmware images in the LFA agent may change after a + * successful activation attempt. Negate all image flags as well. + */ + attrs = NULL; + list_for_each_entry(attrs, &lfa_fw_images, image_node) { + set_image_flags(attrs, -1, 0b1000, 0, 0); + } + + update_fw_images_tree(); + + /* + * Removing non-valid image directories at the end of an activation. + * We can't remove the sysfs attributes while in the respective + * _store() handler, so have to postpone the list removal to a + * workqueue. + */ + INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); + queue_work(fw_images_update_wq, &fw_images_update_work); + mutex_unlock(&lfa_lock); + + return ret; +} + +static int prime_fw_image(struct image_props *attrs) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + int ret; + + mutex_lock(&lfa_lock); + /* Avoid SMC calls on invalid firmware images */ + if (attrs->fw_seq_id == -1) { + pr_err("Arm LFA: Invalid firmware sequence id\n"); + mutex_unlock(&lfa_lock); + + return -ENODEV; + } + + if (attrs->may_reset_cpu) { + pr_err("CPU reset not supported by kernel driver\n"); + mutex_unlock(&lfa_lock); + + return -EINVAL; + } + + /* + * LFA_PRIME/ACTIVATE will return 1 in reg.a1 if the firmware + * priming/activation is still in progress. In that case + * LFA_PRIME/ACTIVATE will need to be called again. + * reg.a1 will become 0 once the prime/activate process completes. + */ + reg.a0 = LFA_1_0_FN_PRIME; + reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + for (;;) { + arm_smccc_1_2_invoke(®, ®); + + if ((long)reg.a0 < 0) { + pr_err("LFA_PRIME for image %s failed: %s\n", + attrs->image_name, lfa_error_strings[-reg.a0]); + mutex_unlock(&lfa_lock); + + return reg.a0; + } + if (!(reg.a1 & LFA_PRIME_CALL_AGAIN)) { + ret = 0; + break; /* PRIME successful */ + } + } + + mutex_unlock(&lfa_lock); + return ret; +} + +static ssize_t name_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_NAME]); + + return sysfs_emit(buf, "%s\n", attrs->image_name); +} + +static ssize_t activation_capable_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_ACT_CAPABLE]); + + return sysfs_emit(buf, "%d\n", attrs->activation_capable); +} + +static ssize_t activation_pending_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_ACT_PENDING]); + struct arm_smccc_1_2_regs reg = { 0 }; + + /* + * Activation pending status can change anytime thus we need to update + * and return its current value + */ + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + reg.a1 = attrs->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == LFA_SUCCESS) + attrs->activation_pending = !!(reg.a3 & BIT(1)); + + return sysfs_emit(buf, "%d\n", attrs->activation_pending); +} + +static ssize_t may_reset_cpu_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_MAY_RESET_CPU]); + + return sysfs_emit(buf, "%d\n", attrs->may_reset_cpu); +} + +static ssize_t cpu_rendezvous_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_CPU_RENDEZVOUS]); + + return sysfs_emit(buf, "%d\n", attrs->cpu_rendezvous); +} + +static ssize_t force_cpu_rendezvous_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_FORCE_CPU_RENDEZVOUS]); + int ret; + + ret = kstrtobool(buf, &attrs->cpu_rendezvous_forced); + if (ret) + return ret; + + return count; +} + +static ssize_t force_cpu_rendezvous_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_FORCE_CPU_RENDEZVOUS]); + + return sysfs_emit(buf, "%d\n", attrs->cpu_rendezvous_forced); +} + +static ssize_t current_version_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_CURRENT_VERSION]); + u32 maj, min; + + maj = attrs->current_version >> 32; + min = attrs->current_version & 0xffffffff; + return sysfs_emit(buf, "%u.%u\n", maj, min); +} + +static ssize_t pending_version_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_ACT_PENDING]); + struct arm_smccc_1_2_regs reg = { 0 }; + u32 maj, min; + + /* + * Similar to activation pending, this value can change following an + * update, we need to retrieve fresh info instead of stale information. + */ + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + reg.a1 = attrs->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == LFA_SUCCESS) { + if (reg.a5 != 0 && attrs->activation_pending) + { + attrs->pending_version = reg.a5; + maj = reg.a5 >> 32; + min = reg.a5 & 0xffffffff; + } + } + + return sysfs_emit(buf, "%u.%u\n", maj, min); +} + +static ssize_t activate_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_ACTIVATE]); + int ret; + + ret = prime_fw_image(attrs); + if (ret) { + pr_err("Firmware prime failed: %s\n", + lfa_error_strings[-ret]); + return -ECANCELED; + } + + ret = activate_fw_image(attrs); + if (ret) { + pr_err("Firmware activation failed: %s\n", + lfa_error_strings[-ret]); + return -ECANCELED; + } + + pr_info("Firmware activation succeeded\n"); + + return count; +} + +static ssize_t cancel_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct image_props *attrs = container_of(attr, struct image_props, + image_attrs[LFA_ATTR_CANCEL]); + int ret; + + ret = lfa_cancel(attrs); + if (ret != 0) + return ret; + + return count; +} + +static struct kobj_attribute image_attrs_group[LFA_ATTR_NR_IMAGES] = { + [LFA_ATTR_NAME] = __ATTR_RO(name), + [LFA_ATTR_CURRENT_VERSION] = __ATTR_RO(current_version), + [LFA_ATTR_PENDING_VERSION] = __ATTR_RO(pending_version), + [LFA_ATTR_ACT_CAPABLE] = __ATTR_RO(activation_capable), + [LFA_ATTR_ACT_PENDING] = __ATTR_RO(activation_pending), + [LFA_ATTR_MAY_RESET_CPU] = __ATTR_RO(may_reset_cpu), + [LFA_ATTR_CPU_RENDEZVOUS] = __ATTR_RO(cpu_rendezvous), + [LFA_ATTR_FORCE_CPU_RENDEZVOUS] = __ATTR_RW(force_cpu_rendezvous), + [LFA_ATTR_ACTIVATE] = __ATTR_WO(activate), + [LFA_ATTR_CANCEL] = __ATTR_WO(cancel) +}; + +static void clean_fw_images_tree(void) +{ + struct image_props *attrs, *tmp; + + list_for_each_entry_safe(attrs, tmp, &lfa_fw_images, image_node) + delete_fw_image_node(attrs); +} + +static int update_fw_image_node(char *fw_uuid, int seq_id, + u32 image_flags, u64 reg_current_ver, + u64 reg_pending_ver) +{ + const char *image_name = "(unknown)"; + struct image_props *attrs; + int ret; + + /* + * If a fw_image is already in the images list then we just update + * its flags and seq_id instead of trying to recreate it. + */ + list_for_each_entry(attrs, &lfa_fw_images, image_node) { + if (!strcmp(attrs->image_dir->name, fw_uuid)) { + set_image_flags(attrs, seq_id, image_flags, + reg_current_ver, reg_pending_ver); + return 0; + } + } + + attrs = kzalloc(sizeof(*attrs), GFP_KERNEL); + if (!attrs) + return -ENOMEM; + + for (int i = 0; i < ARRAY_SIZE(fw_images_uuids); i++) { + if (!strcmp(fw_images_uuids[i].uuid, fw_uuid)) + image_name = fw_images_uuids[i].name; + else + image_name = fw_uuid; + } + + attrs->image_dir = kobject_create_and_add(fw_uuid, lfa_dir); + if (!attrs->image_dir) + return -ENOMEM; + + INIT_LIST_HEAD(&attrs->image_node); + attrs->image_name = image_name; + attrs->cpu_rendezvous_forced = 1; + set_image_flags(attrs, seq_id, image_flags, reg_current_ver, + reg_pending_ver); + + /* + * The attributes for each sysfs file are constant (handler functions, + * name and permissions are the same within each directory), but we + * need a per-directory copy regardless, to get a unique handle + * for each directory, so that container_of can do its magic. + * Also this requires an explicit sysfs_attr_init(), since it's a new + * copy, to make LOCKDEP happy. + */ + memcpy(attrs->image_attrs, image_attrs_group, + sizeof(attrs->image_attrs)); + for (int i = 0; i < LFA_ATTR_NR_IMAGES; i++) { + struct attribute *attr = &attrs->image_attrs[i].attr; + + sysfs_attr_init(attr); + ret = sysfs_create_file(attrs->image_dir, attr); + if (ret) { + pr_err("creating sysfs file for uuid %s: %d\n", + fw_uuid, ret); + clean_fw_images_tree(); + + return ret; + } + } + list_add(&attrs->image_node, &lfa_fw_images); + + return ret; +} + +static int update_fw_images_tree(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + struct uuid_regs image_uuid; + char image_id_str[40]; + int ret, num_of_components; + + num_of_components = get_nr_lfa_components(); + if (num_of_components <= 0) { + pr_err("Error getting number of LFA components\n"); + return -ENODEV; + } + + for (int i = 0; i < num_of_components; i++) { + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + reg.a1 = i; /* fw_seq_id under consideration */ + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == LFA_SUCCESS) { + image_uuid.uuid_lo = reg.a1; + image_uuid.uuid_hi = reg.a2; + + snprintf(image_id_str, sizeof(image_id_str), "%pUb", + &image_uuid); + ret = update_fw_image_node(image_id_str, i, + reg.a3, reg.a4, reg.a5); + if (ret) + return ret; + } + } + + return 0; +} + +static int __init lfa_init(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + int err; + + reg.a0 = LFA_1_0_FN_GET_VERSION; + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == -LFA_NOT_SUPPORTED) { + pr_info("Live Firmware activation: no firmware agent found\n"); + return -ENODEV; + } + + fw_images_update_wq = alloc_workqueue("fw_images_update_wq", + WQ_UNBOUND | WQ_MEM_RECLAIM, 1); + if (!fw_images_update_wq) { + pr_err("Live Firmware Activation: Failed to allocate workqueue.\n"); + + return -ENOMEM; + } + + pr_info("Live Firmware Activation: detected v%ld.%ld\n", + reg.a0 >> 16, reg.a0 & 0xffff); + + lfa_dir = kobject_create_and_add("lfa", firmware_kobj); + if (!lfa_dir) + return -ENOMEM; + + mutex_lock(&lfa_lock); + err = update_fw_images_tree(); + if (err != 0) + kobject_put(lfa_dir); + + mutex_unlock(&lfa_lock); + return err; +} +module_init(lfa_init); + +static void __exit lfa_exit(void) +{ + flush_workqueue(fw_images_update_wq); + destroy_workqueue(fw_images_update_wq); + + mutex_lock(&lfa_lock); + clean_fw_images_tree(); + mutex_unlock(&lfa_lock); + + kobject_put(lfa_dir); +} +module_exit(lfa_exit); + +MODULE_DESCRIPTION("ARM Live Firmware Activation (LFA)"); +MODULE_LICENSE("GPL"); From c074c391c3cb94a914a1b758010ed26353502cf7 Mon Sep 17 00:00:00 2001 From: Vedashree Vidwans Date: Tue, 27 Jan 2026 20:34:09 +0000 Subject: [PATCH 125/464] NVIDIA: VR: SAUCE: firmware: smccc: add timeout, touch wdt BugLink: https://bugs.launchpad.net/bugs/2138342 Enhance PRIME/ACTIVATION functions to touch watchdog and implement timeout mechanism. This update ensures that any potential hangs are detected promptly and that the LFA process is allocated sufficient execution time before the watchdog timer expires. These changes improve overall system reliability by reducing the risk of undetected process stalls and unexpected watchdog resets. Signed-off-by: Vedashree Vidwans Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit ec83357fdf8fa5e5afec4bdfad9d350a39e229ff noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/firmware/smccc/lfa_fw.c | 82 ++++++++++++++++++++++++--------- 1 file changed, 61 insertions(+), 21 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index adf4171066437..ed439daf49319 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -17,6 +17,9 @@ #include #include #include +#include +#include +#include #undef pr_fmt #define pr_fmt(fmt) "Arm LFA: " fmt @@ -37,6 +40,14 @@ #define LFA_PRIME_CALL_AGAIN BIT(0) #define LFA_ACTIVATE_CALL_AGAIN BIT(0) +/* Prime loop limits, TODO: tune after testing */ +#define LFA_PRIME_BUDGET_US 30000000 /* 30s cap */ +#define LFA_PRIME_POLL_DELAY_US 10 /* 10us between polls */ + +/* Activation loop limits, TODO: tune after testing */ +#define LFA_ACTIVATE_BUDGET_US 20000000 /* 20s cap */ +#define LFA_ACTIVATE_POLL_DELAY_US 10 /* 10us between polls */ + /* LFA return values */ #define LFA_SUCCESS 0 #define LFA_NOT_SUPPORTED 1 @@ -217,10 +228,12 @@ static int lfa_cancel(void *data) static int call_lfa_activate(void *data) { struct image_props *attrs = data; - struct arm_smccc_1_2_regs reg = { 0 }; + struct arm_smccc_1_2_regs args = { 0 }; + struct arm_smccc_1_2_regs res = { 0 }; + ktime_t end = ktime_add_us(ktime_get(), LFA_ACTIVATE_BUDGET_US); - reg.a0 = LFA_1_0_FN_ACTIVATE; - reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + args.a0 = LFA_1_0_FN_ACTIVATE; + args.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ /* * As we do not support updates requiring a CPU reset (yet), * we pass 0 in reg.a3 and reg.a4, holding the entry point and context @@ -228,21 +241,32 @@ static int call_lfa_activate(void *data) * cpu_rendezvous_forced is set by the administrator, via sysfs, * cpu_rendezvous is dictated by each firmware component. */ - reg.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); + args.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); for (;;) { - arm_smccc_1_2_invoke(®, ®); + /* Touch watchdog, ACTIVATE shouldn't take longer than watchdog_thresh */ + touch_nmi_watchdog(); + arm_smccc_1_2_invoke(&args, &res); - if ((long)reg.a0 < 0) { + if ((long)res.a0 < 0) { pr_err("ACTIVATE for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-reg.a0]); - return reg.a0; + attrs->image_name, lfa_error_strings[-res.a0]); + return res.a0; } - if (!(reg.a1 & LFA_ACTIVATE_CALL_AGAIN)) + if (!(res.a1 & LFA_ACTIVATE_CALL_AGAIN)) break; /* ACTIVATE successful */ + + /* SMC returned with call_again flag set */ + if (ktime_before(ktime_get(), end)) { + udelay(LFA_ACTIVATE_POLL_DELAY_US); + continue; + } + + pr_err("ACTIVATE for image %s timed out", attrs->image_name); + return -ETIMEDOUT; } - return reg.a0; + return res.a0; } static int activate_fw_image(struct image_props *attrs) @@ -287,7 +311,9 @@ static int activate_fw_image(struct image_props *attrs) static int prime_fw_image(struct image_props *attrs) { - struct arm_smccc_1_2_regs reg = { 0 }; + struct arm_smccc_1_2_regs args = { 0 }; + struct arm_smccc_1_2_regs res = { 0 }; + ktime_t end = ktime_add_us(ktime_get(), LFA_PRIME_BUDGET_US); int ret; mutex_lock(&lfa_lock); @@ -307,27 +333,41 @@ static int prime_fw_image(struct image_props *attrs) } /* - * LFA_PRIME/ACTIVATE will return 1 in reg.a1 if the firmware + * LFA_PRIME/ACTIVATE will return 1 in res.a1 if the firmware * priming/activation is still in progress. In that case * LFA_PRIME/ACTIVATE will need to be called again. - * reg.a1 will become 0 once the prime/activate process completes. + * res.a1 will become 0 once the prime/activate process completes. */ - reg.a0 = LFA_1_0_FN_PRIME; - reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + args.a0 = LFA_1_0_FN_PRIME; + args.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ for (;;) { - arm_smccc_1_2_invoke(®, ®); + /* Touch watchdog, PRIME shouldn't take longer than watchdog_thresh */ + touch_nmi_watchdog(); + arm_smccc_1_2_invoke(&args, &res); - if ((long)reg.a0 < 0) { + if ((long)res.a0 < 0) { pr_err("LFA_PRIME for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-reg.a0]); + attrs->image_name, lfa_error_strings[-res.a0]); mutex_unlock(&lfa_lock); - return reg.a0; + return res.a0; } - if (!(reg.a1 & LFA_PRIME_CALL_AGAIN)) { - ret = 0; + if (!(res.a1 & LFA_PRIME_CALL_AGAIN)) break; /* PRIME successful */ + + /* SMC returned with call_again flag set */ + if (ktime_before(ktime_get(), end)) { + udelay(LFA_PRIME_POLL_DELAY_US); + continue; } + + pr_err("LFA_PRIME for image %s timed out", attrs->image_name); + mutex_unlock(&lfa_lock); + + ret = lfa_cancel(attrs); + if (ret != 0) + return ret; + return -ETIMEDOUT; } mutex_unlock(&lfa_lock); From 6788bd15638d924879ec34b059d0ae0628b565ba Mon Sep 17 00:00:00 2001 From: Vedashree Vidwans Date: Wed, 28 Jan 2026 07:39:44 +0000 Subject: [PATCH 126/464] NVIDIA: VR: SAUCE: firmware: smccc: register as platform driver BugLink: https://bugs.launchpad.net/bugs/2138342 - Register the LFA driver as a platform driver corresponding to 'arml0003' ACPI device. The driver will be invoked when the device is detected on a platform. NOTE: current functionality only available for ACPI configuration. - Add functionality to register ACPI notify handler for LFA in the driver probe(). - When notify handler is invoked, driver will query latest FW component details and trigger activation of capable and pending FW component in a loop until all FWs are activated. ACPI node snippet from LFA spec[1]: Device (LFA0) { Name (_HID, "ARML0003") Name (_UID, 0) } [1] https://developer.arm.com/documentation/den0147/latest/ Signed-off-by: Vedashree Vidwans Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 9ebe11f823ed8e18d6520b10be49bf2bd0c544a1 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/firmware/smccc/lfa_fw.c | 153 ++++++++++++++++++++++++++++---- 1 file changed, 134 insertions(+), 19 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index ed439daf49319..90727a66e49a5 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -20,7 +20,10 @@ #include #include #include +#include +#include +#define DRIVER_NAME "ARM_LFA" #undef pr_fmt #define pr_fmt(fmt) "Arm LFA: " fmt @@ -284,26 +287,7 @@ static int activate_fw_image(struct image_props *attrs) return lfa_cancel(attrs); } - /* - * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the - * number of firmware images in the LFA agent may change after a - * successful activation attempt. Negate all image flags as well. - */ - attrs = NULL; - list_for_each_entry(attrs, &lfa_fw_images, image_node) { - set_image_flags(attrs, -1, 0b1000, 0, 0); - } - update_fw_images_tree(); - - /* - * Removing non-valid image directories at the end of an activation. - * We can't remove the sysfs attributes while in the respective - * _store() handler, so have to postpone the list removal to a - * workqueue. - */ - INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); - queue_work(fw_images_update_wq, &fw_images_update_work); mutex_unlock(&lfa_lock); return ret; @@ -627,6 +611,7 @@ static int update_fw_images_tree(void) { struct arm_smccc_1_2_regs reg = { 0 }; struct uuid_regs image_uuid; + struct image_props *attrs; char image_id_str[40]; int ret, num_of_components; @@ -636,6 +621,15 @@ static int update_fw_images_tree(void) return -ENODEV; } + /* + * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the + * number of firmware images in the LFA agent may change after a + * successful activation attempt. Negate all image flags as well. + */ + list_for_each_entry(attrs, &lfa_fw_images, image_node) { + set_image_flags(attrs, -1, 0b1000, 0, 0); + } + for (int i = 0; i < num_of_components; i++) { reg.a0 = LFA_1_0_FN_GET_INVENTORY; reg.a1 = i; /* fw_seq_id under consideration */ @@ -653,9 +647,121 @@ static int update_fw_images_tree(void) } } + /* + * Removing non-valid image directories at the end of an activation. + * We can't remove the sysfs attributes while in the respective + * _store() handler, so have to postpone the list removal to a + * workqueue. + */ + INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); + queue_work(fw_images_update_wq, &fw_images_update_work); + + return 0; +} + +#if defined(CONFIG_ACPI) +static void lfa_notify_handler(acpi_handle handle, u32 event, void *data) +{ + struct image_props *attrs = NULL; + int ret; + bool found_activable_image = false; + + /* Get latest FW inventory */ + mutex_lock(&lfa_lock); + ret = update_fw_images_tree(); + mutex_unlock(&lfa_lock); + if (ret != 0) { + pr_err("FW images tree update failed"); + return; + } + + /* + * Go through all FW images in a loop and trigger activation + * of all activable and pending images. + */ + do { + /* Reset activable image flag */ + found_activable_image = false; + list_for_each_entry(attrs, &lfa_fw_images, image_node) { + if (attrs->fw_seq_id == -1) + continue; /* Invalid FW component */ + + if ((!attrs->activation_capable) || (!attrs->activation_pending)) + continue; /* FW component is not activable */ + + /* + * Found an image that is activable. + * As the FW images tree is revised after activation, it is + * not ideal to invoke activation from inside + * list_for_each_entry() loop. + * So, set the flasg and exit loop. + */ + found_activable_image = true; + break; + } + + if (found_activable_image) { + ret = prime_fw_image(attrs); + if (ret) { + pr_err("Firmware prime failed: %s\n", + lfa_error_strings[-ret]); + return; + } + + ret = activate_fw_image(attrs); + if (ret) { + pr_err("Firmware activation failed: %s\n", + lfa_error_strings[-ret]); + return; + } + + pr_info("Firmware %s activation succeeded", attrs->image_name); + } + } while(found_activable_image); + + return; +} + +static int lfa_probe(struct platform_device *pdev) +{ + acpi_status status; + acpi_handle handle = ACPI_HANDLE(&pdev->dev); + if (!handle) + return -ENODEV; + + /* Register notify handler that indicates if LFA updates are available */ + status = acpi_install_notify_handler(handle, + ACPI_DEVICE_NOTIFY, lfa_notify_handler, pdev); + if (ACPI_FAILURE(status)) + return -EIO; + return 0; } +static void lfa_remove(struct platform_device *pdev) { + acpi_handle handle = ACPI_HANDLE(&pdev->dev); + + if (handle) + acpi_remove_notify_handler(handle, + ACPI_DEVICE_NOTIFY, lfa_notify_handler); +} + +static const struct acpi_device_id lfa_acpi_ids[] = { + {"ARML0003"}, + {}, +}; +MODULE_DEVICE_TABLE(acpi, lfa_acpi_ids); + +static struct platform_driver lfa_driver = { + .probe = lfa_probe, + .remove = lfa_remove, + .driver = { + .name = DRIVER_NAME, + .acpi_match_table = ACPI_PTR(lfa_acpi_ids), + }, +}; +#endif + static int __init lfa_init(void) { struct arm_smccc_1_2_regs reg = { 0 }; @@ -679,6 +785,12 @@ static int __init lfa_init(void) pr_info("Live Firmware Activation: detected v%ld.%ld\n", reg.a0 >> 16, reg.a0 & 0xffff); +#if defined(CONFIG_ACPI) + err = platform_driver_register(&lfa_driver); + if (err < 0) + pr_err("Platform driver register failed"); +#endif + lfa_dir = kobject_create_and_add("lfa", firmware_kobj); if (!lfa_dir) return -ENOMEM; @@ -703,6 +815,9 @@ static void __exit lfa_exit(void) mutex_unlock(&lfa_lock); kobject_put(lfa_dir); +#if defined(CONFIG_ACPI) + platform_driver_unregister(&lfa_driver); +#endif } module_exit(lfa_exit); From 637ae3e43f0abc3729847a1975097dc0888cfd15 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Tue, 13 Jan 2026 13:34:51 -0800 Subject: [PATCH 127/464] NVIDIA: VR: SAUCE: [Config] nvidia: Enable ARM LFA support BugLink: https://bugs.launchpad.net/bugs/2138342 Enable Arm Live Firmware Activation support by setting CONFIG_ARM_LFA=y. Signed-off-by: Jamie Nguyen Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Jacob Martin Acked-by: Noah Wager Signed-off-by: Brad Figg (cherry picked from commit 51a34dd0245af0652cf2e4d6adbbe588ff06cca7 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 4d7e2164e3de0..db41b44b7b64e 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -39,6 +39,9 @@ CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablem CONFIG_ARM_FFA_TRANSPORT policy<{'arm64': 'y'}> CONFIG_ARM_FFA_TRANSPORT note<'LP: #2111511'> +CONFIG_ARM_LFA policy<{'arm64': 'y'}> +CONFIG_ARM_LFA note<'LP: #2138342'> + CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> From b5698e708fb66a752df6f205b5f3239c306de02b Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Fri, 13 Feb 2026 16:27:35 -0600 Subject: [PATCH 128/464] UBUNTU: [Packaging] Depend on 580 NVIDIA graphics driver components explicitly The virtual nvidia-kernel-source and nvidia-dkms-kernel dependencies would sometimes pull the 470 driver, which is incompatible with the nvidia-fs build. Stick to the latest LTS. This could be made to use the virtual packages again once the 470 driver transitionals are released. Ignore: yes Signed-off-by: Jacob Martin (cherry picked from commit 73a27bbf7e984cd94f37ca5f9d414f530755a78c noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/control.stub.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/debian.nvidia-bos/control.stub.in b/debian.nvidia-bos/control.stub.in index 6b9adaf973954..4c38024864f0a 100644 --- a/debian.nvidia-bos/control.stub.in +++ b/debian.nvidia-bos/control.stub.in @@ -53,8 +53,8 @@ Build-Depends: uuid-dev , zstd , bpftool:native [amd64 arm64] , - nvidia-dkms-kernel [amd64 arm64] , - nvidia-kernel-source [amd64 arm64] , + nvidia-dkms-580-open [amd64 arm64] , + nvidia-kernel-source-580-open [amd64 arm64] , Build-Depends-Indep: asciidoc , bzip2 , From 18f162428e8f0a78ea1677e228631e5fec1e3e96 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Sat, 14 Feb 2026 07:56:30 -0600 Subject: [PATCH 129/464] UBUNTU: [Packaging] Add libopencsd-dev as a build dependency BugLink: https://bugs.launchpad.net/bugs/2093957 The patch "UBUNTU: [Packaging] Enable coresight in Perf if arm64" enables perf to be built with CORESIGHT=1 on arm64. This requires libopencsd. Signed-off-by: Jacob Martin (cherry picked from commit f95fa30baf72c7446f22c2e7c53883a9d2fc85fe noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- debian.nvidia-bos/control.stub.in | 1 + 1 file changed, 1 insertion(+) diff --git a/debian.nvidia-bos/control.stub.in b/debian.nvidia-bos/control.stub.in index 4c38024864f0a..2d57a649be27c 100644 --- a/debian.nvidia-bos/control.stub.in +++ b/debian.nvidia-bos/control.stub.in @@ -55,6 +55,7 @@ Build-Depends: bpftool:native [amd64 arm64] , nvidia-dkms-580-open [amd64 arm64] , nvidia-kernel-source-580-open [amd64 arm64] , + libopencsd-dev [arm64] , Build-Depends-Indep: asciidoc , bzip2 , From d6573dee229003f626305d90d79b2a4be60e8989 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Fri, 13 Feb 2026 09:32:59 -0800 Subject: [PATCH 130/464] NVIDIA: SAUCE: r8127: fix NAPI warning on module removal BugLink: https://bugs.launchpad.net/bugs/2141780 When the r8127 module is unloaded, __netif_napi_del_locked() can trigger a WARN because NAPI is removed while still enabled. unregister_netdev() calls ndo_stop, which disables NAPI; deleting NAPI before that runs violates the netdev/NAPI teardown order. Move rtl8127_del_napi() to after unregister_netdev() so NAPI is disabled in ndo_stop before it is removed. Aligns with the upstream r8169 fix in commit 12b1bc75cd46 ("r8169: improve rtl_remove_one"). Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Noah Wager Acked-by: Jacob Martin Signed-off-by: Brad Figg (cherry picked from commit cda2af9cd9d2b45f09e9a615e4354b4e236e3909 noble:linux-nvidia-6.17) Signed-off-by: Jacob Martin --- drivers/net/ethernet/realtek/r8127/r8127_n.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/realtek/r8127/r8127_n.c b/drivers/net/ethernet/realtek/r8127/r8127_n.c index 9e39016ea2c80..03d99ae8d3853 100755 --- a/drivers/net/ethernet/realtek/r8127/r8127_n.c +++ b/drivers/net/ethernet/realtek/r8127/r8127_n.c @@ -14334,9 +14334,6 @@ rtl8127_remove_one(struct pci_dev *pdev) rtl8127_cancel_all_schedule_work(tp); -#ifdef CONFIG_R8127_NAPI - rtl8127_del_napi(tp); -#endif if (HW_DASH_SUPPORT_DASH(tp)) rtl8127_driver_stop(tp); @@ -14347,6 +14344,9 @@ rtl8127_remove_one(struct pci_dev *pdev) #endif //ENABLE_R8127_SYSFS unregister_netdev(dev); +#ifdef CONFIG_R8127_NAPI + rtl8127_del_napi(tp); +#endif rtl8127_disable_msi(pdev, tp); #ifdef ENABLE_R8127_PROCFS rtl8127_proc_remove(dev); From a5a92df98a78c3d269f95db14b8a17f54bda37ed Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 25 Mar 2026 10:18:16 -0500 Subject: [PATCH 131/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 5e34f09477c96..ec020a872f7fa 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2001.1) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Wed, 25 Mar 2026 10:18:16 -0500 + linux-nvidia-bos (6.19.0-2000.0) resolute; urgency=medium * Initial changelog entry. From 72a828e97a2c204472af1edd539784e79298b1b0 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 26 Mar 2026 10:46:07 -0500 Subject: [PATCH 132/464] UBUNTU: [Packaging] update variants BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/variants | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/debian.nvidia-bos/variants b/debian.nvidia-bos/variants index 6606318691bcd..b90e0071ab8fb 100644 --- a/debian.nvidia-bos/variants +++ b/debian.nvidia-bos/variants @@ -1,4 +1,3 @@ --6.19 --- --hwe-24.04 --hwe-24.04-edge +-7.0 +-hwe-26.04 +-hwe-26.04-edge From 4ea39568a928f30d2beeaa0b7ce2ce1024490e6a Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 26 Mar 2026 10:46:52 -0500 Subject: [PATCH 133/464] UBUNTU: [Config] nvidia: update configs for new compiler versions Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index db41b44b7b64e..a5b362d04a12f 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -232,5 +232,8 @@ CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> # ---- Annotations without notes ---- CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> -CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-15ubuntu1) 15.2.0"'}> +CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0"'}> CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_PAHOLE_VERSION policy<{'amd64': '131', 'arm64': '131'}> +CONFIG_RUSTC_VERSION policy<{'amd64': '109301', 'arm64': '109301'}> +CONFIG_RUSTC_VERSION_TEXT policy<{'amd64': '"rustc 1.93.1 (01f6ddf75 2026-02-11) (built from a source tarball)"', 'arm64': '"rustc 1.93.1 (01f6ddf75 2026-02-11) (built from a source tarball)"'}> From a99c32d9cbfa4dbc8189fff114560f3a1e376073 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 26 Mar 2026 10:48:56 -0500 Subject: [PATCH 134/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2001.1 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 490 +++++++++++++++++++++++++++++++++- debian.nvidia-bos/reconstruct | 33 --- 2 files changed, 485 insertions(+), 38 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index ec020a872f7fa..8311009682d4a 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,490 @@ -linux-nvidia-bos (7.0.0-2001.1) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2001.1) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * Packaging resync (LP: #1786013) + - [Packaging] update variants - -- Jacob Martin Wed, 25 Mar 2026 10:18:16 -0500 + * Backport NVIDIA: SAUCE: vfio/nvgrace-egm: split zapping EGM into 1GB + chunks (LP: #2142160) + - NVIDIA: SAUCE: vfio/nvgrace-egm: split zapping EGM into 1GB chunks + + * r8127 module unload triggers NAPI WARN in netif_napi_del_locked() + (LP: #2141780) + - NVIDIA: SAUCE: r8127: fix NAPI warning on module removal + + * Enable Coresight in Perf (LP: #2093957) + - [Packaging] Enable coresight in Perf if arm64 + - [Packaging] Add libopencsd-dev as a build dependency + + * Add ARM CCA host support (LP: #2139249) + - NVIDIA: VR: SAUCE: kvm: arm64: Include kvm_emulate.h in kvm/arm_psci.h + - NVIDIA: VR: SAUCE: arm64: RME: Handle Granule Protection Faults (GPFs) + - NVIDIA: VR: SAUCE: arm64: RME: Add SMC definitions for calling the RMM + - NVIDIA: VR: SAUCE: arm64: RME: Add wrappers for RMI calls + - NVIDIA: VR: SAUCE: arm64: RME: Check for RME support at KVM init + - NVIDIA: VR: SAUCE: arm64: RME: Define the user ABI + - NVIDIA: VR: SAUCE: arm64: RME: ioctls to create and configure realms + - NVIDIA: VR: SAUCE: kvm: arm64: Don't expose debug capabilities for realm + guests + - NVIDIA: VR: SAUCE: KVM: arm64: Allow passing machine type in KVM + creation + - NVIDIA: VR: SAUCE: arm64: RME: RTT tear down + - NVIDIA: VR: SAUCE: arm64: RME: Allocate/free RECs to match vCPUs + - NVIDIA: VR: SAUCE: KVM: arm64: vgic: Provide helper for number of list + registers + - NVIDIA: VR: SAUCE: arm64: RME: Support for the VGIC in realms + - NVIDIA: VR: SAUCE: KVM: arm64: Support timers in realm RECs + - NVIDIA: VR: SAUCE: arm64: RME: Allow VMM to set RIPAS + - NVIDIA: VR: SAUCE: arm64: RME: Handle realm enter/exit + - NVIDIA: VR: SAUCE: arm64: RME: Handle RMI_EXIT_RIPAS_CHANGE + - NVIDIA: VR: SAUCE: KVM: arm64: Handle realm MMIO emulation + - NVIDIA: VR: SAUCE: arm64: RME: Allow populating initial contents + - NVIDIA: VR: SAUCE: arm64: RME: Runtime faulting of memory + - NVIDIA: VR: SAUCE: KVM: arm64: Handle realm VCPU load + - NVIDIA: VR: SAUCE: KVM: arm64: Validate register access for a Realm VM + - NVIDIA: VR: SAUCE: KVM: arm64: Handle Realm PSCI requests + - NVIDIA: VR: SAUCE: KVM: arm64: WARN on injected undef exceptions + - NVIDIA: VR: SAUCE: arm64: Don't expose stolen time for realm guests + - NVIDIA: VR: SAUCE: arm64: RME: allow userspace to inject aborts + - NVIDIA: VR: SAUCE: arm64: RME: support RSI_HOST_CALL + - NVIDIA: VR: SAUCE: arm64: RME: Allow checking SVE on VM instance + - NVIDIA: VR: SAUCE: arm64: RME: Always use 4k pages for realms + - NVIDIA: VR: SAUCE: arm64: RME: Prevent Device mappings for Realms + - NVIDIA: VR: SAUCE: arm_pmu: Provide a mechanism for disabling the + physical IRQ + - NVIDIA: VR: SAUCE: arm64: RME: Enable PMU support with a realm guest + - NVIDIA: VR: SAUCE: arm64: RME: Hide KVM_CAP_READONLY_MEM for realm + guests + - NVIDIA: VR: SAUCE: arm64: RME: Propagate number of breakpoints and + watchpoints to userspace + - NVIDIA: VR: SAUCE: arm64: RME: Set breakpoint parameters through + SET_ONE_REG + - NVIDIA: VR: SAUCE: arm64: RME: Initialize PMCR.N with number counter + supported by RMM + - NVIDIA: VR: SAUCE: arm64: RME: Propagate max SVE vector length from RMM + - NVIDIA: VR: SAUCE: arm64: RME: Configure max SVE vector length for a + Realm + - NVIDIA: VR: SAUCE: arm64: RME: Provide register list for unfinalized RME + RECs + - NVIDIA: VR: SAUCE: arm64: RME: Provide accurate register list + - NVIDIA: VR: SAUCE: KVM: arm64: Expose support for private memory + - NVIDIA: VR: SAUCE: KVM: arm64: Expose KVM_ARM_VCPU_REC to user space + - NVIDIA: VR: SAUCE: KVM: arm64: Allow activating realms + - NVIDIA: VR: SAUCE: arm64: RME: Add MECID support + - NVIDIA: VR: SAUCE: arm64: RME: Add bounds check + - NVIDIA: VR: SAUCE: KVM: arm64: Expose KVM_CAP_ARM_RME via module + parameter + - NVIDIA: VR: SAUCE: [Config] nvidia: Update annotations for ARM CCA + + * VR: Add Live Firmware Activation (LFA) Support (LP: #2138342) + - NVIDIA: VR: SAUCE: firmware: smccc: add support for Live Firmware + Activation (LFA) + - NVIDIA: VR: SAUCE: firmware: smccc: add timeout, touch wdt + - NVIDIA: VR: SAUCE: firmware: smccc: register as platform driver + - NVIDIA: VR: SAUCE: [Config] nvidia: Enable ARM LFA support + + * Backport nvgrace-gpu hugepfnmap, ecc patches and miscellaneous cleanups + (LP: #2138892) + - NVIDIA: SAUCE: vfio/nvgrace-egm: register EGM PFNMAP range with + memory_failure + - NVIDIA: SAUCE: vfio: Remove vfio_device_from_file() declaration + + * Backport support for T410 PMU (LP: #2139315) + - NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Rename doc to Tegra241 + - NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU + - NVIDIA: VR: SAUCE: perf/arm_cspmu: Add arm_cspmu_acpi_dev_get + - NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE PMU + - NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT PMU + - NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 CPU Memory Latency PMU + - NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 C2C PMU + - NVIDIA: VR: SAUCE: arm64: defconfig: Enable NVIDIA TEGRA410 PMU + - NVIDIA: VR: SAUCE: perf vendor events arm64: Add Tegra410 Olympus PMU + events + - NVIDIA: VR: SAUCE: [Config] nvidia: enable TEGRA410_C2C_PMU and + TEGRA410_CMEM_LATENCY_PMU + + * Backport patches to enable ATS to remain always-on for CXL.cache devices + and specific NVIDIA GPUs by adding pci_ats_always_on() API and SMMU driver + support. (LP: #2139088) + - NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable + devices + - NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for non-CXL NVIDIA + GPUs + - NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on + + * backport "soc/tegra: pmc: Add PMC support for Tegra410" (LP: #2139082) + - NVIDIA: VR: SAUCE: soc/tegra: pmc: Add PMC support for Tegra410 + + * Backport NVIDIA: VR: SAUCE: arm64: Add workaround to convert MT_NORMAL_NC + to Device-nGnRE (LP: #2138952) + - NVIDIA: VR: SAUCE: arm64: Add workaround to convert MT_NORMAL_NC to + Device-nGnRE + - NVIDIA: VR: SAUCE: [Config] Enable arm64 NC-to-NGNRE workaround + + * missing prototype for vfio_device_from_file() (LP: #2138132) + - NVIDIA: SAUCE: vfio: Fix missing prototype warning + + * r8127: Downgrade GPL claim to info (LP: #2137588) + - NVIDIA: SAUCE: r8127: print GPL_CLAIM with KERN_INFO + + * Backport i2c patches for Tegra256, Tegra264, and Tegra410 (LP: #2138238) + - NVIDIA: VR: SAUCE: i2c: tegra: Introduce tegra_i2c_variant to identify + DVC and VI + - NVIDIA: VR: SAUCE: i2c: tegra: Move variant to tegra_i2c_hw_feature + - NVIDIA: VR: SAUCE: i2c: tegra: Add logic to support different register + offsets + - NVIDIA: VR: SAUCE: i2c: tegra: Add support for Tegra410 + + * Add kernel patches for CXL type 3 device support (LP: #2138266) + - NVIDIA: VR: SAUCE: cxl: add support for cxl reset + - NVIDIA: VR: SAUCE: cxl_test: enable zero sized decoders under hb0 + - NVIDIA: VR: SAUCE: cxl: Allow zero sized HDM decoders + - NVIDIA: VR: SAUCE: cxl/region: Validate partition index before array + access + - NVIDIA: VR: SAUCE: [Config] Add a CXL config for CXL type 3 devices + + * [linux-nvidia-6.17] Backport NVIDIA: VR: SAUCE: soc/tegra: misc: Use SMCCC + to get chipid (LP: #2138329) + - NVIDIA: VR: SAUCE: soc/tegra: misc: Use SMCCC to get chipid + + * mt7925: Incorrect MLO mode in firmware control (LP: #2138755) + - NVIDIA: SAUCE: wifi: mt76: mt7925: Fix incorrect MLO mode in firmware + control + + * Enable GDS in the 6.8 based linux-nvidia kernel (LP: #2059814) + - NVIDIA: [Packaging] Add nvidia-fs build dependencies + + * Add PCIe Hotplug Driver for CX7 on DGX Spark (LP: #2138269) + - NVIDIA: SAUCE: MEDIATEK: platform: Add PCIe Hotplug Driver for CX7 on + DGX Spark + + * Backport support for Grace MPAM (LP: #2122432) + - NVIDIA: SAUCE: DT: cacheinfo: Expose the code to generate a cache-id + from a device_node + - NVIDIA: SAUCE: DT: dt-bindings: arm: Add MPAM MSC binding + - NVIDIA: SAUCE: arm64: mpam: Context switch the MPAM registers + - NVIDIA: SAUCE: arm64: mpam: Re-initialise MPAM regs when CPU comes + online + - NVIDIA: SAUCE: arm64: mpam: Advertise the CPUs MPAM limits to the driver + - NVIDIA: SAUCE: arm64: mpam: Add cpu_pm notifier to restore MPAM sysregs + - NVIDIA: SAUCE: arm64: mpam: Add helpers to change a tasks and cpu mpam + partid/pmg values + - NVIDIA: SAUCE: cacheinfo: Add helper to find the cache size from + cpu+level + - NVIDIA: SAUCE: arm_mpam: resctrl: Add boilerplate cpuhp and domain + allocation + - NVIDIA: SAUCE: arm_mpam: resctrl: Pick the caches we will use as resctrl + resources + - NVIDIA: SAUCE: arm_mpam: resctrl: Implement + resctrl_arch_reset_all_ctrls() + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_get_config() + - NVIDIA: SAUCE: arm_mpam: resctrl: Implement helpers to update + configuration + - NVIDIA: SAUCE: arm_mpam: resctrl: Add plumbing against arm64 task and + cpu hooks + - NVIDIA: SAUCE: arm_mpam: resctrl: Add CDP emulation + - NVIDIA: SAUCE: arm_mpam: resctrl: Add rmid index helpers + - NVIDIA: SAUCE: arm_mpam: resctrl: Convert to/from MPAMs bitmaps and + fixed-point formats + - NVIDIA: SAUCE: arm_mpam: resctrl: Add support for 'MB' resource + - NVIDIA: SAUCE: arm_mpam: resctrl: Reject oversized memory bandwidth + portion bitmaps + - NVIDIA: SAUCE: arm_mpam: resctrl: Fix MB min_bandwidth value exposed to + userspace + - NVIDIA: SAUCE: arm_mpam: resctrl: Add kunit test for control format + conversions + - NVIDIA: SAUCE: arm_mpam: resctrl: Add support for csu counters + - NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate free running monitors + - NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate assignable monitors + - NVIDIA: SAUCE: arm_mpam: resctrl: Add kunit test for ABMC/CDP + interactions + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_config_cntr() for + ABMC use + - NVIDIA: SAUCE: arm_mpam: resctrl: Allow resctrl to allocate monitors + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_rmid_read() and + resctrl_arch_reset_rmid() + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_cntr_read() & + resctrl_arch_reset_cntr() + - NVIDIA: SAUCE: untested: arm_mpam: resctrl: Allow monitors to be + configured with filters + - NVIDIA: SAUCE: arm_mpam: resctrl: Add empty definitions for fine-grained + enables + - NVIDIA: SAUCE: arm64: mpam: Select ARCH_HAS_CPU_RESCTRL + - NVIDIA: SAUCE: fs/resctrl: Don't touch rmid_ptrs[] in free_rmid() when + there are no monitors + - NVIDIA: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and + closid_num_dirty_rmid[] + - NVIDIA: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and + rmid_ptrs[] + - NVIDIA: SAUCE: perf/arm-cmn: Stop claiming all the resources + - NVIDIA: SAUCE: arm_mpam: resctrl: Call resctrl_init() on platforms that + can support resctrl + - NVIDIA: SAUCE: arm_mpam: resctrl: Call resctrl_exit() in the event of + errors + - NVIDIA: SAUCE: arm_mpam: resctrl: Update the rmid reallocation limit + - NVIDIA: SAUCE: arm_mpam: resctrl: Sort the order of the domain lists + - NVIDIA: SAUCE: arm_mpam: Generate a configuration for min controls + - NVIDIA: SAUCE: arm_mpam: Add quirk framework + - NVIDIA: SAUCE: arm_mpam: Add workaround for T241-MPAM-1 + - NVIDIA: SAUCE: arm_mpam: Add workaround for T241-MPAM-4 + - NVIDIA: SAUCE: arm_mpam: Add workaround for T241-MPAM-6 + - NVIDIA: SAUCE: arm_mpam: Quirk CMN-650's CSU NRDY behaviour + - NVIDIA: SAUCE: debugfs: Add helpers for creating cpumask entries in + debugfs + - NVIDIA: SAUCE: arm_mpam: Add debugfs entries to show the MSC/RIS the + driver discovered + - NVIDIA: SAUCE: arm_mpam: Add force-disable debugfs trigger + - NVIDIA: SAUCE: arm_mpam: Expose the number of NRDY retries in debugfs + - NVIDIA: SAUCE: arm_mpam: Add resctrl_arch_round_bw() + - NVIDIA: SAUCE: fs/resctrl,x86/resctrl: Factor mba rounding to be per- + arch + - NVIDIA: SAUCE: arm_mpam: Relax num_rmids parameter advertised to + userspace + - NVIDIA: SAUCE: arm_mpam: Split the locking around the mon_sel registers + - NVIDIA: SAUCE: arm_mpam: Allow the maximum partid to be overridden from + the command line + - NVIDIA: SAUCE: arm_mpam: Allow MSC to be forced to have an unknown + location + - NVIDIA: SAUCE: fs/resctrl: Add this_is_not_abi mount option + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Register SMMU capabilities with MPAM + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Add mpam helpers to query and set + state + - NVIDIA: SAUCE: iommu: Add helpers to get and set the QoS state + - NVIDIA: SAUCE: iommu: Add helpers to retrieve iommu_groups by id or + kobject + - NVIDIA: SAUCE: iommu: Add helper to retrieve iommu kset + - NVIDIA: SAUCE: kobject: Add kset_get_next_obj() to allow a kset to be + walked + - NVIDIA: SAUCE: arm_mpam: resctrl: Add iommu helpers to get/set the + partid and pmg + - NVIDIA: SAUCE: fs/resctrl: Add support for assigning iommu_groups to + resctrl groups + - NVIDIA: SAUCE: firmware: arm_scmi: add MPAM-FB SCMI protocol stub + - NVIDIA: SAUCE: arm_mpam: add MPAM-FB MSC firmware access support + - NVIDIA: SAUCE: arm_mpam: Allow duplicate PCC subspace_ids + - NVIDIA: SAUCE: untested: mpam: Convert pcc_channels list to XArray and + cleanup + - NVIDIA: SAUCE: x86/resctrl: Add stub to allow other architecture to + disable monitor overflow + - NVIDIA: SAUCE: arm_mpam: resctrl: Determine if any exposed counter can + overflow + - NVIDIA: SAUCE: fs/restrl: Allow the overflow handler to be disabled + - NVIDIA: SAUCE: fs/resctrl: Uniform data type of + component_id/domid/id/cache_id + - NVIDIA: SAUCE: arm_mpam: Allow cmax/cmin to be configured + - NVIDIA: SAUCE: arm_mpam: Rename mbw conversion to 'fract16' for code re- + use + - NVIDIA: SAUCE: fs/resctrl: Group all the MBA specific properties in a + separate struct + - NVIDIA: SAUCE: fs/resctrl: Abstract duplicate domain test to a helper + - NVIDIA: SAUCE: fs/resctrl: Move MBA supported check to parse_line() + instead of parse_bw() + - NVIDIA: SAUCE: fs/resctrl: Rename resctrl_get_default_ctrl() to include + resource + - NVIDIA: SAUCE: fs/resctrl: Add a schema format to the schema, allowing + it to be different + - NVIDIA: SAUCE: fs/resctrl: Use schema format to check the resource is a + bitmap + - NVIDIA: SAUCE: fs/resctrl: Add specific schema types for 'range' + - NVIDIA: SAUCE: x86/resctrl: Move over to specifying MBA control formats + - NVIDIA: SAUCE: arm_mpam: resctrl: Convert MB resource to use percentage + - NVIDIA: SAUCE: fs/resctrl: Remove 'range' schema format + - NVIDIA: SAUCE: fs/resctrl: Add additional files for percentage and + bitmap controls + - NVIDIA: SAUCE: fs/resctrl: Add fflags_from_schema() for files based on + schema format + - NVIDIA: SAUCE: fs/resctrl: Expose the schema format to user-space + - NVIDIA: SAUCE: fs/resctrl: Add L2 and L3 'MAX' resource schema + - NVIDIA: SAUCE: arm_mpam: resctrl: Add the glue code to convert to/from + cmax + - NVIDIA: SAUCE: mm,memory_hotplug: Add lockdep assertion helper + - NVIDIA: SAUCE: fs/resctrl: Take memory hotplug lock whenever taking CPU + hotplug lock + - NVIDIA: SAUCE: fs/resctrl: Add mount option for mb_uses_numa_nid and + arch stubs + - NVIDIA: SAUCE: Fix unused variable warning + - NVIDIA: SAUCE: arm_mpam: resctrl: Pick whether MB can use NUMA nid + instead of cache-id + - NVIDIA: SAUCE: arm_mpam: resctrl: Change domain_hdr online/offline to + work with a set of CPUs + - NVIDIA: SAUCE: untested: arm_mpam: resctrl: Split + mpam_resctrl_alloc_domain() to have CPU and node + - NVIDIA: SAUCE: arm_mpam: resctrl: Add NUMA node notifier for domain + online/offline + - NVIDIA: SAUCE: untested: arm_mpam: resctrl: Allow resctrl to enable NUMA + nid as MB domain-id + - NVIDIA: SAUCE: [Config] RESCTRL configs added to annotations + - NVIDIA: SAUCE: arm_mpam: Fix missing SHIFT definitions + - NVIDIA: SAUCE: arm_mpam: resctrl: Fix MPAM kunit + - NVIDIA: SAUCE: resctrl/mpam: Align packed mpam_props to fix arm64 KUnit + alignment fault + - NVIDIA: SAUCE: resctrl/tests: mpam_devices: compare only meaningful + bytes of mpam_props + + * r8127: fix for LTS test panic (LP: #2134991) + - NVIDIA: SAUCE: r8127: Remove registers2 proc entry + + * Add two more Spark iGPU IDs for the existing iommu quirk (LP: #2132033) + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Add two more DGX Spark iGPU IDs for + existing iommu quirk + + * Pull CPPC mailing list patches for Spark (LP: #2131705) + - NVIDIA: SAUCE: ACPI: CPPC: Add cppc_get_perf() API to read performance + controls + - NVIDIA: SAUCE: ACPI: CPPC: extend APIs to support auto_sel and epp + - NVIDIA: SAUCE: ACPI: CPPC: add APIs and sysfs interface for min/max_perf + - NVIDIA: SAUCE: ACPI: CPPC: add APIs and sysfs interface for perf_limited + register + - NVIDIA: SAUCE: cpufreq: CPPC: Add sysfs for min/max_perf and + perf_limited + - NVIDIA: SAUCE: cpufreq: CPPC: update policy min/max when toggling + auto_select + - NVIDIA: SAUCE: cpufreq: CPPC: add autonomous mode boot parameter support + + * r8127: fix kernel panic when dump all registers (LP: #2130445) + - NVIDIA: SAUCE: r8127: fix a kernel panic when dump all registers + - NVIDIA: SAUCE: r8127: add support for RTL8127 cable diagnostic test + + * Set CONFIG_IOMMU_DEFAULT_PASSTHROUGH as default for Nvidia CPUs + (LP: #2129776) + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Set DGX Spark iGPU default domain type + to DMA + - [Config] nvidia: Update annotations to set + CONFIG_IOMMU_DEFAULT_PASSTHROUGH + + * mt7925: Introduce CSA support in non-MLO mode (LP: #2129209) + - NVIDIA: SAUCE: wifi: mt76: mt7925: introduce CSA support in non-MLO mode + + * IOMMU: Support contiguous bit in translation tables (LP: #2112600) + - NVIDIA: SAUCE: iommu/io-pgtable-arm: backport contiguous bit support + + * NVIDIA: SAUCE: MEDIATEK: usb: host: xhci-hub: fix MT89xx SoCs return + PORTLI value (LP: #2125126) + - NVIDIA: SAUCE: MEDIATEK: usb: host: xhci-hub: fix MT89xx SoCs return + PORTLI value + + * NVIDIA: SAUCE: ffa notification count initialization fix (LP: #2123861) + - NVIDIA: SAUCE: Fix FFA notification count initialization + + * Pull-request for setting CPU frequency gov to performance (LP: #2028576) + - [Config] nvidia: Use performance CPU frequency governor on amd64 + + * Set CONFIG_IOMMU_DEFAULT_DMA_LAZY as default for Nvidia CPUs + (LP: #2119661) + - [Config] nvidia: Update annotations to set CONFIG_IOMMU_DEFAULT_DMA_LAZY + + * Backport support for Grace virtualization features: vEVENTQ, HW QUEUE, and + vEGM (LP: #2119656) + - NVIDIA: SAUCE: arm64: configs: Build NVGRACE_GPU_VFIO_PCI as LKM + - NVIDIA: SAUCE: arm64: configs: Enable IOMMUFD and VFIO_DEVICE_CDEV + - NVIDIA: SAUCE: vfio/nvgrace-egm: Introduce module to manage EGM + - NVIDIA: SAUCE: vfio/nvgrace-egm: Handle pages with ECC errors on the EGM + - NVIDIA: SAUCE: arm64: configs: Build CONFIG_NVGRACE_EGM as LKM + - NVIDIA: SAUCE: vfio/nvgrace-egm: Move the egm header file to include + - NVIDIA: SAUCE: vfio/nvgrace-egm: Free region memory during + unregistration + - NVIDIA: SAUCE: vfio/nvgrace-egm: Move region hash initialization + - NVIDIA: SAUCE: vfio/nvgrace-egm: Handle and convey EGM registration + errors + - NVIDIA: SAUCE: vfio/nvgrace-gpu: Handle EGM registration failure + - NVIDIA: SAUCE: vfio/nvgrace-egm: Address sparse errors + - NVIDIA: SAUCE: vfio/nvgrace-gpu: Address smatch errors + - NVIDIA: SAUCE: vfio/nvgrace-egm: Ensure ACPI value reads are successful + - NVIDIA: SAUCE: vfio/nvgrace-egm: Avoid invalid retired pages base + - NVIDIA: SAUCE: vfio/nvgrace-egm: Update EGM unregistration API + - NVIDIA: SAUCE: vfio/nvgrace-egm: track GPUs associated with the EGM + regions + - NVIDIA: SAUCE: vfio/nvgrace-egm: list gpus through sysfs + - NVIDIA: SAUCE: vfio/nvgrace-egm: expose the egm size through sysfs + - NVIDIA: SAUCE: arm64: configs: enable NVGRACE_EGM as module + + * Backport support for arm64 BRBE and a future NVIDIA CPU ID (LP: #2118663) + - [Config] nvidia: Enable BRBE + + * nvidia-ffa-ec: Fix FFH data response length (LP: #2118357) + - NVIDIA: SAUCE: Fix FFH data response length + + * Add pincontrol driver for MT8901 chip (LP: #2117784) + - NVIDIA: SAUCE: MEDIATEK: pinctrl: mediatek: Add gpio-range record in + pinctrl driver + - NVIDIA: SAUCE: MEDIATEK: pinctrl: mediatek: Add acpi support + - NVIDIA: SAUCE: MEDIATEK: pinctrl: mt8901: Add pinctrl driver + - [Config] nvidia: Update annotations to enable CONFIG_PINCTRL_MT8901 + + * NVIDIA: SAUCE: Add FFA and EC Secure Service Driver to -nvidia kernel + (LP: #2114230) + - NVIDIA: SAUCE: Add support for custom ARM FFH offset handler + - NVIDIA: SAUCE: Add nvidia ffa driver for EC communication + - NVIDIA: SAUCE: Add ffa driver for each secure EC service + - NVIDIA: SAUCE: Add support for EC secure service communication + - NVIDIA: SAUCE: Rescan acpi devices that uses secure EC communication + - NVIDIA: SAUCE: irqchip/gic-v3: Allow unused SGIs for drivers/modules + - NVIDIA: SAUCE: Add support for notifications from secure EC services + - [Config] nvidia: Update annotations to enable NVIDIA FFA EC driver + + * Backport: TPM Service Command Response Buffer Interface Over FF-A + (LP: #2111511) + - [Config] nvidia-6.14: Update annotations to enable TPM over FFA + + * Backport: ALSA: hda - Add new driver for HDA controllers listed via ACPI + (LP: #2111447) + - NVIDIA: SAUCE: [Config] nvidia: CONFIG_SND_HDA_ACPI=m on arm64 + + * Pull request to enable GPU passthrough for CUDA (LP: #2095028) + - NVIDIA: SAUCE: WAR: iommufd/pages: Bypass PFNMAP + - NVIDIA: SAUCE: [Config] nvidia: Update annotations for Grace I/O + virtualization + - [Config] nvidia-6.14: Drop CONFIG_TEGRA241_CMDQV from annotations + + * Add Realtek r8127 ethernet driver (LP: #2109730) + - NVIDIA: SAUCE: r8127: Add Realtek r8127 ethernet driver + - NVIDIA: SAUCE: r8127: Remove Realtek r8127 non required files + - NVIDIA: SAUCE: r8127: Moved files from r8127/src to r8127 folder + - NVIDIA: SAUCE: Add r8127 in kernel build + - [Config] nvidia-6.11: Update annotations to enable realtek R8127 module + + * Pull request: Add quirk and disable SBR on Gen5 ports (LP: #2107509) + - NVIDIA: SAUCE: MEDIATEK: usb: host: xhci-plat: support usb3 bulks stream + low power + + * Apply backport of upstream commit to enable Realtek Bluetooth module + (LP: #2096882) + - NVIDIA: SAUCE: Adds MT7925 BT devices + + * Apply SAUCE patch to enable 8250 serial device (LP: #2096888) + - NVIDIA: SAUCE: serial: 8250_mtk: Add ACPI support + + * Backport: "Add support for AArch64 AMUv1-based average freq" Series + (LP: #2100032) + - NVIDIA: [Config] set CONFIG_CPUFREQ_ARCH_CUR_FREQ=y for x86 + + * MANA: include driver fixes and enable module on ARM64 (LP: #2084598) + - [Config] nvidia-6.17: Enable MANA configs on x86 and arm64 + + * Apply patch to set CONFIG_EFI_CAPSULE_LOADER=y for arm64 (LP: #2067111) + - NVIDIA: [Config] EFI: set CAPSULE_LOADER=y for arm64 + + * linux-nvidia-6.5_6.5.0-1014.14 breaks with earlier BIOS release, and + modeset/resolutions are wrong (LP: #2061930) // Blacklist coresight_etm4x + (LP: #2067106) + - [Packaging] blacklist coresight_etm4x + + * backport arm64 THP improvements from 6.9 (LP: #2059316) + - NVIDIA: [Config] arm64: ARM64_CONTPTE=y + + * Reapply the linux-nvidia kernel config options from the 5.15 and 6.5 + kernels (LP: #2060327) + - NVIDIA: [Config]: Disable the NOUVEAU driver which is not used with + -nvidia kernels + - NVIDIA: [Config]: Adding CORESIGHT and ARM64_ERRATUM configs to + annotations + + [ Ubuntu: 7.0.0-6.6 ] + + * resolute/linux: 7.0.0-6.6 -proposed tracker (LP: #2143745) + * Miscellaneous Ubuntu changes + - [Packaging] drop unstable suffix + + -- Jacob Martin Thu, 26 Mar 2026 10:48:56 -0500 linux-nvidia-bos (6.19.0-2000.0) resolute; urgency=medium diff --git a/debian.nvidia-bos/reconstruct b/debian.nvidia-bos/reconstruct index 16e52ee71b8a0..ca916d098dabf 100644 --- a/debian.nvidia-bos/reconstruct +++ b/debian.nvidia-bos/reconstruct @@ -1,34 +1 @@ -# Recreate any symlinks created since the orig. -[ ! -L 'ubuntu/igh-ecat/master/rtdm-ioctl.c' ] && ln -sf 'ioctl.c' 'ubuntu/igh-ecat/master/rtdm-ioctl.c' -chmod +x 'debian/cloud-tools/hv_get_dhcp_info' -chmod +x 'debian/cloud-tools/hv_get_dns_info' -chmod +x 'debian/cloud-tools/hv_set_ifconfig' -chmod +x 'debian/rules' -chmod +x 'debian/scripts/checks/final-checks' -chmod +x 'debian/scripts/checks/module-signature-check' -chmod +x 'debian/scripts/control-create' -chmod +x 'debian/scripts/dkms-build' -chmod +x 'debian/scripts/dkms-build--nvidia-N' -chmod +x 'debian/scripts/dkms-build-configure--zfs' -chmod +x 'debian/scripts/file-downloader' -chmod +x 'debian/scripts/link-headers' -chmod +x 'debian/scripts/link-lib-rust' -chmod +x 'debian/scripts/misc/annotations' -chmod +x 'debian/scripts/misc/find-missing-sauce.sh' -chmod +x 'debian/scripts/misc/gen-auto-reconstruct' -chmod +x 'debian/scripts/misc/git-ubuntu-log' -chmod +x 'debian/scripts/misc/insert-changes' -chmod +x 'debian/scripts/misc/insert-ubuntu-changes' -chmod +x 'debian/scripts/misc/kernelconfig' -chmod +x 'debian/scripts/sign-module' -chmod +x 'debian/templates/extra.postinst.in' -chmod +x 'debian/templates/extra.postrm.in' -chmod +x 'debian/templates/headers.postinst.in' -chmod +x 'debian/templates/image.postinst.in' -chmod +x 'debian/templates/image.postrm.in' -chmod +x 'debian/templates/image.preinst.in' -chmod +x 'debian/templates/image.prerm.in' -chmod +x 'debian/tests/rebuild' -chmod +x 'debian/tests/ubuntu-regression-suite' -# Remove any files deleted from the orig. exit 0 From 92609e58ba4f196d11a57a8799c88a70fedb7320 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 31 Mar 2026 15:55:06 -0500 Subject: [PATCH 135/464] UBUNTU: [Packaging] update variants BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/variants | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/debian.nvidia-bos/variants b/debian.nvidia-bos/variants index b90e0071ab8fb..cab2f62a9df90 100644 --- a/debian.nvidia-bos/variants +++ b/debian.nvidia-bos/variants @@ -1,3 +1,2 @@ -7.0 --hwe-26.04 --hwe-26.04-edge +-- From 7f8bd2564abb7cc3d905ecbe120de0e1b234db4c Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 31 Mar 2026 16:03:43 -0500 Subject: [PATCH 136/464] UBUNTU: [Packaging] nvidia-bos: Convert nvidia flavor to nvidia-bos We cannot use the nvidia flavor without clashing with the resolute/linux-nvidia packaging, as both kernels will try to produce a linux-nvidia-7.0 meta package, which is not allowed. Instead, convert this kernel to use the nvidia-bos flavor. Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 6 +++--- .../control.d/{vars.nvidia => vars.nvidia-bos} | 0 .../control.d/{vars.nvidia-64k => vars.nvidia-bos-64k} | 0 debian.nvidia-bos/rules.d/amd64.mk | 2 +- debian.nvidia-bos/rules.d/arm64.mk | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) rename debian.nvidia-bos/control.d/{vars.nvidia => vars.nvidia-bos} (100%) rename debian.nvidia-bos/control.d/{vars.nvidia-64k => vars.nvidia-bos-64k} (100%) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index a5b362d04a12f..69a19c9aaac3e 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -1,8 +1,8 @@ # Menu: HEADER # FORMAT: 4 # ARCH: amd64 arm64 -# FLAVOUR: amd64-nvidia arm64-nvidia arm64-nvidia-64k -# FLAVOUR_DEP: {'amd64-nvidia': 'amd64-generic', 'arm64-nvidia': 'arm64-generic', 'arm64-nvidia-64k': 'arm64-generic-64k'} +# FLAVOUR: amd64-nvidia-bos arm64-nvidia-bos arm64-nvidia-bos-64k +# FLAVOUR_DEP: {'amd64-nvidia-bos': 'amd64-generic', 'arm64-nvidia-bos': 'arm64-generic', 'arm64-nvidia-bos-64k': 'arm64-generic-64k'} include "../../debian.master/config/annotations" @@ -45,7 +45,7 @@ CONFIG_ARM_LFA note<'LP: #2138342'> CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> -CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '32', 'arm64-nvidia': '128', 'arm64-nvidia-64k': '1024'}> +CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '32', 'arm64-nvidia-bos': '128', 'arm64-nvidia-bos-64k': '1024'}> CONFIG_CMA_SIZE_MBYTES note<'LP: #2095028'> CONFIG_CORESIGHT policy<{'arm64': 'm'}> diff --git a/debian.nvidia-bos/control.d/vars.nvidia b/debian.nvidia-bos/control.d/vars.nvidia-bos similarity index 100% rename from debian.nvidia-bos/control.d/vars.nvidia rename to debian.nvidia-bos/control.d/vars.nvidia-bos diff --git a/debian.nvidia-bos/control.d/vars.nvidia-64k b/debian.nvidia-bos/control.d/vars.nvidia-bos-64k similarity index 100% rename from debian.nvidia-bos/control.d/vars.nvidia-64k rename to debian.nvidia-bos/control.d/vars.nvidia-bos-64k diff --git a/debian.nvidia-bos/rules.d/amd64.mk b/debian.nvidia-bos/rules.d/amd64.mk index 8aa96b3e758cc..754a0ed1ca809 100644 --- a/debian.nvidia-bos/rules.d/amd64.mk +++ b/debian.nvidia-bos/rules.d/amd64.mk @@ -1,6 +1,6 @@ build_arch = x86 defconfig = defconfig -flavours = nvidia +flavours = nvidia-bos build_image = bzImage kernel_file = arch/$(build_arch)/boot/bzImage install_file = vmlinuz diff --git a/debian.nvidia-bos/rules.d/arm64.mk b/debian.nvidia-bos/rules.d/arm64.mk index f086214eb37ad..782c502b9b014 100644 --- a/debian.nvidia-bos/rules.d/arm64.mk +++ b/debian.nvidia-bos/rules.d/arm64.mk @@ -1,6 +1,6 @@ build_arch = arm64 defconfig = defconfig -flavours = nvidia nvidia-64k +flavours = nvidia-bos nvidia-bos-64k build_image = vmlinuz.efi kernel_file = arch/$(build_arch)/boot/vmlinuz.efi install_file = vmlinuz From 34105213d64ac6cb8bb5a2d98115f3111437366d Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 31 Mar 2026 16:58:59 -0500 Subject: [PATCH 137/464] UBUNTU: [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel-versions (adhoc/d2026.02.16) BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/dkms-versions | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/debian.nvidia-bos/dkms-versions b/debian.nvidia-bos/dkms-versions index 5ccbdb3d3e79c..fd8e9c633b136 100644 --- a/debian.nvidia-bos/dkms-versions +++ b/debian.nvidia-bos/dkms-versions @@ -1,8 +1,2 @@ -zfs-linux 2.4.0-1ubuntu3 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms -evdi 1.14.12+dfsg-1ubuntu1 modulename=evdi debpath=pool/universe/e/%package%/evdi-dkms_%version%_all.deb rprovides=evdi-modules rprovides=evdi-dkms type=standalone -ipu6-drivers 0~git202511120800.9766e218-0ubuntu2 modulename=ipu6 debpath=pool/universe/i/%package%/intel-ipu6-dkms_%version%_amd64.deb arch=amd64 rprovides=ipu6-modules rprovides=intel-ipu6-dkms type=standalone -ipu7-drivers 0~git202511120800.fc335577-0ubuntu1 modulename=ipu7 debpath=pool/universe/i/%package%/intel-ipu7-dkms_%version%_amd64.deb arch=amd64 rprovides=ipu7-modules rprovides=intel-ipu7-dkms type=standalone -backport-iwlwifi-dkms 1:0~96.13623-gitd16e74cc-0ubuntu2 modulename=iwlwifi debpath=pool/universe/b/%package%/backport-iwlwifi-dkms_%version%_all.deb arch=amd64 rprovides=iwlwifi-modules rprovides=backport-iwlwifi-dkms type=standalone -v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms -usbio-drivers 0~git202510282139.ee221eca-0ubuntu1 modulename=usbio debpath=pool/universe/u/%package%/intel-usbio-dkms_%version%_amd64.deb arch=amd64 rprovides=usbio-modules rprovides=intel-usbio-dkms type=standalone -vision-drivers 0~git202511121832.a8d772f2-0ubuntu1 modulename=vision debpath=pool/universe/v/%package%/intel-vision-dkms_%version%_amd64.deb arch=amd64 rprovides=vision-modules rprovides=intel-vision-dkms type=standalone +zfs-linux 2.4.1-1ubuntu1 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=riscv64 arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms off_series=true +v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms off_series=true From 9207526d4d713a5d977878a5598ef3944b2a44d9 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Apr 2026 17:06:53 -0500 Subject: [PATCH 138/464] UBUNTU: [Packaging] Add missing libnl and libbpf dependencies Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/control.stub.in | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/control.stub.in b/debian.nvidia-bos/control.stub.in index 2d57a649be27c..deea8e8ccef33 100644 --- a/debian.nvidia-bos/control.stub.in +++ b/debian.nvidia-bos/control.stub.in @@ -21,6 +21,7 @@ Build-Depends: java-common , kmod , libaudit-dev , + libbpf-dev , libcap-dev , libdebuginfod-dev [amd64 arm64] , libdw-dev , @@ -28,6 +29,8 @@ Build-Depends: libiberty-dev , liblzma-dev , libnewt-dev , + libnl-3-dev, + libnl-genl-3-dev, libnuma-dev [amd64 arm64] , libpci-dev , libssl-dev , From 9104e62dbdb0d5b8cd02a6f926b0eda3cf733e73 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 31 Mar 2026 16:51:31 -0500 Subject: [PATCH 139/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 8311009682d4a..2dcbad3116bc0 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2003.3) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Tue, 31 Mar 2026 16:51:30 -0500 + linux-nvidia-bos (7.0.0-2001.1) resolute; urgency=medium * Packaging resync (LP: #1786013) From 5d893c5d4643370cd734150a99e9a1db1d0bf016 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Apr 2026 17:08:17 -0500 Subject: [PATCH 140/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2003.3 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 131 ++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 2dcbad3116bc0..37e36c1788978 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,131 @@ -linux-nvidia-bos (7.0.0-2003.3) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2003.3) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * Packaging resync (LP: #1786013) + - [Packaging] update variants + - [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel- + versions (adhoc/d2026.02.16) + + [ Ubuntu: 7.0.0-10.10 ] + + * resolute/linux: 7.0.0-10.10 -proposed tracker (LP: #2144865) + * Miscellaneous upstream changes + - Revert "powerpc: fix KUAP warning in VMX usercopy path" + + [ Ubuntu: 7.0.0-9.9 ] + + * resolute/linux: 7.0.0-9.9 -proposed tracker (LP: #2144735) + * Please make dracut the default initrd generator (LP: #2142775) + - [Packaging] recommends dracut instead of initramfs-tools + * Miscellaneous Ubuntu changes + - SAUCE: Change RISC-V target to RVA23 (riscv64a23-unknown-linux-gnu) + + [ Ubuntu: 7.0.0-8.8 ] + + * resolute/linux: 7.0.0-8.8 -proposed tracker (LP: #2144652) + * UBUNTU: SAUCE: igc: Increase Thunderbolt MAC passthrough delay to 1000ms + (LP: #2143197) + - SAUCE: igc: Increase Thunderbolt MAC passthrough delay to 1000ms + * [usrmerge] evaluate kernel owned packages for DEP17 compliance + (LP: #2139276) + - [Packaging] Install modules in /usr/lib/modules + * Miscellaneous Ubuntu changes + - [Config] hardening: enable LIST_HARDENED + - [Config] hardening: disable LDISC_AUTOLOAD + - [Config] hardening: disable LEGACY_PTYS + - [Config] updateconfigs following v7.0-rc4 rebase + + [ Ubuntu: 7.0.0-7.7 ] + + * resolute/linux: 7.0.0-7.7 -proposed tracker (LP: #2143974) + * unconfined profile denies userns_create for chromium based processes + (LP: #1990064) + - [Config] disable CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS + * Jellyfin Desktop Flatpak doesn't work with the current AppArmor profile + (LP: #2142956) + - SAUCE: apparmor5.0.0 [29/29]: apparmor: fix fine grained inet mediation + sock_file_perm + * AppArmor blocks write(2) to network sockets with Linux 6.19 (LP: #2141298) + - SAUCE: apparmor5.0.0 [28/29]: apparmor: fix aa_label_sk_perm to check + for RULE_MEDIATES_NET + * update apparmor and LSM stacking patch set (LP: #2028253) + - SAUCE: apparmor5.0.0 [1/29]: Stacking: LSM: Single calls in secid hooks + - SAUCE: apparmor5.0.0 [2/29]: Stacking: LSM: Exclusive secmark usage + - SAUCE: apparmor5.0.0 [3/29]: Stacking: AppArmor: Remove the exclusive + flag + - SAUCE: apparmor5.0.0 [4/29]: Revert "apparmor: fix dbus permission + queries to v9 ABI" + - SAUCE: apparmor5.0.0 [5/29]: Revert "apparmor: gate make fine grained + unix mediation behind v9 abi" + - SAUCE: apparmor5.0.0 [6/29]: apparmor: net: patch to provide + compatibility with v2.x net rules + - SAUCE: apparmor5.0.0 [7/29]: apparmor: net: add fine grained ipv4/ipv6 + mediation + - SAUCE: apparmor5.0.0 [8/29]: apparmor: lift compatibility check out of + profile_af_perm + - SAUCE: apparmor5.0.0 [9/29]: apparmor: userns: add unprivileged user ns + mediation + - SAUCE: apparmor5.0.0 [10/29]: apparmor: userns: Add sysctls for + additional controls of unpriv userns restrictions + - SAUCE: apparmor5.0.0 [12/29]: apparmor: userns: open userns related + sysctl so lxc can check if restriction are in place + - SAUCE: apparmor5.0.0 [13/29]: apparmor: userns: allow profile to be + transitioned when a userns is created + - SAUCE: apparmor5.0.0 [14/29]: apparmor: mqueue: call + security_inode_init_security on inode creation + - SAUCE: apparmor5.0.0 [15/29]: apparmor: mqueue: add fine grained + mediation of posix mqueues + - SAUCE: apparmor5.0.0 [16/29]: apparmor: uring: add io_uring mediation + - SAUCE: apparmor5.0.0 [19/29]: apparmor: prompt: setup slab cache for + audit data + - SAUCE: apparmor5.0.0 [20/29]: apparmor: prompt: add the ability for + profiles to have a learning cache + - SAUCE: apparmor5.0.0 [21/29]: apparmor: prompt: enable userspace upcall + for mediation + - SAUCE: apparmor5.0.0 [22/29]: apparmor: prompt: pass prompt boolean + through into path_name as well + - SAUCE: apparmor5.0.0 [23/29]: apparmor: check for supported version in + notification messages. + - SAUCE: apparmor5.0.0 [24/29]: apparmor: refactor building notice so it + is easier to extend + - SAUCE: apparmor5.0.0 [25/29]: apparmor: switch from ENOTSUPP to + EPROTONOSUPPORT + - SAUCE: apparmor5.0.0 [26/29]: apparmor: add support for meta data tags + - SAUCE: apparmor5.0.0 [27/29]: apparmor: prevent profile->disconnected + double free in aa_free_profile + * update apparmor and LSM stacking patch set (LP: #2028253) // Installation + of AppArmor on a 6.14 kernel produces error message "Illegal number: yes" + (LP: #2102680) + - SAUCE: apparmor5.0.0 [17/29]: apparmor: create an + AA_SFS_TYPE_BOOLEAN_INTPRINT sysctl variant + - SAUCE: apparmor5.0.0 [18/29]: apparmor: Use AA_SFS_FILE_BOOLEAN_INTPRINT + for userns and io_uring sysctls + * update apparmor and LSM stacking patch set (LP: #2028253) // [FFe] + apparmor-4.0.0-alpha2 for unprivileged user namespace restrictions in + mantic (LP: #2032602) + - SAUCE: apparmor5.0.0 [11/29]: apparmor: userns - make it so special + unconfined profiles can mediate user namespaces + * NPU utilization on amdxdna is missing (LP: #2143243) + - SAUCE: accel/amdxdna: Add IOCTL to retrieve realtime NPU power estimate + - SAUCE: accel/amdxdna: Support sensors for column utilization + - SAUCE: accel/amdxdna: Import AMD_PMF namespace + * Adopting dark mode by default for OLED panel (LP: #2143203) + - SAUCE: drm/connector: Add a new 'panel_type' property + - SAUCE: drm/amd/display: Attach OLED property to eDP panels + * Support AMD Image Signal Processing (ISP) unit V4.0 (LP: #2110092) + - SAUCE: media: platform: amd: Introduce amd isp4 capture driver + - SAUCE: media: platform: amd: low level support for isp4 firmware + - SAUCE: media: platform: amd: Add isp4 fw and hw interface + - SAUCE: media: platform: amd: isp4 subdev and firmware loading handling + added + - SAUCE: media: platform: amd: isp4 video node and buffers handling added + - SAUCE: Documentation: add documentation of AMD isp 4 driver + - SAUCE: media: platform: amd: isp4 debug fs logging and more descriptive + errors + - [Config] Enable VIDEO_AMD_ISP4_CAPTURE + * Miscellaneous Ubuntu changes + - [Config] temporarily disable OBJTOOL_WERROR - -- Jacob Martin Tue, 31 Mar 2026 16:51:30 -0500 + -- Jacob Martin Wed, 01 Apr 2026 17:08:17 -0500 linux-nvidia-bos (7.0.0-2001.1) resolute; urgency=medium From f21040d74f3b3ef61f340c45dce6d371e3978e8d Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Apr 2026 09:27:59 -0500 Subject: [PATCH 141/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 37e36c1788978..be25ee3372967 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2004.4) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Tue, 07 Apr 2026 09:27:59 -0500 + linux-nvidia-bos (7.0.0-2003.3) resolute; urgency=medium * Packaging resync (LP: #1786013) From d100f65888dc57216dbfe26de8e3b4d4ceabd6f8 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Apr 2026 09:48:58 -0500 Subject: [PATCH 142/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2146933 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index eaf24103d1343..033a0348ec6c7 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2142114 d2026.02.16-1 +2146933 d2026.03.17-1 From 1ae5c2c67b79f1a42ea106f6fea872386be51871 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Apr 2026 10:12:45 -0500 Subject: [PATCH 143/464] UBUNTU: [Config] update configs Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 4 ---- 1 file changed, 4 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 69a19c9aaac3e..386aabc4bc8cc 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -232,8 +232,4 @@ CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> # ---- Annotations without notes ---- CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> -CONFIG_CC_VERSION_TEXT policy<{'amd64': '"x86_64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0"', 'arm64': '"aarch64-linux-gnu-gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0"'}> CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> -CONFIG_PAHOLE_VERSION policy<{'amd64': '131', 'arm64': '131'}> -CONFIG_RUSTC_VERSION policy<{'amd64': '109301', 'arm64': '109301'}> -CONFIG_RUSTC_VERSION_TEXT policy<{'amd64': '"rustc 1.93.1 (01f6ddf75 2026-02-11) (built from a source tarball)"', 'arm64': '"rustc 1.93.1 (01f6ddf75 2026-02-11) (built from a source tarball)"'}> From fce3f67543f1b50b909ae6dcff74d0ff4cea1d61 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Apr 2026 10:13:31 -0500 Subject: [PATCH 144/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2004.4 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 238 +++++++++++++++++++++++++++++++++++- 1 file changed, 233 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index be25ee3372967..738db0f1809a0 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,238 @@ -linux-nvidia-bos (7.0.0-2004.4) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2004.4) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2004.4 -proposed tracker (LP: #2146933) - -- Jacob Martin Tue, 07 Apr 2026 09:27:59 -0500 + [ Ubuntu: 7.0.0-12.12 ] + + * resolute/linux: 7.0.0-12.12 -proposed tracker (LP: #2146778) + * Packaging resync (LP: #1786013) + - [Packaging] update variants + * linux-generic does not run scripts in /usr/share/kernel/*.d (LP: #2147005) + - [Packaging] templates: Use consistent indentation + - [Packaging] templates: Run scripts in /usr/share/kernel/*.d too + * RISC-V kernel config is out of sync with other archs (LP: #1981437) + - [Config] riscv64: Enable COUNTER=m + - [Config] riscv64: Use GENDWARFKSYMS like other architectures + * unconfined profile denies userns_create for chromium based processes + (LP: #1990064) + - [Config] disable CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS + * FFe: add network interface mediation to 26.04 (LP: #2144679) + - SAUCE: apparmor5.0.0 [57/57]: apparmor: add the ability to use interface + in network mediation. + * Jellyfin Desktop Flatpak doesn't work with the current AppArmor profile + (LP: #2142956) + - SAUCE: apparmor5.0.0 [29/57]: apparmor: fix fine grained inet mediation + sock_file_perm + - SAUCE: apparmor5.0.0 [30/57]: apparmor-next 7.1: aapparmor: use target + task's context in apparmor_getprocattr() + - SAUCE: apparmor5.0.0 [31/57]: apparmor-next 7.1: apparmor: return error + on namespace mismatch in verify_header + - SAUCE: apparmor5.0.0 [32/57]: apparmor-next 7.1: apparmor: enable + differential encoding + - SAUCE: apparmor5.0.0 [33/57]: apparmor-next 7.1: apparmor: propagate + -ENOMEM correctly in unpack_table + - SAUCE: apparmor5.0.0 [34/57]: apparmor-next 7.1: apparmor: Replace + memcpy + NUL termination with kmemdup_nul in do_setattr + - SAUCE: apparmor5.0.0 [35/57]: apparmor-next 7.1: apparmor: Remove + redundant if check in sk_peer_get_label + - SAUCE: apparmor5.0.0 [36/57]: apparmor-next 7.1: apparmor: use + __label_make_stale in __aa_proxy_redirect + - SAUCE: apparmor5.0.0 [37/57]: apparmor-next 7.1: apparmor: fix net.h and + policy.h circular include pattern + - SAUCE: apparmor5.0.0 [39/57]: apparmor-next 7.1: apparmor: make include + headers self-contained + - SAUCE: apparmor5.0.0 [40/57]: apparmor-next 7.1: apparmor: Use + sysfs_emit in param_get_{audit,mode} + - SAUCE: apparmor5.0.0 [41/57]: apparmor-next 7.1: apparmor: fix + rawdata_f_data implicit flex array + - SAUCE: apparmor5.0.0 [42/57]: apparmor-next 7.1: apparmor: free rawdata + as soon as possible + - SAUCE: apparmor5.0.0 [43/57]: apparmor-next 7.1: apparmor: Initial + support for compressed policies + - SAUCE: apparmor5.0.0 [44/57]: apparmor-next 7.1: apparmor: fix potential + UAF in aa_replace_profiles + - SAUCE: apparmor5.0.0 [45/57]: apparmor-next 7.1: apparmor: hide unused + get_loaddata_common_ref() function + - SAUCE: apparmor5.0.0 [46/57]: apparmor-next 7.1: apparmor: Fix string + overrun due to missing termination + - SAUCE: apparmor5.0.0 [47/57]: apparmor: fix packed tag on v5 header + struct + - SAUCE: apparmor5.0.0 [48/57]: apparmor: add temporal caching to audit + responses. + - SAUCE: apparmor5.0.0 [49/57]: apparmor: change fn_label_build() call to + not return NULL + - SAUCE: apparmor5.0.0 [50/57]: apparmor: make fn_label_build() capable of + handling not supported + - SAUCE: apparmor5.0.0 [51/57]: apparmor: move netfilter functions next to + the LSM network operations + - SAUCE: apparmor5.0.0 [52/57]: apparmor: move sock_rvc_skb() next to + inet_conn_request + - SAUCE: apparmor5.0.0 [53/57]: apparmor: fix af_unix local addr mediation + binding + - SAUCE: apparmor5.0.0 [54/57]: cleanups of apparmor af_unix mediation + - SAUCE: apparmor5.0.0 [55/57]: apparmor: fix apparmor_secmark_check() + when !inet and secmark defined. + - SAUCE: apparmor5.0.0 [56/57]: apparmor: fix auditing of non-mediation + falures + * snap service cannot change apparmor hat (LP: #2139664) // Jellyfin Desktop + Flatpak doesn't work with the current AppArmor profile (LP: #2142956) + - SAUCE: apparmor5.0.0 [38/57]: apparmor-next 7.1: apparmor: grab ns lock + and refresh when looking up changehat child profiles + * AppArmor blocks write(2) to network sockets with Linux 6.19 (LP: #2141298) + - SAUCE: apparmor5.0.0 [28/57]: apparmor: fix aa_label_sk_perm to check + for RULE_MEDIATES_NET + * update apparmor and LSM stacking patch set (LP: #2028253) + - SAUCE: apparmor5.0.0 [1/57]: Stacking: LSM: Single calls in secid hooks + - SAUCE: apparmor5.0.0 [2/57]: Stacking: LSM: Exclusive secmark usage + - SAUCE: apparmor5.0.0 [3/57]: Stacking: AppArmor: Remove the exclusive + flag + - SAUCE: apparmor5.0.0 [4/57]: Revert "apparmor: fix dbus permission + queries to v9 ABI" + - SAUCE: apparmor5.0.0 [5/57]: Revert "apparmor: gate make fine grained + unix mediation behind v9 abi" + - SAUCE: apparmor5.0.0 [6/57]: apparmor: net: patch to provide + compatibility with v2.x net rules + - SAUCE: apparmor5.0.0 [7/57]: apparmor: net: add fine grained ipv4/ipv6 + mediation + - SAUCE: apparmor5.0.0 [8/57]: apparmor: lift compatibility check out of + profile_af_perm + - SAUCE: apparmor5.0.0 [9/57]: apparmor: userns: add unprivileged user ns + mediation + - SAUCE: apparmor5.0.0 [10/57]: apparmor: userns: Add sysctls for + additional controls of unpriv userns restrictions + - SAUCE: apparmor5.0.0 [12/57]: apparmor: userns: open userns related + sysctl so lxc can check if restriction are in place + - SAUCE: apparmor5.0.0 [13/57]: apparmor: userns: allow profile to be + transitioned when a userns is created + - SAUCE: apparmor5.0.0 [14/57]: apparmor: mqueue: call + security_inode_init_security on inode creation + - SAUCE: apparmor5.0.0 [15/57]: apparmor: mqueue: add fine grained + mediation of posix mqueues + - SAUCE: apparmor5.0.0 [16/57]: apparmor: uring: add io_uring mediation + - SAUCE: apparmor5.0.0 [19/57]: apparmor: prompt: setup slab cache for + audit data + - SAUCE: apparmor5.0.0 [20/57]: apparmor: prompt: add the ability for + profiles to have a learning cache + - SAUCE: apparmor5.0.0 [21/57]: apparmor: prompt: enable userspace upcall + for mediation + - SAUCE: apparmor5.0.0 [22/57]: apparmor: prompt: pass prompt boolean + through into path_name as well + - SAUCE: apparmor5.0.0 [23/57]: apparmor: check for supported version in + notification messages. + - SAUCE: apparmor5.0.0 [24/57]: apparmor: refactor building notice so it + is easier to extend + - SAUCE: apparmor5.0.0 [25/57]: apparmor: switch from ENOTSUPP to + EPROTONOSUPPORT + - SAUCE: apparmor5.0.0 [26/57]: apparmor: add support for meta data tags + - SAUCE: apparmor5.0.0 [27/57]: apparmor: prevent profile->disconnected + double free in aa_free_profile + * update apparmor and LSM stacking patch set (LP: #2028253) // Installation + of AppArmor on a 6.14 kernel produces error message "Illegal number: yes" + (LP: #2102680) + - SAUCE: apparmor5.0.0 [17/57]: apparmor: create an + AA_SFS_TYPE_BOOLEAN_INTPRINT sysctl variant + - SAUCE: apparmor5.0.0 [18/57]: apparmor: Use AA_SFS_FILE_BOOLEAN_INTPRINT + for userns and io_uring sysctls + * update apparmor and LSM stacking patch set (LP: #2028253) // [FFe] + apparmor-4.0.0-alpha2 for unprivileged user namespace restrictions in + mantic (LP: #2032602) + - SAUCE: apparmor5.0.0 [11/57]: apparmor: userns - make it so special + unconfined profiles can mediate user namespaces + * Enable new Intel WCL soundwire support (LP: #2143301) + - ASoC: sdw_utils: Add CS42L43B codec info + - ASoC: dt-bindings: cirrus, cs42l43: Add CS42L43B variant + - mfd: cs42l43: Add support for the B variant + - ASoC: cs42l43: Add support for the B variant + * Enable audio functions on Dell Huracan/Renegade platforms w/o built-in + microphone (LP: #2143902) + - ASoC: SDCA: Add default value for mipi-sdca-function-reset-max-delay + - ASoC: SDCA: Update counting of SU/GE DAPM routes + - ASoC: SDCA: Improve mapping of Q7.8 SDCA volumes + - ASoC: SDCA: Pull the Q7.8 volume helpers out of soc-ops + - ASoC: add snd_soc_lookup_component_by_name helper + - ASoC: soc_sdw_utils: partial match the codec name + - ASoC: soc_sdw_utils: remove index from sdca codec name + * [SRU] MIPI camera is not working after upgrading to 6.17-oem + (LP: #2145171) + - SAUCE: ACPI: respect items already in honor_dep before skipping + * linux-tools: consider linking perf against LLVM (LP: #2138328) + - [Packaging] Actually enable llvm for perf + * Pull patch in qla2xxx to Resolute (LP: #2144856) + - scsi: qla2xxx: Add support to report MPI FW state + * Ubuntu Resolute Desktop image arm64 - Boot on SC8280XP stalls with gpi-dma + errors (LP: #2142403) + - Revert "arm64: dts: qcom: sc8280xp: Enable GPI DMA" + * 26.04 Snapdragon X Elite: Sync concept kernel changes (LP: #2144643) + - SAUCE: arm64: dts: add missing denali-oled.dtb to Makefile + - SAUCE: dt-bindings: phy: qcom: Add CSI2 C-PHY/DPHY schema + - SAUCE: phy: qcom-mipi-csi2: Add a CSI2 MIPI DPHY driver + - SAUCE: dt-bindings: media: qcom,x1e80100-camss: Add simple-mfd + compatible + - SAUCE: dt-bindings: media: qcom,x1e80100-camss: Add optional PHY handle + definitions + - SAUCE: dt-bindings: media: qcom,x1e80100-camss: Add support for combo- + mode endpoints + - SAUCE: dt-bindings: media: qcom,x1e80100-camss: Describe iommu entries + - SAUCE: media: qcom: camss: Add legacy_phy flag to SoC definition + structures + - SAUCE: media: qcom: camss: Add support for PHY API devices + - SAUCE: media: qcom: camss: Drop legacy PHY descriptions from x1e + - SAUCE: arm64: dts: qcom: x1e80100: Add CAMCC block definition + - SAUCE: arm64: dts: qcom: x1e80100: Add CCI definitions + - SAUCE: arm64: dts: qcom: x1e80100: Add CAMSS block definition + - SAUCE: arm64: dts: qcom: x1e80100-crd: Add pm8010 CRD pmic,id=m + regulators + - SAUCE: arm64: dts: qcom: x1e80100-crd: Add ov08x40 RGB sensor on CSIPHY4 + - SAUCE: arm64: dts: qcom: x1e80100-t14s: Add pm8010 camera PMIC with + voltage levels for IR and RGB camera + - SAUCE: arm64: dts: qcom: x1e80100-t14s: Add on ov02c10 RGB sensor on + CSIPHY4 + - SAUCE: arm64: dts: qcom: x1e80100-lenovo-yoga-slim7x: Add pm8010 camera + PMIC with voltage levels for IR and RGB camera + - SAUCE: arm64: dts: qcom: x1e80100-lenovo-yoga-slim7x: Add l7b_2p8 + voltage regulator for RGB camera + - SAUCE: arm64: dts: qcom: x1e80100-lenovo-yoga-slim7x: Add ov02c10 RGB + sensor on CSIPHY4 + - SAUCE: arm64: dts: qcom: x1e80100-dell-inspiron14-7441: Switch on CAMSS + RGB sensor + - SAUCE: arm64: dts: qcom: x1-asus-zenbook-a14: Add on OV02C10 RGB sensor + on CSIPHY4 + - SAUCE: arm64: dts: qcom: x1e80100-dell-xps13-9345: add camera support + - SAUCE: arm64: dts: qcom: x1e78100-t14s: enable camera privacy indicator + - SAUCE: arm64: dts: qcom: x1e80100-lenovo-yoga-slim7x: enable camera + privacy indicator + - SAUCE: arm64: dts: qcom: x1e80100-dell-xps13-9345: enable camera privacy + indicator + - SAUCE: dt-bindings: arm: qcom: Add ASUS Vivobook X1P42100 variant + - SAUCE: arm64: dts: qcom: x1-vivobook-s15: create a common dtsi for Hamoa + and Purwa variants + - SAUCE: arm64: dts: qcom: x1-vivobook-s15: add Purwa-compatible device + tree + - SAUCE: firmware: qcom: scm: allow QSEECOM on ASUS Vivobook X1P42100 + variant + - SAUCE: arm64: dts: qcom: hamoa: Move PCIe PERST and Wake GPIOs to port + nodes + - SAUCE: arm64: dts: qcom: x1e-acer-swift-14: Move PCIe PERST and Wake + GPIOs to port nodes + * 25.10 Snapdragon X Elite: Sync concept kernel changes (LP: #2121477) + - SAUCE: wip: arm64: dts: qcom: x1e78100-t14s: enable bluetooth + * Miscellaneous Ubuntu changes + - SAUCE: dt-bindings: arm: qcom: Document HP EliteBook 6 G1q + - SAUCE: firmware: qcom: scm: Allow QSEECOM for HP EliteBook 6 G1q + - SAUCE: arm64: dts: qcom: x1p42100-hp-elitebook-6-g1q: DT for HP + EliteBook 6 G1q + - [Config] PHY_QCOM_MIPI_CSI2=m + - SAUCE: arm64: dts: x1e80100-lenovo-yoga-slim7x: Fix RGB camera supplies + - [Config] toolchain version update + - Update Changes.md after v7.0-rc5 rebase + - [Packaging] update Ubuntu.md + - [Config] enable SECURITY_APPARMOR_PACKET_MEDIATION_ENABLED + - [Packaging] Add linux-main-modules-zfs to linux-modules depends + * Miscellaneous upstream changes + - Revert "UBUNTU: SAUCE: Add Bluetooth support for the Lenovo Yoga Slim + 7x" + + -- Jacob Martin Tue, 07 Apr 2026 10:13:31 -0500 linux-nvidia-bos (7.0.0-2003.3) resolute; urgency=medium From a02740e1dc0bd3a9b29f12d566a6c8e91e51fa67 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 15 Apr 2026 09:56:04 -0500 Subject: [PATCH 145/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 738db0f1809a0..c0142a11d57a3 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2005.5) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Wed, 15 Apr 2026 09:56:04 -0500 + linux-nvidia-bos (7.0.0-2004.4) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2004.4 -proposed tracker (LP: #2146933) From 9cc858fb5baf1caa9cc57dcba80386d8874eae4f Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 15 Apr 2026 15:14:47 -0500 Subject: [PATCH 146/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2148362 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index 033a0348ec6c7..445d5824e700a 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2146933 d2026.03.17-1 +2148362 d2026.04.13-1 From a66b2edb64ba3f55074192afd13c88db2e3ee587 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 15 Apr 2026 22:18:32 -0500 Subject: [PATCH 147/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2005.5 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 63 +++++++++++++++++++++++++++++++---- debian.nvidia-bos/reconstruct | 45 +++++++++++++++++++++++++ 2 files changed, 102 insertions(+), 6 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index c0142a11d57a3..afcbeefd4ec40 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,61 @@ -linux-nvidia-bos (7.0.0-2005.5) UNRELEASED; urgency=medium - - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. +linux-nvidia-bos (7.0.0-2005.5) resolute; urgency=medium + + * resolute/linux-nvidia-bos: 7.0.0-2005.5 -proposed tracker (LP: #2148362) + + [ Ubuntu: 7.0.0-14.14 ] + + * resolute/linux: 7.0.0-14.14 -proposed tracker (LP: #2148159) + * support vflip/hflip for Sony IMX471 camera sensor (LP: #2138841) + - SAUCE: media: ipu-bridge: add TBE20A0 ACPI id for Sony IMX471 + * AA: disable SECURITY_APPARMOR_PACKET_MEDIATION_ENABLED (LP: #2147533) + - [Config] disable SECURITY_APPARMOR_PACKET_MEDIATION_ENABLED + * System doesn't response with mt76 call trace (LP: #2137448) + - wifi: mt76: mt792x: Fix a potential deadlock in high-load situations + * The second tbt storage plugged on the dock will not be recognized + (LP: #2139572) + - SAUCE: thunderbolt: Fix PCIe device enumeration with delayed rescan + * dma-buf filesystem flags fix (LP: #2139656) + - SAUCE: dma-buf: set SB_I_NOEXEC and SB_I_NODEV on dmabuf filesystem + * Bluetooth device (MT7925) not detected on USB bus with linux-oem-6.17 + (LP: #2145164) + - SAUCE: USB: hub: call ACPI _PRR reset during port power-cycle on + enumeration failure + * drm/i915/lnl+/tc: Fix false disconnect of active DP-alt TC port during + long HPD pulse (LP: #2143879) + - SAUCE: drm/i915/lnl+/tc: Fix false disconnect of active DP-alt TC port + during long HPD pulse + * i915 WARN_ON call trace during CB/WB on MTL/ARL platforms (LP: #2144537) + - SAUCE: drm/i915/xelpdp/tc: Convert TCSS power check WARN to a debug + message + * Miscellaneous Ubuntu changes + - [Packaging] Add support for per-flavour depends + - [Packaging] Don't hard-code lmm zfs dependency + - [Config] updateconfigs following v7.0 release + + [ Ubuntu: 7.0.0-13.13 ] + + * resolute/linux: 7.0.0-13.13 -proposed tracker (LP: #2147403) + * ubuntu_kselftests:_net/net:gre_gso.sh failing (LP: #2136820) + - SAUCE increase socat timeout in gre_gso.sh + * Canonical Kmod 2025 key rotation (LP: #2147447) + - [Packaging] ubuntu-compatible-signing -- make Ubuntu-Compatible-Signing + extensible + - [Packaging] ubuntu-compatible-signing -- allow consumption of positive + certs + - [Packaging] ubuntu-compatible-signing -- report the livepatch:2025 key + - [Config] prepare for Canonical Kmod key rotation + - [Packaging] ubuntu-compatible-signing -- report the kmod:2025 key + - [Packaging] ensure our cert rollups are always fresh + * On Dell system, the internal OLED display drops to a visibly low FPS after + suspend/resume (LP: #2144712) + - drm/i915/psr: Disable Panel Replay on Dell XPS 14 DA14260 as a quirk + - drm/i915/psr: Fixes for Dell XPS DA14260 quirk + * Realtek RTL8116AF SFP option module fails to get connected (LP: #2116144) + - SAUCE: r8169: add quirk for RTL8116af SerDes + * Miscellaneous Ubuntu changes + - [Config] updateconfigs following v7.0-rc7 rebase - -- Jacob Martin Wed, 15 Apr 2026 09:56:04 -0500 + -- Jacob Martin Wed, 15 Apr 2026 22:18:32 -0500 linux-nvidia-bos (7.0.0-2004.4) resolute; urgency=medium diff --git a/debian.nvidia-bos/reconstruct b/debian.nvidia-bos/reconstruct index ca916d098dabf..ef633a946371a 100644 --- a/debian.nvidia-bos/reconstruct +++ b/debian.nvidia-bos/reconstruct @@ -1 +1,46 @@ +# Recreate any symlinks created since the orig. +[ ! -L 'ubuntu/igh-ecat/master/rtdm-ioctl.c' ] && ln -sf 'ioctl.c' 'ubuntu/igh-ecat/master/rtdm-ioctl.c' +chmod +x 'debian/cloud-tools/hv_get_dhcp_info' +chmod +x 'debian/cloud-tools/hv_get_dns_info' +chmod +x 'debian/cloud-tools/hv_set_ifconfig' +chmod +x 'debian/rules' +chmod +x 'debian/scripts/checks/final-checks' +chmod +x 'debian/scripts/checks/module-signature-check' +chmod +x 'debian/scripts/control-create' +chmod +x 'debian/scripts/dkms-build' +chmod +x 'debian/scripts/dkms-build--nvidia-N' +chmod +x 'debian/scripts/dkms-build-configure--zfs' +chmod +x 'debian/scripts/file-downloader' +chmod +x 'debian/scripts/link-headers' +chmod +x 'debian/scripts/link-lib-rust' +chmod +x 'debian/scripts/misc/annotations' +chmod +x 'debian/scripts/misc/find-missing-sauce.sh' +chmod +x 'debian/scripts/misc/gen-auto-reconstruct' +chmod +x 'debian/scripts/misc/git-ubuntu-log' +chmod +x 'debian/scripts/misc/insert-changes' +chmod +x 'debian/scripts/misc/insert-ubuntu-changes' +chmod +x 'debian/scripts/misc/kernelconfig' +chmod +x 'debian/scripts/sign-module' +chmod +x 'debian/templates/extra.postinst.in' +chmod +x 'debian/templates/extra.postrm.in' +chmod +x 'debian/templates/headers.postinst.in' +chmod +x 'debian/templates/image.postinst.in' +chmod +x 'debian/templates/image.postrm.in' +chmod +x 'debian/templates/image.preinst.in' +chmod +x 'debian/templates/image.prerm.in' +chmod +x 'debian/tests/rebuild' +chmod +x 'debian/tests/ubuntu-regression-suite' +chmod +x 'drivers/net/ethernet/realtek/r8127/Makefile' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127_dash.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127_firmware.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127_n.c' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127_realwow.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/r8127_rss.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/rtl_eeprom.c' +chmod +x 'drivers/net/ethernet/realtek/r8127/rtl_eeprom.h' +chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.c' +chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.h' +# Remove any files deleted from the orig. +rm -f 'Documentation/admin-guide/perf/nvidia-pmu.rst' exit 0 From 6d26bdc8ec44cf5c16c161fafbc74d4c951c9aef Mon Sep 17 00:00:00 2001 From: Ankit Agrawal Date: Thu, 16 Apr 2026 01:45:04 +0000 Subject: [PATCH 148/464] NVIDIA: SAUCE: vfio/nvgrace-gpu: Add Blackwell-Next GPU readiness check via CXL DVSEC BugLink: https://bugs.launchpad.net/bugs/2148701 Add a CXL DVSEC-based readiness check for Blackwell-Next GPUs alongside the existing legacy BAR0 polling path. On probe and after reset, the driver reads the CXL Device DVSEC capability to determine whether the GPU memory is valid. This is checked by polling on the Memory_Active bit based on the Memory_Active_Timeout. Also check if MEM_INFO_VALID is set within 1 second per CXL spec 4.0 Tables 8-13. If not, return error. A static inline wrapper dispatches to the appropriate readiness check based on whether the CXL DVSEC capability is present. Add PCI_DVSEC_CXL_MEM_ACTIVE_TIMEOUT to pci_regs.h for the timeout field encoding. cc: Kevin Tian Suggested-by: Alex Williamson Signed-off-by: Ankit Agrawal (backported from https://lore.kernel.org/all/20260416014504.63067-1-ankita@nvidia.com/) [nirmoy: kept both egm_node (existing EGM SAUCE) and cxl_dvsec in struct to avoid conflict with EGM backport] Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/nvgrace-gpu/main.c | 96 ++++++++++++++++++++++++++--- include/uapi/linux/pci_regs.h | 1 + 2 files changed, 89 insertions(+), 8 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 967313b9e0029..8a1313534dfa9 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -66,6 +66,8 @@ struct nvgrace_gpu_pci_core_device { /* GPU has just been reset */ bool reset_done; int egm_node; + /* CXL Device DVSEC offset; 0 if not present (legacy GB path) */ + int cxl_dvsec; }; static bool egm_enabled; @@ -246,7 +248,7 @@ static void nvgrace_gpu_close_device(struct vfio_device *core_vdev) vfio_pci_core_close_device(core_vdev); } -static int nvgrace_gpu_wait_device_ready(void __iomem *io) +static int nvgrace_gpu_wait_device_ready_legacy(void __iomem *io) { unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS); @@ -260,6 +262,76 @@ static int nvgrace_gpu_wait_device_ready(void __iomem *io) return -ETIME; } +/* + * Decode the 3-bit Memory_Active_Timeout field from CXL DVSEC Range 1 Low + * (bits 15:13) into milliseconds. Encoding per CXL spec r4.0 sec 8.1.3.8.2: + * 000b = 1s, 001b = 4s, 010b = 16s, 011b = 64s, 100b = 256s, + * 101b-111b = reserved (clamped to 256s). + */ +static inline unsigned long cxl_mem_active_timeout_ms(u8 timeout) +{ + return 1000UL << (2 * min_t(u8, timeout, 4)); +} + +/* + * Check if CXL DVSEC reports memory as valid and active. + */ +static inline bool cxl_dvsec_mem_is_active(u32 status) +{ + return (status & PCI_DVSEC_CXL_MEM_INFO_VALID) && + (status & PCI_DVSEC_CXL_MEM_ACTIVE); +} + +static int nvgrace_gpu_wait_device_ready_cxl(struct nvgrace_gpu_pci_core_device *nvdev) +{ + struct pci_dev *pdev = nvdev->core_device.pdev; + int cxl_dvsec = nvdev->cxl_dvsec; + unsigned long mem_info_valid_deadline; + unsigned long timeout; + u32 dvsec_memory_status; + u8 mem_active_timeout; + + pci_read_config_dword(pdev, cxl_dvsec + PCI_DVSEC_CXL_RANGE_SIZE_LOW(0), + &dvsec_memory_status); + + if (cxl_dvsec_mem_is_active(dvsec_memory_status)) + return 0; + + mem_active_timeout = FIELD_GET(PCI_DVSEC_CXL_MEM_ACTIVE_TIMEOUT, + dvsec_memory_status); + + timeout = jiffies + + msecs_to_jiffies(cxl_mem_active_timeout_ms(mem_active_timeout)); + + mem_info_valid_deadline = jiffies + msecs_to_jiffies(POLL_QUANTUM_MS); + + do { + pci_read_config_dword(pdev, + cxl_dvsec + PCI_DVSEC_CXL_RANGE_SIZE_LOW(0), + &dvsec_memory_status); + + if (cxl_dvsec_mem_is_active(dvsec_memory_status)) + return 0; + + /* Bail early if MEM_INFO_VALID is not set within 1 second */ + if (!(dvsec_memory_status & PCI_DVSEC_CXL_MEM_INFO_VALID) && + time_after(jiffies, mem_info_valid_deadline)) + return -ETIME; + + msleep(POLL_QUANTUM_MS); + } while (!time_after(jiffies, timeout)); + + return -ETIME; +} + +static inline int nvgrace_gpu_wait_device_ready(struct nvgrace_gpu_pci_core_device *nvdev, + void __iomem *io) +{ + return nvdev->cxl_dvsec ? + nvgrace_gpu_wait_device_ready_cxl(nvdev) : + nvgrace_gpu_wait_device_ready_legacy(io); +} + /* * If the GPU memory is accessed by the CPU while the GPU is not ready * after reset, it can cause harmless corrected RAS events to be logged. @@ -279,7 +351,7 @@ nvgrace_gpu_check_device_ready(struct nvgrace_gpu_pci_core_device *nvdev) if (!__vfio_pci_memory_enabled(vdev)) return -EIO; - ret = nvgrace_gpu_wait_device_ready(vdev->barmap[0]); + ret = nvgrace_gpu_wait_device_ready(nvdev, vdev->barmap[0]); if (ret) return ret; @@ -1157,11 +1229,16 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) * Ensure that the BAR0 region is enabled before accessing the * registers. */ -static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev) +static int nvgrace_gpu_probe_check_device_ready(struct nvgrace_gpu_pci_core_device *nvdev) { + struct pci_dev *pdev = nvdev->core_device.pdev; void __iomem *io; int ret; + /* CXL path only reads PCI config space; no need to map BAR0. */ + if (nvdev->cxl_dvsec) + return nvgrace_gpu_wait_device_ready_cxl(nvdev); + ret = pci_enable_device(pdev); if (ret) return ret; @@ -1176,7 +1253,7 @@ static int nvgrace_gpu_probe_check_device_ready(struct pci_dev *pdev) goto iomap_exit; } - ret = nvgrace_gpu_wait_device_ready(io); + ret = nvgrace_gpu_wait_device_ready_legacy(io); pci_iounmap(pdev, io); iomap_exit: @@ -1195,10 +1272,6 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, u64 egmpxm; int ret; - ret = nvgrace_gpu_probe_check_device_ready(pdev); - if (ret) - return ret; - ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength); if (!ret) { ops = &nvgrace_gpu_pci_ops; @@ -1215,6 +1288,13 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, dev_set_drvdata(&pdev->dev, &nvdev->core_device); + nvdev->cxl_dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + + ret = nvgrace_gpu_probe_check_device_ready(nvdev); + if (ret) + goto out_put_vdev; + if (ops == &nvgrace_gpu_pci_ops) { nvdev->has_mig_hw_bug = nvgrace_gpu_has_mig_hw_bug(pdev); diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index c0371efe7f376..ea7c124490fdb 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1371,6 +1371,7 @@ #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0) #define PCI_DVSEC_CXL_MEM_ACTIVE _BITUL(1) +#define PCI_DVSEC_CXL_MEM_ACTIVE_TIMEOUT __GENMASK(15, 13) #define PCI_DVSEC_CXL_MEM_SIZE_LOW __GENMASK(31, 28) #define PCI_DVSEC_CXL_RANGE_BASE_HIGH(i) (0x20 + (i * 0x10)) #define PCI_DVSEC_CXL_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) From 86f65c280ad36aed78ce28ac94cb296c40c3953e Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Fri, 13 Mar 2026 09:55:03 -0700 Subject: [PATCH 149/464] NVIDIA: SAUCE: r8169: remove PCI IDs claimed by r8127 driver BugLink: https://bugs.launchpad.net/bugs/2144345 Remove device IDs 0x8127 and 0x0e10 from the r8169 PCI device table so that the r8127 vendor driver binds to these devices instead. Tested-by: Keith Berger Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/net/ethernet/realtek/r8169_main.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c index 4f27f2bbb391e..fbd0d7d903b50 100644 --- a/drivers/net/ethernet/realtek/r8169_main.c +++ b/drivers/net/ethernet/realtek/r8169_main.c @@ -241,10 +241,8 @@ static const struct pci_device_id rtl8169_pci_tbl[] = { { 0x0001, 0x8168, PCI_ANY_ID, 0x2410 }, { PCI_VDEVICE(REALTEK, 0x8125) }, { PCI_VDEVICE(REALTEK, 0x8126) }, - { PCI_VDEVICE(REALTEK, 0x8127) }, { PCI_VDEVICE(REALTEK, 0x3000) }, { PCI_VDEVICE(REALTEK, 0x5000) }, - { PCI_VDEVICE(REALTEK, 0x0e10) }, {} }; From 07e89fd1fc70cf679e7f79e44aeab9854a22369f Mon Sep 17 00:00:00 2001 From: Prathamesh Shete Date: Tue, 17 Feb 2026 08:14:30 +0000 Subject: [PATCH 150/464] gpio: tegra186: Simplify GPIO line name prefix handling BugLink: https://bugs.launchpad.net/bugs/2148664 Introduce TEGRA_GPIO_PREFIX() to define the Tegra SoC GPIO name prefix in one place. Use it for the Tegra410 COMPUTE and SYSTEM controllers so the prefix is "COMPUTE-" and "SYSTEM-" respectively. Signed-off-by: Prathamesh Shete Acked-by: Thierry Reding Reviewed-by: Jon Hunter Link: https://patch.msgid.link/20260217081431.1208351-1-pshete@nvidia.com Signed-off-by: Bartosz Golaszewski (cherry picked from commit 2423e336d94868f0d2fcd81a87b90c5ea59736e0) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/gpio/gpio-tegra186.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/drivers/gpio/gpio-tegra186.c b/drivers/gpio/gpio-tegra186.c index 9c874f07be755..f04cc240b5ec5 100644 --- a/drivers/gpio/gpio-tegra186.c +++ b/drivers/gpio/gpio-tegra186.c @@ -942,12 +942,8 @@ static int tegra186_gpio_probe(struct platform_device *pdev) char *name; for (j = 0; j < port->pins; j++) { - if (gpio->soc->prefix) - name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, "%s-P%s.%02x", - gpio->soc->prefix, port->name, j); - else - name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, "P%s.%02x", - port->name, j); + name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, "%sP%s.%02x", + gpio->soc->prefix ?: "", port->name, j); if (!name) return -ENOMEM; @@ -1373,6 +1369,9 @@ static const struct tegra_gpio_soc tegra256_main_soc = { .has_vm_support = true, }; +/* Macro to define GPIO name prefix with separator */ +#define TEGRA_GPIO_PREFIX(_x) _x "-" + #define TEGRA410_COMPUTE_GPIO_PORT(_name, _bank, _port, _pins) \ TEGRA_GPIO_PORT(TEGRA410_COMPUTE, _name, _bank, _port, _pins) @@ -1388,7 +1387,7 @@ static const struct tegra_gpio_soc tegra410_compute_soc = { .num_ports = ARRAY_SIZE(tegra410_compute_ports), .ports = tegra410_compute_ports, .name = "tegra410-gpio-compute", - .prefix = "COMPUTE", + .prefix = TEGRA_GPIO_PREFIX("COMPUTE"), .num_irqs_per_bank = 8, .instance = 0, }; @@ -1418,7 +1417,7 @@ static const struct tegra_gpio_soc tegra410_system_soc = { .num_ports = ARRAY_SIZE(tegra410_system_ports), .ports = tegra410_system_ports, .name = "tegra410-gpio-system", - .prefix = "SYSTEM", + .prefix = TEGRA_GPIO_PREFIX("SYSTEM"), .num_irqs_per_bank = 8, .instance = 0, }; From 16684cf28244fb21192b1b1f82012e0831046cd7 Mon Sep 17 00:00:00 2001 From: Prathamesh Shete Date: Tue, 17 Feb 2026 08:14:31 +0000 Subject: [PATCH 151/464] gpio: tegra186: Support multi-socket devices BugLink: https://bugs.launchpad.net/bugs/2148664 On Tegra platforms, multiple SoC instances may be present with each defining the same GPIO name. For such devices, this results in duplicate GPIO names. When the device has a valid NUMA node, prepend the NUMA node ID to the GPIO name prefix. The node ID identifies each socket, ensuring GPIO line names remain distinct across multiple sockets. Signed-off-by: Prathamesh Shete Acked-by: Thierry Reding Reviewed-by: Jon Hunter Link: https://patch.msgid.link/20260217081431.1208351-2-pshete@nvidia.com Signed-off-by: Bartosz Golaszewski (cherry picked from commit 2c299030c6813eaa9ef95773c64d65c50fa706ac) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/gpio/gpio-tegra186.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/gpio/gpio-tegra186.c b/drivers/gpio/gpio-tegra186.c index f04cc240b5ec5..fb26402b6c473 100644 --- a/drivers/gpio/gpio-tegra186.c +++ b/drivers/gpio/gpio-tegra186.c @@ -857,7 +857,7 @@ static int tegra186_gpio_probe(struct platform_device *pdev) struct device_node *np; struct resource *res; char **names; - int err; + int node, err; gpio = devm_kzalloc(&pdev->dev, sizeof(*gpio), GFP_KERNEL); if (!gpio) @@ -937,13 +937,23 @@ static int tegra186_gpio_probe(struct platform_device *pdev) if (!names) return -ENOMEM; + node = dev_to_node(&pdev->dev); + for (i = 0, offset = 0; i < gpio->soc->num_ports; i++) { const struct tegra_gpio_port *port = &gpio->soc->ports[i]; char *name; for (j = 0; j < port->pins; j++) { - name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, "%sP%s.%02x", - gpio->soc->prefix ?: "", port->name, j); + if (node >= 0) + name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, + "%d-%sP%s.%02x", node, + gpio->soc->prefix ?: "", + port->name, j); + else + name = devm_kasprintf(gpio->gpio.parent, GFP_KERNEL, + "%sP%s.%02x", + gpio->soc->prefix ?: "", + port->name, j); if (!name) return -ENOMEM; From 8d69cbf60681d5796cc586a35c02a9164484ef83 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Fri, 17 Apr 2026 05:30:06 -0700 Subject: [PATCH 152/464] Revert "NVIDIA: SAUCE: serial: 8250_mtk: Add ACPI support" BugLink: https://bugs.launchpad.net/bugs/2148664 This reverts commit 135705011943b89d4cb861472ef42db1991f133a. Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/tty/serial/8250/8250_mtk.c | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/drivers/tty/serial/8250/8250_mtk.c b/drivers/tty/serial/8250/8250_mtk.c index 39e8268cd4b9a..5875a7b9b4b10 100644 --- a/drivers/tty/serial/8250/8250_mtk.c +++ b/drivers/tty/serial/8250/8250_mtk.c @@ -19,7 +19,6 @@ #include #include #include -#include #include "8250.h" @@ -522,7 +521,6 @@ static int mtk8250_probe(struct platform_device *pdev) struct mtk8250_data *data; struct resource *regs; int irq, err; - struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev); irq = platform_get_irq(pdev, 0); if (irq < 0) @@ -545,13 +543,12 @@ static int mtk8250_probe(struct platform_device *pdev) data->clk_count = 0; - if (is_of_node(fwnode)) { + if (pdev->dev.of_node) { err = mtk8250_probe_of(pdev, &uart.port, data); if (err) return err; - } else if (!fwnode) { + } else return -ENODEV; - } spin_lock_init(&uart.port.lock); uart.port.mapbase = regs->start; @@ -567,18 +564,14 @@ static int mtk8250_probe(struct platform_device *pdev) uart.port.startup = mtk8250_startup; uart.port.set_termios = mtk8250_set_termios; uart.port.uartclk = clk_get_rate(data->uart_clk); - if (!uart.port.uartclk) - uart.port.uartclk = 26 * HZ_PER_MHZ; #ifdef CONFIG_SERIAL_8250_DMA if (data->dma) uart.dma = data->dma; #endif - if (is_of_node(fwnode)) { - /* Disable Rate Fix function */ - writel(0x0, uart.port.membase + + /* Disable Rate Fix function */ + writel(0x0, uart.port.membase + (MTK_UART_RATE_FIX << uart.port.regshift)); - } platform_set_drvdata(pdev, data); @@ -656,18 +649,11 @@ static const struct of_device_id mtk8250_of_match[] = { }; MODULE_DEVICE_TABLE(of, mtk8250_of_match); -static const struct acpi_device_id mtk8250_acpi_match[] = { - { "MTKI0511" }, - {} -}; -MODULE_DEVICE_TABLE(acpi, mtk8250_acpi_match); - static struct platform_driver mtk8250_platform_driver = { .driver = { .name = "mt6577-uart", .pm = &mtk8250_pm_ops, .of_match_table = mtk8250_of_match, - .acpi_match_table = mtk8250_acpi_match, }, .probe = mtk8250_probe, .remove = mtk8250_remove, From 4c0da21e99bc417d08d0019adaa128f1e10200ad Mon Sep 17 00:00:00 2001 From: "Zhiyong.Tao" Date: Mon, 5 Jan 2026 10:39:55 +0800 Subject: [PATCH 153/464] NVIDIA: SAUCE: MEDIATEK: serial: 8250_mtk: Add ACPI support BugLink: https://bugs.launchpad.net/bugs/2148664 Add ACPI support to 8250_mtk driver. This makes it possible to use UART on ARM-based desktops with EDK2 UEFI firmware. Signed-off-by: Yenchia Chen Signed-off-by: Zhiyong.Tao (backported from https://lore.kernel.org/all/20260105024103.2027085-2-zhiyong.tao@mediatek.com/) Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/tty/serial/8250/8250_mtk.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/tty/serial/8250/8250_mtk.c b/drivers/tty/serial/8250/8250_mtk.c index 5875a7b9b4b10..e6a56cf54ae0c 100644 --- a/drivers/tty/serial/8250/8250_mtk.c +++ b/drivers/tty/serial/8250/8250_mtk.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "8250.h" @@ -521,6 +522,7 @@ static int mtk8250_probe(struct platform_device *pdev) struct mtk8250_data *data; struct resource *regs; int irq, err; + struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev); irq = platform_get_irq(pdev, 0); if (irq < 0) @@ -543,12 +545,13 @@ static int mtk8250_probe(struct platform_device *pdev) data->clk_count = 0; - if (pdev->dev.of_node) { + if (is_of_node(fwnode)) { err = mtk8250_probe_of(pdev, &uart.port, data); if (err) return err; - } else + } else if (!fwnode) { return -ENODEV; + } spin_lock_init(&uart.port.lock); uart.port.mapbase = regs->start; @@ -564,14 +567,18 @@ static int mtk8250_probe(struct platform_device *pdev) uart.port.startup = mtk8250_startup; uart.port.set_termios = mtk8250_set_termios; uart.port.uartclk = clk_get_rate(data->uart_clk); + if (!uart.port.uartclk) + uart.port.uartclk = 26 * HZ_PER_MHZ; #ifdef CONFIG_SERIAL_8250_DMA if (data->dma) uart.dma = data->dma; #endif - /* Disable Rate Fix function */ - writel(0x0, uart.port.membase + + if (is_of_node(fwnode)) { + /* Disable Rate Fix function */ + writel(0x0, uart.port.membase + (MTK_UART_RATE_FIX << uart.port.regshift)); + } platform_set_drvdata(pdev, data); @@ -649,11 +656,19 @@ static const struct of_device_id mtk8250_of_match[] = { }; MODULE_DEVICE_TABLE(of, mtk8250_of_match); +static const struct acpi_device_id mtk8250_acpi_match[] = { + { "MTKI0511" }, + { "NVDA0240" }, + {} +}; +MODULE_DEVICE_TABLE(acpi, mtk8250_acpi_match); + static struct platform_driver mtk8250_platform_driver = { .driver = { .name = "mt6577-uart", .pm = &mtk8250_pm_ops, .of_match_table = mtk8250_of_match, + .acpi_match_table = mtk8250_acpi_match, }, .probe = mtk8250_probe, .remove = mtk8250_remove, From e75726a89e184bafd17fc733d2b99014b0ac7c12 Mon Sep 17 00:00:00 2001 From: Sourab Gupta Date: Thu, 20 Nov 2025 18:10:03 +0000 Subject: [PATCH 154/464] NVIDIA: SAUCE: Patch NVMe/NVMeoF driver to support GDS on Linux 7.0 Kernel BugLink: https://bugs.launchpad.net/bugs/2150289 BugLink: https://bugs.launchpad.net/bugs/2134960 With this change, the NVMe and NVMeoF driver would be enabled to support GPUDirectStorage(GDS). NVMe driver introduced a way to use the blk_rq_dma_map API to DMA map requests instead of scatter gather lists. With these changes, GDS path also adopts a similar framework where we introduce blk based APIs(nvfs_blk_rq_dma_map_iter_start and nvfs_blk_rq_dma_map_iter_next) to map a DMA request. The NVMeoF path remains the same as previous releases. Signed-off-by: Sourab Gupta Reviewed-by: Kiran Modukuri Acked-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/nvme/host/Makefile | 5 +- drivers/nvme/host/nvfs-dma.c | 51 +++++++++ drivers/nvme/host/nvfs-dma.h | 197 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvfs-rdma.c | 52 +++++++++ drivers/nvme/host/nvfs-rdma.h | 86 +++++++++++++++ drivers/nvme/host/nvfs.h | 156 +++++++++++++++++++++++++++ drivers/nvme/host/pci.c | 114 +++++++++++++++++++- drivers/nvme/host/rdma.c | 22 ++++ 8 files changed, 676 insertions(+), 7 deletions(-) create mode 100644 drivers/nvme/host/nvfs-dma.c create mode 100644 drivers/nvme/host/nvfs-dma.h create mode 100644 drivers/nvme/host/nvfs-rdma.c create mode 100644 drivers/nvme/host/nvfs-rdma.h create mode 100644 drivers/nvme/host/nvfs.h diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile index 6414ec968f99a..2fdd327bf6a88 100644 --- a/drivers/nvme/host/Makefile +++ b/drivers/nvme/host/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 ccflags-y += -I$(src) - +ccflags-y += -DCONFIG_NVFS obj-$(CONFIG_NVME_CORE) += nvme-core.o obj-$(CONFIG_BLK_DEV_NVME) += nvme.o obj-$(CONFIG_NVME_FABRICS) += nvme-fabrics.o @@ -20,10 +20,11 @@ nvme-core-$(CONFIG_NVME_HWMON) += hwmon.o nvme-core-$(CONFIG_NVME_HOST_AUTH) += auth.o nvme-y += pci.o - +nvme-y += nvfs-dma.o nvme-fabrics-y += fabrics.o nvme-rdma-y += rdma.o +nvme-rdma-y += nvfs-rdma.o nvme-fc-y += fc.o diff --git a/drivers/nvme/host/nvfs-dma.c b/drivers/nvme/host/nvfs-dma.c new file mode 100644 index 0000000000000..8d821e7b08846 --- /dev/null +++ b/drivers/nvme/host/nvfs-dma.c @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ +#ifdef CONFIG_NVFS +#define NVFS_USE_DMA_ITER_API +#define MODULE_PREFIX nvme_v2 +#include "nvfs.h" + +struct nvfs_dma_rw_blk_iter_ops *nvfs_ops = NULL; + +atomic_t nvfs_shutdown = ATOMIC_INIT(1); + +DEFINE_PER_CPU(long, nvfs_n_ops); + +#define NVIDIA_FS_COMPAT_FT(ops) \ + (NVIDIA_FS_CHECK_FT_BLK_DMA_MAP_ITER_START(ops) && NVIDIA_FS_CHECK_FT_BLK_DMA_MAP_ITER_NEXT(ops)) + +// protected via nvfs_module_mutex +int REGISTER_FUNC(struct nvfs_dma_rw_blk_iter_ops *ops) +{ + if (NVIDIA_FS_COMPAT_FT(ops)) { + nvfs_ops = ops; + atomic_set(&nvfs_shutdown, 0); + return 0; + } else + return -EOPNOTSUPP; + +} +EXPORT_SYMBOL_GPL(REGISTER_FUNC); + +// protected via nvfs_module_mutex +void UNREGISTER_FUNC(void) +{ + (void) atomic_cmpxchg(&nvfs_shutdown, 0, 1); + do { + msleep(NVFS_HOLD_TIME_MS); + } while(nvfs_count_ops()); + nvfs_ops = NULL; +} +EXPORT_SYMBOL_GPL(UNREGISTER_FUNC); +#endif diff --git a/drivers/nvme/host/nvfs-dma.h b/drivers/nvme/host/nvfs-dma.h new file mode 100644 index 0000000000000..6dc6654adb2a8 --- /dev/null +++ b/drivers/nvme/host/nvfs-dma.h @@ -0,0 +1,197 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ + +#ifndef NVFS_DMA_H +#define NVFS_DMA_H + +/* Forward declarations for functions from pci.c that we need */ +static blk_status_t nvme_pci_setup_data_prp(struct request *req, + struct blk_dma_iter *iter); +static blk_status_t nvme_pci_setup_data_sgl(struct request *req, + struct blk_dma_iter *iter); +static inline struct dma_pool *nvme_dma_pool(struct nvme_queue *nvmeq, + struct nvme_iod *iod); +static inline dma_addr_t nvme_pci_first_desc_dma_addr(struct nvme_command *cmd); + +static inline bool nvme_nvfs_unmap_sgls(struct request *req) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + struct nvme_queue *nvmeq = req->mq_hctx->driver_data; + struct device *dma_dev = nvmeq->dev->dev; + unsigned int sqe_dma_len = le32_to_cpu(iod->cmd.common.dptr.sgl.length); + struct nvme_sgl_desc *sg_list = iod->descriptors[0]; + enum dma_data_direction dir = rq_dma_dir(req); + + /* + * nr_descriptors == 0 means dma_pool_alloc failed before any SGL + * entries were recorded; the first iter mapping is handled by + * nvme_nvfs_map_data() directly, so nothing to unmap here. + */ + if (iod->nr_descriptors) { + unsigned int nr_entries = sqe_dma_len / sizeof(*sg_list), i; + + for (i = 0; i < nr_entries; i++) { + nvfs_ops->nvfs_dma_unmap_page(dma_dev, + iod->nvfs_cookie, + le64_to_cpu(sg_list[i].addr), + le32_to_cpu(sg_list[i].length), + dir); + } + } + + return true; +} + +static inline bool nvme_nvfs_unmap_prps(struct request *req) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + struct nvme_queue *nvmeq = req->mq_hctx->driver_data; + struct device *dma_dev = nvmeq->dev->dev; + enum dma_data_direction dma_dir = rq_dma_dir(req); + unsigned int i; + + /* Check if dma_vecs was allocated - if setup failed early, it might be NULL */ + if (!iod->dma_vecs) + return true; + + /* Unmap all DMA vectors - pass page pointer from dma_vecs */ + for (i = 0; i < iod->nr_dma_vecs; i++) { + nvfs_ops->nvfs_dma_unmap_page(dma_dev, + iod->nvfs_cookie, + iod->dma_vecs[i].addr, + iod->dma_vecs[i].len, + dma_dir); + } + + /* Free the dma_vecs mempool allocation */ + mempool_free(iod->dma_vecs, nvmeq->dev->dmavec_mempool); + iod->dma_vecs = NULL; + iod->nr_dma_vecs = 0; + + return true; +} + +static inline void nvme_nvfs_free_descriptors(struct request *req) +{ + struct nvme_queue *nvmeq = req->mq_hctx->driver_data; + const int last_prp = NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + dma_addr_t dma_addr = nvme_pci_first_desc_dma_addr(&iod->cmd); + int i; + + if (iod->nr_descriptors == 1) { + dma_pool_free(nvme_dma_pool(nvmeq, iod), iod->descriptors[0], + dma_addr); + return; + } + + for (i = 0; i < iod->nr_descriptors; i++) { + __le64 *prp_list = iod->descriptors[i]; + dma_addr_t next_dma_addr = le64_to_cpu(prp_list[last_prp]); + + dma_pool_free(nvmeq->descriptor_pools.large, prp_list, + dma_addr); + dma_addr = next_dma_addr; + } +} + +static inline bool nvme_nvfs_unmap_data(struct request *req) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + bool ret; + + /* Check if this was an NVFS I/O by checking the IOD_NVFS_IO flag */ + if (!(iod->flags & IOD_NVFS_IO)) + return false; + + /* Clear the NVFS flag */ + iod->flags &= ~IOD_NVFS_IO; + + /* Call appropriate unmap function based on command type */ + if (nvme_pci_cmd_use_sgl(&iod->cmd)) + ret = nvme_nvfs_unmap_sgls(req); + else + ret = nvme_nvfs_unmap_prps(req); + + if (iod->nr_descriptors) + nvme_nvfs_free_descriptors(req); + + nvfs_put_ops(); + return ret; +} + +static inline blk_status_t nvme_nvfs_map_data(struct request *req, + bool *is_nvfs_io) +{ + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + struct nvme_queue *nvmeq = req->mq_hctx->driver_data; + struct nvme_dev *dev = nvmeq->dev; + struct device *dma_dev = nvmeq->dev->dev; + enum nvme_use_sgl use_sgl = nvme_pci_use_sgls(dev, req); + struct blk_dma_iter iter; + blk_status_t ret = BLK_STS_RESOURCE; + + *is_nvfs_io = false; + + /* Check integrity and try to get nvfs_ops */ + if (blk_integrity_rq(req) || !nvfs_get_ops()) { + return ret; + } + + /* Initialize total_len for this request */ + iod->total_len = 0; + + if (!nvfs_ops->nvfs_blk_rq_dma_map_iter_start(req, dma_dev, + &iod->dma_state, &iter, &iod->nvfs_cookie)) { + nvfs_put_ops(); + if (iter.status == BLK_STS_IOERR) { + /* GPU DMA error — do not fall through to CPU path */ + *is_nvfs_io = true; + ret = iter.status; + } + /* else: CPU page, let caller fall through to CPU path */ + return ret; + } + + /* NVFS can handle this request, set the flag */ + *is_nvfs_io = true; + iod->flags |= IOD_NVFS_IO; + + if (use_sgl == SGL_FORCED || + (use_sgl == SGL_SUPPORTED && + (sgl_threshold && nvme_pci_avg_seg_size(req) >= sgl_threshold))) + ret = nvme_pci_setup_data_sgl(req, &iter); + else + ret = nvme_pci_setup_data_prp(req, &iter); + + /* If setup failed, cleanup: unmap DMA, clear flag, release ops */ + if (ret != BLK_STS_OK) { + /* + * If setup failed before any mappings were tracked (dma_vecs is + * NULL for PRP, or nr_descriptors is 0 for SGL), the first page + * mapped by nvfs_blk_rq_dma_map_iter_start() won't be covered by + * nvme_nvfs_unmap_data(). Unmap it directly using iter. + */ + bool early_fail = nvme_pci_cmd_use_sgl(&iod->cmd) ? + !iod->nr_descriptors : !iod->dma_vecs; + if (early_fail) + nvfs_ops->nvfs_dma_unmap_page(dma_dev, iod->nvfs_cookie, + iter.addr, iter.len, rq_dma_dir(req)); + nvme_nvfs_unmap_data(req); + } + + return ret; +} + +#endif /* NVFS_DMA_H */ diff --git a/drivers/nvme/host/nvfs-rdma.c b/drivers/nvme/host/nvfs-rdma.c new file mode 100644 index 0000000000000..75a269143f3a8 --- /dev/null +++ b/drivers/nvme/host/nvfs-rdma.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ + +#ifdef CONFIG_NVFS +#define MODULE_PREFIX nvme_rdma_v1 +#include "nvfs.h" + +struct nvfs_dma_rw_ops *nvfs_ops; + +atomic_t nvfs_shutdown = ATOMIC_INIT(1); + +DEFINE_PER_CPU(long, nvfs_n_ops); + +// must have for compatability +#define NVIDIA_FS_COMPAT_FT(ops) \ + (NVIDIA_FS_CHECK_FT_SGLIST_PREP(ops) && NVIDIA_FS_CHECK_FT_SGLIST_DMA(ops)) + +// protected via nvfs_module_mutex +int REGISTER_FUNC(struct nvfs_dma_rw_ops *ops) +{ + if (NVIDIA_FS_COMPAT_FT(ops)) { + nvfs_ops = ops; + atomic_set(&nvfs_shutdown, 0); + return 0; + } else + return -EOPNOTSUPP; + +} +EXPORT_SYMBOL_GPL(REGISTER_FUNC); + +// protected via nvfs_module_mutex +void UNREGISTER_FUNC(void) +{ + (void) atomic_cmpxchg(&nvfs_shutdown, 0, 1); + do { + msleep(NVFS_HOLD_TIME_MS); + } while(nvfs_count_ops()); + nvfs_ops = NULL; +} +EXPORT_SYMBOL_GPL(UNREGISTER_FUNC); +#endif diff --git a/drivers/nvme/host/nvfs-rdma.h b/drivers/nvme/host/nvfs-rdma.h new file mode 100644 index 0000000000000..f9051e2ab22b3 --- /dev/null +++ b/drivers/nvme/host/nvfs-rdma.h @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ + +#ifndef NVFS_RDMA_H +#define NVFS_RDMA_H + +static bool nvme_rdma_nvfs_unmap_data(struct ib_device *ibdev, + struct request *rq) + +{ + struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); + enum dma_data_direction dma_dir = rq_dma_dir(rq); + int count; + + if (!blk_integrity_rq(rq) && nvfs_ops != NULL) { + count = nvfs_ops->nvfs_dma_unmap_sg(ibdev->dma_device, req->data_sgl.sg_table.sgl, req->data_sgl.nents, + dma_dir); + if (count) { + nvfs_put_ops(); + sg_free_table_chained(&req->data_sgl.sg_table, NVME_INLINE_SG_CNT); + return true; + } + } + return false; +} + +static int nvme_rdma_nvfs_map_data(struct ib_device *ibdev, struct request *rq, bool *is_nvfs_io, int* count) +{ + struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); + enum dma_data_direction dma_dir = rq_dma_dir(rq); + int ret = 0; + + *is_nvfs_io = false; + *count = 0; + if (!blk_integrity_rq(rq) && nvfs_get_ops()) { + + // associates bio pages to scatterlist + *count = nvfs_ops->nvfs_blk_rq_map_sg(rq->q, rq , req->data_sgl.sg_table.sgl); + if (!*count) { + nvfs_put_ops(); + return 0; // fall to cpu path + } + + *is_nvfs_io = true; + if (unlikely((*count == NVFS_IO_ERR))) { + nvfs_put_ops(); + pr_err("%s: failed to map sg_nents=:%d\n", __func__, req->data_sgl.nents); + return -EIO; + } + req->data_sgl.nents = *count; + + *count = nvfs_ops->nvfs_dma_map_sg_attrs(ibdev->dma_device, + req->data_sgl.sg_table.sgl, + req->data_sgl.nents, + dma_dir, + DMA_ATTR_NO_WARN); + + if (unlikely((*count == NVFS_IO_ERR))) { + nvfs_put_ops(); + return -EIO; + } + + if (unlikely(*count == NVFS_CPU_REQ)) { + nvfs_put_ops(); + return -EIO; + } + + return ret; + } + + // Fall to CPU path + return 0; +} + +#endif diff --git a/drivers/nvme/host/nvfs.h b/drivers/nvme/host/nvfs.h new file mode 100644 index 0000000000000..0101a88dcc6c7 --- /dev/null +++ b/drivers/nvme/host/nvfs.h @@ -0,0 +1,156 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + */ +#ifndef NVFS_H +#define NVFS_H + +#include +#include +#include +#include +#include +#include +#include + +/* Forward declarations */ +struct blk_dma_iter; +struct dma_iova_state; + +#define REGSTR2(x) x##_register_nvfs_dma_ops +#define REGSTR(x) REGSTR2(x) + +#define UNREGSTR2(x) x##_unregister_nvfs_dma_ops +#define UNREGSTR(x) UNREGSTR2(x) + +#define REGISTER_FUNC REGSTR(MODULE_PREFIX) +#define UNREGISTER_FUNC UNREGSTR(MODULE_PREFIX) + +#define NVFS_IO_ERR -1 +#define NVFS_CPU_REQ -2 + +#define NVFS_HOLD_TIME_MS 1000 + +#ifdef NVFS_USE_DMA_ITER_API +extern struct nvfs_dma_rw_blk_iter_ops *nvfs_ops; +#else +extern struct nvfs_dma_rw_ops *nvfs_ops; +#endif + +extern atomic_t nvfs_shutdown; + +DECLARE_PER_CPU(long, nvfs_n_ops); + +static inline long nvfs_count_ops(void) +{ + int i; + long sum = 0; + for_each_possible_cpu(i) + sum += per_cpu(nvfs_n_ops, i); + return sum; +} + +static inline bool nvfs_get_ops(void) +{ + if (nvfs_ops && !atomic_read(&nvfs_shutdown)) { + this_cpu_inc(nvfs_n_ops); + return true; + } + return false; +} + +static inline void nvfs_put_ops(void) +{ + this_cpu_dec(nvfs_n_ops); +} + + +struct nvfs_dma_rw_blk_iter_ops { + unsigned long long ft_bmap; // feature bitmap + + int (*nvfs_blk_rq_dma_map_iter_start) (struct request *req, + struct device *dma_dev, + struct dma_iova_state *state, + struct blk_dma_iter *iter, + void **cookie); + + int (*nvfs_blk_rq_dma_map_iter_next) (struct request *req, + struct device *dma_dev, + struct dma_iova_state *state, + struct blk_dma_iter *iter); + + int (*nvfs_dma_unmap_page) (struct device *device, + void* cookie, + dma_addr_t addr, + size_t size, + enum dma_data_direction dir); + + bool (*nvfs_is_gpu_page) (struct page *page); + + unsigned int (*nvfs_gpu_index) (struct page *page); + + unsigned int (*nvfs_device_priority) (struct device *dev, unsigned int gpu_index); + +}; + +struct nvfs_dma_rw_ops { + unsigned long long ft_bmap; // feature bitmap + + int (*nvfs_blk_rq_map_sg) (struct request_queue *q, + struct request *req, + struct scatterlist *sglist); + + int (*nvfs_dma_map_sg_attrs) (struct device *device, + struct scatterlist *sglist, + int nents, + enum dma_data_direction dma_dir, + unsigned long attrs); + + int (*nvfs_dma_unmap_sg) (struct device *device, + struct scatterlist *sglist, + int nents, + enum dma_data_direction dma_dir); + + bool (*nvfs_is_gpu_page) (struct page *page); + + unsigned int (*nvfs_gpu_index) (struct page *page); + + unsigned int (*nvfs_device_priority) (struct device *dev, unsigned int gpu_index); +}; + +// feature list for dma_ops, values indicate bit pos +enum ft_bits { + nvfs_ft_prep_sglist = 1ULL << 0, + nvfs_ft_map_sglist = 1ULL << 1, + nvfs_ft_is_gpu_page = 1ULL << 2, + nvfs_ft_device_priority = 1ULL << 3, + nvfs_ft_blk_dma_map_iter_start = 1ULL << 5, + nvfs_ft_blk_dma_map_iter_next = 1ULL << 6, +}; + +// check features for use in registration with vendor drivers +#define NVIDIA_FS_CHECK_FT_SGLIST_PREP(ops) ((ops)->ft_bmap & nvfs_ft_prep_sglist) +#define NVIDIA_FS_CHECK_FT_SGLIST_DMA(ops) ((ops)->ft_bmap & nvfs_ft_map_sglist) +#define NVIDIA_FS_CHECK_FT_GPU_PAGE(ops) ((ops)->ft_bmap & nvfs_ft_is_gpu_page) +#define NVIDIA_FS_CHECK_FT_DEVICE_PRIORITY(ops) ((ops)->ft_bmap & nvfs_ft_device_priority) +#define NVIDIA_FS_CHECK_FT_BLK_DMA_MAP_ITER_START(ops) ((ops)->ft_bmap & nvfs_ft_blk_dma_map_iter_start) +#define NVIDIA_FS_CHECK_FT_BLK_DMA_MAP_ITER_NEXT(ops) ((ops)->ft_bmap & nvfs_ft_blk_dma_map_iter_next) + +#ifdef NVFS_USE_DMA_ITER_API +int REGISTER_FUNC(struct nvfs_dma_rw_blk_iter_ops *ops); +#else +int REGISTER_FUNC(struct nvfs_dma_rw_ops *ops); +#endif + +void UNREGISTER_FUNC(void); + +#endif /* NVFS_H */ diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index a0e9767bc21e6..3729edcc33562 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -30,6 +30,10 @@ #include "trace.h" #include "nvme.h" +#ifdef CONFIG_NVFS +#define NVFS_USE_DMA_ITER_API +#include "nvfs.h" +#endif #define SQ_SIZE(q) ((q)->q_depth << (q)->sqes) #define CQ_SIZE(q) ((q)->q_depth * sizeof(struct nvme_completion)) @@ -37,6 +41,15 @@ /* Optimisation for I/Os between 4k and 128k */ #define NVME_SMALL_POOL_SIZE 256 +#ifdef CONFIG_NVFS +/* GPU physical pages are minimum 64K. Worst-case SGL entries with misalignment: + * ceil(payload/64K) + 1. Safe payload for small pool = (entries - 1) * 64K, + * where entries = NVME_SMALL_POOL_SIZE / sizeof(struct nvme_sgl_desc). */ +#define NVFS_GPU_PAGE_SIZE (64UL * 1024) +#define NVFS_SMALL_POOL_PAYLOAD \ + ((NVME_SMALL_POOL_SIZE / sizeof(struct nvme_sgl_desc) - 1) * NVFS_GPU_PAGE_SIZE) +#endif + /* * Arbitrary upper bound. */ @@ -418,6 +431,11 @@ enum nvme_iod_flags { /* Metadata using non-coalesced MPTR */ IOD_SINGLE_META_SEGMENT = 1U << 7, + +#ifdef CONFIG_NVFS + /* NVFS GPU Direct Storage I/O */ + IOD_NVFS_IO = 1U << 31, +#endif }; struct nvme_dma_vec { @@ -431,7 +449,11 @@ struct nvme_dma_vec { struct nvme_iod { struct nvme_request req; struct nvme_command cmd; +#ifdef CONFIG_NVFS + u32 flags; +#else u8 flags; +#endif u8 nr_descriptors; size_t total_len; @@ -444,6 +466,9 @@ struct nvme_iod { size_t meta_total_len; struct dma_iova_state meta_dma_state; struct nvme_sgl_desc *meta_descriptor; +#ifdef CONFIG_NVFS + void *nvfs_cookie; +#endif }; static inline unsigned int nvme_dbbuf_size(struct nvme_dev *dev) @@ -924,6 +949,10 @@ static void nvme_unmap_metadata(struct request *req) iod->meta_descriptor, iod->meta_dma); } +#ifdef CONFIG_NVFS +#include "nvfs-dma.h" +#endif + static void nvme_unmap_data(struct request *req) { enum pci_p2pdma_map_type map = PCI_P2PDMA_MAP_NONE; @@ -932,6 +961,12 @@ static void nvme_unmap_data(struct request *req) struct device *dma_dev = nvmeq->dev->dev; unsigned int attrs = 0; +#ifdef CONFIG_NVFS + /* Check if this was an NVFS I/O and handle unmapping */ + if (nvme_nvfs_unmap_data(req)) + return; +#endif + if (iod->flags & IOD_SINGLE_SEGMENT) { static_assert(offsetof(union nvme_data_ptr, prp1) == offsetof(union nvme_data_ptr, sgl.addr)); @@ -992,6 +1027,20 @@ static bool nvme_pci_prp_iter_next(struct request *req, struct device *dma_dev, { if (iter->len) return true; +#ifdef CONFIG_NVFS + { + struct nvme_iod *iod = blk_mq_rq_to_pdu(req); + if (iod->flags & IOD_NVFS_IO) { + if (!nvfs_ops->nvfs_blk_rq_dma_map_iter_next(req, dma_dev, + &iod->dma_state, iter)) + return false; + iod->dma_vecs[iod->nr_dma_vecs].addr = iter->addr; + iod->dma_vecs[iod->nr_dma_vecs].len = iter->len; + iod->nr_dma_vecs++; + return true; + } + } +#endif if (!blk_rq_dma_map_iter_next(req, dma_dev, iter)) return false; return nvme_pci_prp_save_mapping(req, dma_dev, iter); @@ -1024,6 +1073,16 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req, unsigned int prp_len, i; __le64 *prp_list; +#ifdef CONFIG_NVFS + if (iod->flags & IOD_NVFS_IO) { + iod->dma_vecs = mempool_alloc(nvmeq->dev->dmavec_mempool, GFP_ATOMIC); + if (!iod->dma_vecs) + return BLK_STS_RESOURCE; + iod->dma_vecs[0].addr = iter->addr; + iod->dma_vecs[0].len = iter->len; + iod->nr_dma_vecs = 1; + } else +#endif if (!nvme_pci_prp_save_mapping(req, nvmeq->dev->dev, iter)) { nvme_unmap_iter(req, iter, &iod->dma_state); return iter->status; @@ -1124,6 +1183,11 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req, */ iod->cmd.common.dptr.prp1 = cpu_to_le64(prp1_dma); iod->cmd.common.dptr.prp2 = cpu_to_le64(prp2_dma); +#ifdef CONFIG_NVFS + /* For NVFS, don't call nvme_unmap_data - cleanup happens in nvme_nvfs_unmap_data */ + if (iod->flags & IOD_NVFS_IO) + return iter->status; +#endif if (unlikely(iter->status)) nvme_unmap_data(req); return iter->status; @@ -1165,12 +1229,34 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req, /* set the transfer type as SGL */ iod->cmd.common.flags = NVME_CMD_SGL_METABUF; - if (entries == 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) { - nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter); - iod->total_len += iter->len; - return BLK_STS_OK; +#ifdef CONFIG_NVFS + if (!(iod->flags & IOD_NVFS_IO)) +#endif + { + if (entries == 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) { + nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter); + iod->total_len += iter->len; + return BLK_STS_OK; + } } +#ifdef CONFIG_NVFS + if (iod->flags & IOD_NVFS_IO) { + /* + * blk_rq_nr_phys_segments() reflects shadow buffer contiguity, + * not GPU physical segments. GPU pages are 64K minimum; worst-case + * entries with misalignment = ceil(payload/64K) + 1. + * Small pool (16 entries) is safe for payload < NVFS_SMALL_POOL_PAYLOAD. + * Large pool capacity = NVME_CTRL_PAGE_SIZE / sizeof(*sg_list) = 256. + */ + if (blk_rq_payload_bytes(req) < NVFS_SMALL_POOL_PAYLOAD) { + entries = NVME_SMALL_POOL_SIZE / sizeof(*sg_list); + iod->flags |= IOD_SMALL_DESCRIPTOR; + } else { + entries = NVME_CTRL_PAGE_SIZE / sizeof(*sg_list); + } + } else +#endif if (entries <= NVME_SMALL_POOL_SIZE / sizeof(*sg_list)) iod->flags |= IOD_SMALL_DESCRIPTOR; @@ -1190,9 +1276,21 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req, } nvme_pci_sgl_set_data(&sg_list[mapped++], iter); iod->total_len += iter->len; - } while (blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter)); + } while ( +#ifdef CONFIG_NVFS + (iod->flags & IOD_NVFS_IO) ? + (mapped < entries && + nvfs_ops->nvfs_blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, + &iod->dma_state, iter)) : +#endif + blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter)); nvme_pci_sgl_set_seg(&iod->cmd.common.dptr.sgl, sgl_dma, mapped); +#ifdef CONFIG_NVFS + /* For NVFS, don't call nvme_unmap_data - cleanup happens in nvme_nvfs_unmap_data */ + if (iod->flags & IOD_NVFS_IO) + return iter->status; +#endif if (unlikely(iter->status)) nvme_unmap_data(req); return iter->status; @@ -1246,6 +1344,12 @@ static blk_status_t nvme_map_data(struct request *req) struct blk_dma_iter iter; blk_status_t ret; +#ifdef CONFIG_NVFS + bool is_nvfs_io = false; + ret = nvme_nvfs_map_data(req, &is_nvfs_io); + if (is_nvfs_io) + return ret; +#endif /* * Try to skip the DMA iterator for single segment requests, as that * significantly improves performances for small I/O sizes. diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 57111139e84fa..53b4823d57c5f 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -27,6 +27,9 @@ #include "nvme.h" #include "fabrics.h" +#ifdef CONFIG_NVFS +#include "nvfs.h" +#endif #define NVME_RDMA_CM_TIMEOUT_MS 3000 /* 3 second */ @@ -1212,6 +1215,9 @@ static int nvme_rdma_inv_rkey(struct nvme_rdma_queue *queue, return ib_post_send(queue->qp, &wr, NULL); } +#ifdef CONFIG_NVFS +#include "nvfs-rdma.h" +#endif static void nvme_rdma_dma_unmap_req(struct ib_device *ibdev, struct request *rq) { struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); @@ -1223,6 +1229,11 @@ static void nvme_rdma_dma_unmap_req(struct ib_device *ibdev, struct request *rq) NVME_INLINE_METADATA_SG_CNT); } +#ifdef CONFIG_NVFS + if (nvme_rdma_nvfs_unmap_data(ibdev, rq)) + return; +#endif + ib_dma_unmap_sg(ibdev, req->data_sgl.sg_table.sgl, req->data_sgl.nents, rq_dma_dir(rq)); sg_free_table_chained(&req->data_sgl.sg_table, NVME_INLINE_SG_CNT); @@ -1476,6 +1487,17 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, if (ret) return -ENOMEM; +#ifdef CONFIG_NVFS + { + bool is_nvfs_io = false; + ret = nvme_rdma_nvfs_map_data(ibdev, rq, &is_nvfs_io, count); + if (is_nvfs_io) { + if (ret) + goto out_free_table; + return 0; + } + } +#endif req->data_sgl.nents = blk_rq_map_sg(rq, req->data_sgl.sg_table.sgl); *count = ib_dma_map_sg(ibdev, req->data_sgl.sg_table.sgl, From a25c0eb80cb81ea87815deb4d594b154a0f8cbfd Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Wed, 15 Apr 2026 11:03:03 +0800 Subject: [PATCH 155/464] NVIDIA: SAUCE: iommu/arm-smmu-v3: Use identity domain for ASPEED BMC devices BugLink: https://bugs.launchpad.net/bugs/2150470 ASPEED BMC devices behind an AST1150 PCIe-to-PCI bridge receive DMA from BMC firmware using host physical addresses that bypass the kernel's DMA API entirely. When these devices are assigned a DMA translated domain, the SMMU generates F_TRANSLATION faults because the BMC's physical addresses have no corresponding IOVA mappings in the SMMU page tables. Fix this by returning IOMMU_DOMAIN_IDENTITY for PCI devices whose parent bridge has both the PCI_BRIDGE_NO_ALIASES flag and an ASPEED vendor ID, so the SMMU passes BMC DMA transactions through untranslated. Signed-off-by: Koba Ko (backported from commit 738fff0e2060b6b383c21afdf1366330d9c79698 linux-nvidia-6.17) [koba: rename PCI_DEV_FLAGS_PCI_BRIDGE_NO_ALIASES -> PCI_DEV_FLAGS_PCI_BRIDGE_NO_ALIAS; Nirmoy's upstream AST1150 NO_ALIAS quirk in 7.0 uses the singular form (bit 14 in include/linux/pci.h) vs the plural form (bit 15) in 6.17-next] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 9f3829d5e3028..de07e00ff5ac0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3791,6 +3791,19 @@ static int arm_smmu_def_domain_type(struct device *dev) if (IS_HISI_PTT_DEVICE(pdev)) return IOMMU_DOMAIN_IDENTITY; + /* + * ASPEED BMC devices behind an AST1150 PCIe-to-PCI bridge + * receive DMA from BMC firmware using host physical addresses + * that bypass the kernel DMA API. Use identity mapping so + * the SMMU passes these transactions through untranslated. + */ + if (pdev->bus->self && + (pdev->bus->self->dev_flags & + PCI_DEV_FLAGS_PCI_BRIDGE_NO_ALIAS) && + pdev->bus->self->vendor == PCI_VENDOR_ID_ASPEED && + pdev->bus->self->device == 0x1150) + return IOMMU_DOMAIN_IDENTITY; + if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && (pdev->device == 0x2E12 || pdev->device == 0x2E2A || pdev->device == 0x2E2B)) From 7756f46af20a8718592e4e8308f721d4fc5078d8 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Thu, 12 Mar 2026 09:12:02 -0700 Subject: [PATCH 156/464] workqueue: fix parse_affn_scope() prefix matching bug BugLink: https://bugs.launchpad.net/bugs/2150467 parse_affn_scope() uses strncasecmp() with the length of the candidate name, which means it only checks if the input *starts with* a known scope name. Given that the upcoming diff will create "cache_shard" affinity scope, writing "cache_shard" to a workqueue's affinity_scope sysfs attribute always matches "cache" first, making it impossible to select "cache_shard" via sysfs, so, this fix enable it to distinguish "cache" and "cache_shard" Fix by replacing the hand-rolled prefix matching loop with sysfs_match_string(), which uses sysfs_streq() for exact matching (modulo trailing newlines). Also add the missing const qualifier to the wq_affn_names[] array declaration. Note that sysfs_streq() is case-sensitive, unlike the previous strncasecmp() approach. This is intentional and consistent with how other sysfs attributes handle string matching in the kernel. Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 1abaae9b38a85c9dabff67a22d8c99f7254c423a) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- kernel/workqueue.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 23053ef54162e..441916a6d11c8 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -405,7 +405,7 @@ struct work_offq_data { u32 flags; }; -static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { +static const char * const wq_affn_names[WQ_AFFN_NR_TYPES] = { [WQ_AFFN_DFL] = "default", [WQ_AFFN_CPU] = "cpu", [WQ_AFFN_SMT] = "smt", @@ -7119,13 +7119,7 @@ int workqueue_unbound_housekeeping_update(const struct cpumask *hk) static int parse_affn_scope(const char *val) { - int i; - - for (i = 0; i < ARRAY_SIZE(wq_affn_names); i++) { - if (!strncasecmp(val, wq_affn_names[i], strlen(wq_affn_names[i]))) - return i; - } - return -EINVAL; + return sysfs_match_string(wq_affn_names, val); } static int wq_affn_dfl_set(const char *val, const struct kernel_param *kp) From b4083df336079e0f5d809aa58e1eeb413abdded3 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:52 -0700 Subject: [PATCH 157/464] workqueue: fix typo in WQ_AFFN_SMT comment BugLink: https://bugs.launchpad.net/bugs/2150467 Fix "poer" -> "per" in the WQ_AFFN_SMT enum comment. Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 9dc42c9070282c81058a875fea5acae057610980) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- include/linux/workqueue.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 218557a8f7577..7f839049727ee 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -131,7 +131,7 @@ struct rcu_work { enum wq_affn_scope { WQ_AFFN_DFL, /* use system default */ WQ_AFFN_CPU, /* one pod per CPU */ - WQ_AFFN_SMT, /* one pod poer SMT */ + WQ_AFFN_SMT, /* one pod per SMT */ WQ_AFFN_CACHE, /* one pod per LLC */ WQ_AFFN_NUMA, /* one pod per NUMA node */ WQ_AFFN_SYSTEM, /* one pod across the whole system */ From 2d4aed2b0d301b8522ad837e2163009ce32e8715 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:53 -0700 Subject: [PATCH 158/464] workqueue: add WQ_AFFN_CACHE_SHARD affinity scope BugLink: https://bugs.launchpad.net/bugs/2150467 On systems where many CPUs share one LLC, unbound workqueues using WQ_AFFN_CACHE collapse to a single worker pool, causing heavy spinlock contention on pool->lock. For example, Chuck Lever measured 39% of cycles lost to native_queued_spin_lock_slowpath on a 12-core shared-L3 NFS-over-RDMA system. The existing affinity hierarchy (cpu, smt, cache, numa, system) offers no intermediate option between per-LLC and per-SMT-core granularity. Add WQ_AFFN_CACHE_SHARD, which subdivides each LLC into groups of at most wq_cache_shard_size cores (default 8, tunable via boot parameter). Shards are always split on core (SMT group) boundaries so that Hyper-Threading siblings are never placed in different pods. Cores are distributed across shards as evenly as possible -- for example, 36 cores in a single LLC with max shard size 8 produces 5 shards of 8+7+7+7+7 cores. The implementation follows the same comparator pattern as other affinity scopes: precompute_cache_shard_ids() pre-fills the cpu_shard_id[] array from the already-initialized WQ_AFFN_CACHE and WQ_AFFN_SMT topology, and cpus_share_cache_shard() is passed to init_pod_type(). Benchmark on NVIDIA Grace (72 CPUs, single LLC, 50k items/thread), show cache_shard delivers ~5x the throughput and ~6.5x lower p50 latency compared to cache scope on this 72-core single-LLC system. Suggested-by: Tejun Heo Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 5920d046f7ae3bf9cf51b9d915c1fff13d299d84) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- include/linux/workqueue.h | 1 + kernel/workqueue.c | 183 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 184 insertions(+) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 7f839049727ee..2cb602ee8c535 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -133,6 +133,7 @@ enum wq_affn_scope { WQ_AFFN_CPU, /* one pod per CPU */ WQ_AFFN_SMT, /* one pod per SMT */ WQ_AFFN_CACHE, /* one pod per LLC */ + WQ_AFFN_CACHE_SHARD, /* synthetic sub-LLC shards */ WQ_AFFN_NUMA, /* one pod per NUMA node */ WQ_AFFN_SYSTEM, /* one pod across the whole system */ diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 441916a6d11c8..3bc68035cc948 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -131,6 +131,14 @@ enum wq_internal_consts { WORKER_ID_LEN = 10 + WQ_NAME_LEN, /* "kworker/R-" + WQ_NAME_LEN */ }; +/* Layout of shards within one LLC pod */ +struct llc_shard_layout { + int nr_large_shards; /* number of large shards (cores_per_shard + 1) */ + int cores_per_shard; /* base number of cores per default shard */ + int nr_shards; /* total number of shards */ + /* nr_default shards = (nr_shards - nr_large_shards) */ +}; + /* * We don't want to trap softirq for too long. See MAX_SOFTIRQ_TIME and * MAX_SOFTIRQ_RESTART in kernel/softirq.c. These are macros because @@ -410,6 +418,7 @@ static const char * const wq_affn_names[WQ_AFFN_NR_TYPES] = { [WQ_AFFN_CPU] = "cpu", [WQ_AFFN_SMT] = "smt", [WQ_AFFN_CACHE] = "cache", + [WQ_AFFN_CACHE_SHARD] = "cache_shard", [WQ_AFFN_NUMA] = "numa", [WQ_AFFN_SYSTEM] = "system", }; @@ -432,6 +441,9 @@ module_param_named(cpu_intensive_warning_thresh, wq_cpu_intensive_warning_thresh static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT); module_param_named(power_efficient, wq_power_efficient, bool, 0444); +static unsigned int wq_cache_shard_size = 8; +module_param_named(cache_shard_size, wq_cache_shard_size, uint, 0444); + static bool wq_online; /* can kworkers be created yet? */ static bool wq_topo_initialized __read_mostly = false; @@ -8179,6 +8191,175 @@ static bool __init cpus_share_numa(int cpu0, int cpu1) return cpu_to_node(cpu0) == cpu_to_node(cpu1); } +/* Maps each CPU to its shard index within the LLC pod it belongs to */ +static int cpu_shard_id[NR_CPUS] __initdata; + +/** + * llc_count_cores - count distinct cores (SMT groups) within an LLC pod + * @pod_cpus: the cpumask of CPUs in the LLC pod + * @smt_pods: the SMT pod type, used to identify sibling groups + * + * A core is represented by the lowest-numbered CPU in its SMT group. Returns + * the number of distinct cores found in @pod_cpus. + */ +static int __init llc_count_cores(const struct cpumask *pod_cpus, + struct wq_pod_type *smt_pods) +{ + const struct cpumask *sibling_cpus; + int nr_cores = 0, c; + + /* + * Count distinct cores by only counting the first CPU in each + * SMT sibling group. + */ + for_each_cpu(c, pod_cpus) { + sibling_cpus = smt_pods->pod_cpus[smt_pods->cpu_pod[c]]; + if (cpumask_first(sibling_cpus) == c) + nr_cores++; + } + + return nr_cores; +} + +/* + * llc_shard_size - number of cores in a given shard + * + * Cores are spread as evenly as possible. The first @nr_large_shards shards are + * "large shards" with (cores_per_shard + 1) cores; the rest are "default + * shards" with cores_per_shard cores. + */ +static int __init llc_shard_size(int shard_id, int cores_per_shard, int nr_large_shards) +{ + /* The first @nr_large_shards shards are large shards */ + if (shard_id < nr_large_shards) + return cores_per_shard + 1; + + /* The remaining shards are default shards */ + return cores_per_shard; +} + +/* + * llc_calc_shard_layout - compute the shard layout for an LLC pod + * @nr_cores: number of distinct cores in the LLC pod + * + * Chooses the number of shards that keeps average shard size closest to + * wq_cache_shard_size. Returns a struct describing the total number of shards, + * the base size of each, and how many are large shards. + */ +static struct llc_shard_layout __init llc_calc_shard_layout(int nr_cores) +{ + struct llc_shard_layout layout; + + /* Ensure at least one shard; pick the count closest to the target size */ + layout.nr_shards = max(1, DIV_ROUND_CLOSEST(nr_cores, wq_cache_shard_size)); + layout.cores_per_shard = nr_cores / layout.nr_shards; + layout.nr_large_shards = nr_cores % layout.nr_shards; + + return layout; +} + +/* + * llc_shard_is_full - check whether a shard has reached its core capacity + * @cores_in_shard: number of cores already assigned to this shard + * @shard_id: index of the shard being checked + * @layout: the shard layout computed by llc_calc_shard_layout() + * + * Returns true if @cores_in_shard equals the expected size for @shard_id. + */ +static bool __init llc_shard_is_full(int cores_in_shard, int shard_id, + const struct llc_shard_layout *layout) +{ + return cores_in_shard == llc_shard_size(shard_id, layout->cores_per_shard, + layout->nr_large_shards); +} + +/** + * llc_populate_cpu_shard_id - populate cpu_shard_id[] for each CPU in an LLC pod + * @pod_cpus: the cpumask of CPUs in the LLC pod + * @smt_pods: the SMT pod type, used to identify sibling groups + * @nr_cores: number of distinct cores in @pod_cpus (from llc_count_cores()) + * + * Walks @pod_cpus in order. At each SMT group leader, advances to the next + * shard once the current shard is full. Results are written to cpu_shard_id[]. + */ +static void __init llc_populate_cpu_shard_id(const struct cpumask *pod_cpus, + struct wq_pod_type *smt_pods, + int nr_cores) +{ + struct llc_shard_layout layout = llc_calc_shard_layout(nr_cores); + const struct cpumask *sibling_cpus; + /* Count the number of cores in the current shard_id */ + int cores_in_shard = 0; + /* This is a cursor for the shards. Go from zero to nr_shards - 1*/ + int shard_id = 0; + int c; + + /* Iterate at every CPU for a given LLC pod, and assign it a shard */ + for_each_cpu(c, pod_cpus) { + sibling_cpus = smt_pods->pod_cpus[smt_pods->cpu_pod[c]]; + if (cpumask_first(sibling_cpus) == c) { + /* This is the CPU leader for the siblings */ + if (llc_shard_is_full(cores_in_shard, shard_id, &layout)) { + shard_id++; + cores_in_shard = 0; + } + cores_in_shard++; + cpu_shard_id[c] = shard_id; + } else { + /* + * The siblings' shard MUST be the same as the leader. + * never split threads in the same core. + */ + cpu_shard_id[c] = cpu_shard_id[cpumask_first(sibling_cpus)]; + } + } + + WARN_ON_ONCE(shard_id != (layout.nr_shards - 1)); +} + +/** + * precompute_cache_shard_ids - assign each CPU its shard index within its LLC + * + * Iterates over all LLC pods. For each pod, counts distinct cores then assigns + * shard indices to all CPUs in the pod. Must be called after WQ_AFFN_CACHE and + * WQ_AFFN_SMT have been initialized. + */ +static void __init precompute_cache_shard_ids(void) +{ + struct wq_pod_type *llc_pods = &wq_pod_types[WQ_AFFN_CACHE]; + struct wq_pod_type *smt_pods = &wq_pod_types[WQ_AFFN_SMT]; + const struct cpumask *cpus_sharing_llc; + int nr_cores; + int pod; + + if (!wq_cache_shard_size) { + pr_warn("workqueue: cache_shard_size must be > 0, setting to 1\n"); + wq_cache_shard_size = 1; + } + + for (pod = 0; pod < llc_pods->nr_pods; pod++) { + cpus_sharing_llc = llc_pods->pod_cpus[pod]; + + /* Number of cores in this given LLC */ + nr_cores = llc_count_cores(cpus_sharing_llc, smt_pods); + llc_populate_cpu_shard_id(cpus_sharing_llc, smt_pods, nr_cores); + } +} + +/* + * cpus_share_cache_shard - test whether two CPUs belong to the same cache shard + * + * Two CPUs share a cache shard if they are in the same LLC and have the same + * shard index. Used as the pod affinity callback for WQ_AFFN_CACHE_SHARD. + */ +static bool __init cpus_share_cache_shard(int cpu0, int cpu1) +{ + if (!cpus_share_cache(cpu0, cpu1)) + return false; + + return cpu_shard_id[cpu0] == cpu_shard_id[cpu1]; +} + /** * workqueue_init_topology - initialize CPU pods for unbound workqueues * @@ -8194,6 +8375,8 @@ void __init workqueue_init_topology(void) init_pod_type(&wq_pod_types[WQ_AFFN_CPU], cpus_dont_share); init_pod_type(&wq_pod_types[WQ_AFFN_SMT], cpus_share_smt); init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache); + precompute_cache_shard_ids(); + init_pod_type(&wq_pod_types[WQ_AFFN_CACHE_SHARD], cpus_share_cache_shard); init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa); wq_topo_initialized = true; From c5fca1aef19b7dcfe9a92d11d3324d322ad10d86 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:54 -0700 Subject: [PATCH 159/464] workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope BugLink: https://bugs.launchpad.net/bugs/2150467 Set WQ_AFFN_CACHE_SHARD as the default affinity scope for unbound workqueues. On systems where many CPUs share one LLC, the previous default (WQ_AFFN_CACHE) collapses all CPUs to a single worker pool, causing heavy spinlock contention on pool->lock. WQ_AFFN_CACHE_SHARD subdivides each LLC into smaller groups, providing a better balance between locality and contention. Users can revert to the previous behavior with workqueue.default_affinity_scope=cache. On systems with 8 or fewer cores per LLC, CACHE_SHARD produces a single shard covering the entire LLC, making it functionally identical to the previous CACHE default. The sharding only activates when an LLC has more than 8 cores. Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 4cdc8a7389d5025051f6c4a60fb5b7cb9b7960bb) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 3bc68035cc948..3e282e14af51a 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -450,7 +450,7 @@ static bool wq_topo_initialized __read_mostly = false; static struct kmem_cache *pwq_cache; static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES]; -static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE; +static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE_SHARD; /* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */ static struct workqueue_attrs *unbound_wq_update_pwq_attrs_buf; From b6e0b67251136af44420db7077022c711f802999 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:55 -0700 Subject: [PATCH 160/464] tools/workqueue: add CACHE_SHARD support to wq_dump.py BugLink: https://bugs.launchpad.net/bugs/2150467 The WQ_AFFN_CACHE_SHARD affinity scope was added to the kernel but wq_dump.py was not updated to enumerate it. Add the missing constant lookup and include it in the affinity scopes iteration so that drgn output shows the CACHE_SHARD pod topology alongside the other scopes. Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 738390a5321c7d34f468bc69f7232db711210bc0) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- tools/workqueue/wq_dump.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/workqueue/wq_dump.py b/tools/workqueue/wq_dump.py index d29b918306b48..06948ffcfc4b6 100644 --- a/tools/workqueue/wq_dump.py +++ b/tools/workqueue/wq_dump.py @@ -107,6 +107,7 @@ def wq_type_str(wq): WQ_AFFN_CPU = prog['WQ_AFFN_CPU'] WQ_AFFN_SMT = prog['WQ_AFFN_SMT'] WQ_AFFN_CACHE = prog['WQ_AFFN_CACHE'] +WQ_AFFN_CACHE_SHARD = prog['WQ_AFFN_CACHE_SHARD'] WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA'] WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM'] @@ -138,7 +139,7 @@ def print_pod_type(pt): print(f' [{cpu}]={pt.cpu_pod[cpu].value_()}', end='') print('') -for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]: +for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_CACHE_SHARD, WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]: print('') print(f'{wq_affn_names[affn].string_().decode().upper()}{" (default)" if affn == wq_affn_dfl else ""}') print_pod_type(wq_pod_types[affn]) From 5394d7b0bb10a7d2200288ebd7ea8ac36553c727 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:56 -0700 Subject: [PATCH 161/464] workqueue: add test_workqueue benchmark module BugLink: https://bugs.launchpad.net/bugs/2150467 Add a kernel module that benchmarks queue_work() throughput on an unbound workqueue to measure pool->lock contention under different affinity scope configurations (cache vs cache_shard). The module spawns N kthreads (default: num_online_cpus()), each bound to a different CPU. All threads start simultaneously and queue work items, measuring the latency of each queue_work() call. Results are reported as p50/p90/p95 latencies for each affinity scope. The affinity scope is switched between runs via the workqueue's sysfs affinity_scope attribute (WQ_SYSFS), avoiding the need for any new exported symbols. The module runs as __init-only, returning -EAGAIN to auto-unload, and can be re-run via insmod. Example of the output: running 50 threads, 50000 items/thread cpu 6806017 items/sec p50=2574 p90=5068 p95=5818 ns smt 6821040 items/sec p50=2624 p90=5168 p95=5949 ns cache_shard 1633653 items/sec p50=5337 p90=9694 p95=11207 ns cache 286069 items/sec p50=72509 p90=82304 p95=85009 ns numa 319403 items/sec p50=63745 p90=73480 p95=76505 ns system 308461 items/sec p50=66561 p90=75714 p95=78048 ns Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 24b2e73f9700e0682575feb34556b756e59d4548) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- lib/Kconfig.debug | 10 ++ lib/Makefile | 1 + lib/test_workqueue.c | 294 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 305 insertions(+) create mode 100644 lib/test_workqueue.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 93f356d2b3d95..38bee649697f3 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2628,6 +2628,16 @@ config TEST_VMALLOC If unsure, say N. +config TEST_WORKQUEUE + tristate "Test module for stress/performance analysis of workqueue" + default n + help + This builds the "test_workqueue" module for benchmarking + workqueue throughput under contention. Useful for evaluating + affinity scope changes (e.g., cache_shard vs cache). + + If unsure, say N. + config TEST_BPF tristate "Test BPF filter functionality" depends on m && NET diff --git a/lib/Makefile b/lib/Makefile index 1b9ee167517f3..ea660cca04f40 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -79,6 +79,7 @@ UBSAN_SANITIZE_test_ubsan.o := y obj-$(CONFIG_TEST_KSTRTOX) += test-kstrtox.o obj-$(CONFIG_TEST_LKM) += test_module.o obj-$(CONFIG_TEST_VMALLOC) += test_vmalloc.o +obj-$(CONFIG_TEST_WORKQUEUE) += test_workqueue.o obj-$(CONFIG_TEST_RHASHTABLE) += test_rhashtable.o obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_keys.o obj-$(CONFIG_TEST_STATIC_KEYS) += test_static_key_base.o diff --git a/lib/test_workqueue.c b/lib/test_workqueue.c new file mode 100644 index 0000000000000..f2ae1ac4bd937 --- /dev/null +++ b/lib/test_workqueue.c @@ -0,0 +1,294 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Test module for stress and performance analysis of workqueue. + * + * Benchmarks queue_work() throughput on an unbound workqueue to measure + * pool->lock contention under different affinity scope configurations + * (e.g., cache vs cache_shard). + * + * The affinity scope is changed between runs via the workqueue's sysfs + * affinity_scope attribute (WQ_SYSFS). + * + * Copyright (c) 2026 Meta Platforms, Inc. and affiliates + * Copyright (c) 2026 Breno Leitao + * + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define WQ_NAME "bench_wq" +#define SCOPE_PATH "/sys/bus/workqueue/devices/" WQ_NAME "/affinity_scope" + +static int nr_threads; +module_param(nr_threads, int, 0444); +MODULE_PARM_DESC(nr_threads, + "Number of threads to spawn (default: 0 = num_online_cpus())"); + +static int wq_items = 50000; +module_param(wq_items, int, 0444); +MODULE_PARM_DESC(wq_items, + "Number of work items each thread queues (default: 50000)"); + +static struct workqueue_struct *bench_wq; +static atomic_t threads_done; +static DECLARE_COMPLETION(start_comp); +static DECLARE_COMPLETION(all_done_comp); + +struct thread_ctx { + struct completion work_done; + struct work_struct work; + u64 *latencies; + int cpu; + int items; +}; + +static void bench_work_fn(struct work_struct *work) +{ + struct thread_ctx *ctx = container_of(work, struct thread_ctx, work); + + complete(&ctx->work_done); +} + +static int bench_kthread_fn(void *data) +{ + struct thread_ctx *ctx = data; + ktime_t t_start, t_end; + int i; + + /* Wait for all threads to be ready */ + wait_for_completion(&start_comp); + + if (kthread_should_stop()) + return 0; + + for (i = 0; i < ctx->items; i++) { + reinit_completion(&ctx->work_done); + INIT_WORK(&ctx->work, bench_work_fn); + + t_start = ktime_get(); + queue_work(bench_wq, &ctx->work); + t_end = ktime_get(); + + ctx->latencies[i] = ktime_to_ns(ktime_sub(t_end, t_start)); + wait_for_completion(&ctx->work_done); + } + + if (atomic_dec_and_test(&threads_done)) + complete(&all_done_comp); + + /* + * Wait for kthread_stop() so the module text isn't freed + * while we're still executing. + */ + while (!kthread_should_stop()) + schedule(); + + return 0; +} + +static int cmp_u64(const void *a, const void *b) +{ + u64 va = *(const u64 *)a; + u64 vb = *(const u64 *)b; + + if (va < vb) + return -1; + if (va > vb) + return 1; + return 0; +} + +static int __init set_affn_scope(const char *scope) +{ + struct file *f; + loff_t pos = 0; + ssize_t ret; + + f = filp_open(SCOPE_PATH, O_WRONLY, 0); + if (IS_ERR(f)) { + pr_err("test_workqueue: open %s failed: %ld\n", + SCOPE_PATH, PTR_ERR(f)); + return PTR_ERR(f); + } + + ret = kernel_write(f, scope, strlen(scope), &pos); + filp_close(f, NULL); + + if (ret < 0) { + pr_err("test_workqueue: write '%s' failed: %zd\n", scope, ret); + return ret; + } + + return 0; +} + +static int __init run_bench(int n_threads, const char *scope, const char *label) +{ + struct task_struct **tasks; + unsigned long total_items; + struct thread_ctx *ctxs; + u64 *all_latencies; + ktime_t start, end; + int cpu, i, j, ret; + s64 elapsed_us; + + ret = set_affn_scope(scope); + if (ret) + return ret; + + ctxs = kcalloc(n_threads, sizeof(*ctxs), GFP_KERNEL); + if (!ctxs) + return -ENOMEM; + + tasks = kcalloc(n_threads, sizeof(*tasks), GFP_KERNEL); + if (!tasks) { + kfree(ctxs); + return -ENOMEM; + } + + total_items = (unsigned long)n_threads * wq_items; + all_latencies = kvmalloc_array(total_items, sizeof(u64), GFP_KERNEL); + if (!all_latencies) { + kfree(tasks); + kfree(ctxs); + return -ENOMEM; + } + + /* Allocate per-thread latency arrays */ + for (i = 0; i < n_threads; i++) { + ctxs[i].latencies = kvmalloc_array(wq_items, sizeof(u64), + GFP_KERNEL); + if (!ctxs[i].latencies) { + while (--i >= 0) + kvfree(ctxs[i].latencies); + kvfree(all_latencies); + kfree(tasks); + kfree(ctxs); + return -ENOMEM; + } + } + + atomic_set(&threads_done, n_threads); + reinit_completion(&all_done_comp); + reinit_completion(&start_comp); + + /* Create kthreads, each bound to a different online CPU */ + i = 0; + for_each_online_cpu(cpu) { + if (i >= n_threads) + break; + + ctxs[i].cpu = cpu; + ctxs[i].items = wq_items; + init_completion(&ctxs[i].work_done); + + tasks[i] = kthread_create(bench_kthread_fn, &ctxs[i], + "wq_bench/%d", cpu); + if (IS_ERR(tasks[i])) { + ret = PTR_ERR(tasks[i]); + pr_err("test_workqueue: failed to create kthread %d: %d\n", + i, ret); + /* Unblock threads waiting on start_comp before stopping them */ + complete_all(&start_comp); + while (--i >= 0) + kthread_stop(tasks[i]); + goto out_free; + } + + kthread_bind(tasks[i], cpu); + wake_up_process(tasks[i]); + i++; + } + + /* Start timing and release all threads */ + start = ktime_get(); + complete_all(&start_comp); + + /* Wait for all threads to finish the benchmark */ + wait_for_completion(&all_done_comp); + + /* Drain any remaining work */ + flush_workqueue(bench_wq); + + /* Ensure all kthreads have fully exited before module memory is freed */ + for (i = 0; i < n_threads; i++) + kthread_stop(tasks[i]); + + end = ktime_get(); + elapsed_us = ktime_us_delta(end, start); + + /* Merge all per-thread latencies and sort for percentile calculation */ + j = 0; + for (i = 0; i < n_threads; i++) { + memcpy(&all_latencies[j], ctxs[i].latencies, + wq_items * sizeof(u64)); + j += wq_items; + } + + sort(all_latencies, total_items, sizeof(u64), cmp_u64, NULL); + + pr_info("test_workqueue: %-16s %llu items/sec\tp50=%llu\tp90=%llu\tp95=%llu ns\n", + label, + elapsed_us ? total_items * 1000000ULL / elapsed_us : 0, + all_latencies[total_items * 50 / 100], + all_latencies[total_items * 90 / 100], + all_latencies[total_items * 95 / 100]); + + ret = 0; +out_free: + for (i = 0; i < n_threads; i++) + kvfree(ctxs[i].latencies); + kvfree(all_latencies); + kfree(tasks); + kfree(ctxs); + + return ret; +} + +static const char * const bench_scopes[] = { + "cpu", "smt", "cache_shard", "cache", "numa", "system", +}; + +static int __init test_workqueue_init(void) +{ + int n_threads = min(nr_threads ?: num_online_cpus(), num_online_cpus()); + int i; + + if (wq_items <= 0) { + pr_err("test_workqueue: wq_items must be > 0\n"); + return -EINVAL; + } + + bench_wq = alloc_workqueue(WQ_NAME, WQ_UNBOUND | WQ_SYSFS, 0); + if (!bench_wq) + return -ENOMEM; + + pr_info("test_workqueue: running %d threads, %d items/thread\n", + n_threads, wq_items); + + for (i = 0; i < ARRAY_SIZE(bench_scopes); i++) + run_bench(n_threads, bench_scopes[i], bench_scopes[i]); + + destroy_workqueue(bench_wq); + + /* Return -EAGAIN so the module doesn't stay loaded after the benchmark */ + return -EAGAIN; +} + +module_init(test_workqueue_init); +MODULE_AUTHOR("Breno Leitao "); +MODULE_DESCRIPTION("Stress/performance benchmark for workqueue subsystem"); +MODULE_LICENSE("GPL"); From 8133a7069a444a7011cb1a4abf130af45de39ac2 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Wed, 1 Apr 2026 06:03:57 -0700 Subject: [PATCH 162/464] docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope BugLink: https://bugs.launchpad.net/bugs/2150467 Update kernel-parameters.txt and workqueue.rst to reflect the new cache_shard affinity scope and the default change from cache to cache_shard. Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 41e3ccca00b374b7f39cf68e818b59a921cd7069) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- Documentation/admin-guide/kernel-parameters.txt | 3 ++- Documentation/core-api/workqueue.rst | 14 ++++++++++---- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index d3e7034bece52..a030ce253b4b7 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -8541,7 +8541,8 @@ Kernel parameters workqueue.default_affinity_scope= Select the default affinity scope to use for unbound workqueues. Can be one of "cpu", "smt", "cache", - "numa" and "system". Default is "cache". For more + "cache_shard", "numa" and "system". Default is + "cache_shard". For more information, see the Affinity Scopes section in Documentation/core-api/workqueue.rst. diff --git a/Documentation/core-api/workqueue.rst b/Documentation/core-api/workqueue.rst index 165ca73e83514..411e1b28b8dec 100644 --- a/Documentation/core-api/workqueue.rst +++ b/Documentation/core-api/workqueue.rst @@ -378,9 +378,9 @@ Affinity Scopes An unbound workqueue groups CPUs according to its affinity scope to improve cache locality. For example, if a workqueue is using the default affinity -scope of "cache", it will group CPUs according to last level cache -boundaries. A work item queued on the workqueue will be assigned to a worker -on one of the CPUs which share the last level cache with the issuing CPU. +scope of "cache_shard", it will group CPUs into sub-LLC shards. A work item +queued on the workqueue will be assigned to a worker on one of the CPUs +within the same shard as the issuing CPU. Once started, the worker may or may not be allowed to move outside the scope depending on the ``affinity_strict`` setting of the scope. @@ -402,7 +402,13 @@ Workqueue currently supports the following affinity scopes. ``cache`` CPUs are grouped according to cache boundaries. Which specific cache boundary is used is determined by the arch code. L3 is used in a lot of - cases. This is the default affinity scope. + cases. + +``cache_shard`` + CPUs are grouped into sub-LLC shards of at most ``wq_cache_shard_size`` + cores (default 8, tunable via the ``workqueue.cache_shard_size`` boot + parameter). Shards are always split on core (SMT group) boundaries. + This is the default affinity scope. ``numa`` CPUs are grouped according to NUMA boundaries. From 99f2b94ad54010e3d3a0071bfadc18c4ca1de03d Mon Sep 17 00:00:00 2001 From: Arnd Bergmann Date: Thu, 2 Apr 2026 22:59:03 +0200 Subject: [PATCH 163/464] workqueue: avoid unguarded 64-bit division BugLink: https://bugs.launchpad.net/bugs/2150467 The printk() requires a division that is not allowed on 32-bit architectures: x86_64-linux-ld: lib/test_workqueue.o: in function `test_workqueue_init': test_workqueue.c:(.init.text+0x36f): undefined reference to `__udivdi3' Use div_u64() to print the resulting elapsed microseconds. Fixes: 24b2e73f9700 ("workqueue: add test_workqueue benchmark module") Signed-off-by: Arnd Bergmann Signed-off-by: Tejun Heo (cherry picked from commit c6890f36fc49848c61d2113a3442eb1b59e0bc4b) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- lib/test_workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/test_workqueue.c b/lib/test_workqueue.c index f2ae1ac4bd937..99e160bd5ad17 100644 --- a/lib/test_workqueue.c +++ b/lib/test_workqueue.c @@ -242,7 +242,7 @@ static int __init run_bench(int n_threads, const char *scope, const char *label) pr_info("test_workqueue: %-16s %llu items/sec\tp50=%llu\tp90=%llu\tp95=%llu ns\n", label, - elapsed_us ? total_items * 1000000ULL / elapsed_us : 0, + elapsed_us ? div_u64(total_items * 1000000ULL, elapsed_us) : 0, all_latencies[total_items * 50 / 100], all_latencies[total_items * 90 / 100], all_latencies[total_items * 95 / 100]); From df45de58e46817858e7ea9d02ca0fb5147ddfb2d Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Mon, 13 Apr 2026 07:26:47 -0700 Subject: [PATCH 164/464] workqueue: validate cpumask_first() result in llc_populate_cpu_shard_id() BugLink: https://bugs.launchpad.net/bugs/2150467 On uniprocessor (UP) configs such as nios2, NR_CPUS is 1, so cpu_shard_id[] is a single-element array (int[1]). In llc_populate_cpu_shard_id(), cpumask_first(sibling_cpus) returns an unsigned int that the compiler cannot prove is always 0, triggering a -Warray-bounds warning when the result is used to index cpu_shard_id[]: kernel/workqueue.c:8321:55: warning: array subscript 1 is above array bounds of 'int[1]' [-Warray-bounds] 8321 | cpu_shard_id[c] = cpu_shard_id[cpumask_first(sibling_cpus)]; | ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is a false positive: sibling_cpus can never be empty here because 'c' itself is always set in it, so cpumask_first() will always return a valid CPU. However, the compiler cannot prove this statically, and the warning only manifests on UP configs where the array size is 1. Add a bounds check with WARN_ON_ONCE to silence the warning, and store the result in a local variable to make the code clearer and avoid calling cpumask_first() twice. Fixes: 5920d046f7ae ("workqueue: add WQ_AFFN_CACHE_SHARD affinity scope") Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202604022343.GQtkF2vO-lkp@intel.com/ Signed-off-by: Breno Leitao Signed-off-by: Tejun Heo (cherry picked from commit 76af54648899abbd6b449c035583e47fd407078a) Signed-off-by: Carol L Soto Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- kernel/workqueue.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 3e282e14af51a..3b8edbfea34df 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -8290,6 +8290,7 @@ static void __init llc_populate_cpu_shard_id(const struct cpumask *pod_cpus, const struct cpumask *sibling_cpus; /* Count the number of cores in the current shard_id */ int cores_in_shard = 0; + unsigned int leader; /* This is a cursor for the shards. Go from zero to nr_shards - 1*/ int shard_id = 0; int c; @@ -8310,7 +8311,17 @@ static void __init llc_populate_cpu_shard_id(const struct cpumask *pod_cpus, * The siblings' shard MUST be the same as the leader. * never split threads in the same core. */ - cpu_shard_id[c] = cpu_shard_id[cpumask_first(sibling_cpus)]; + leader = cpumask_first(sibling_cpus); + + /* + * This check silences a Warray-bounds warning on UP + * configs where NR_CPUS=1 makes cpu_shard_id[] + * a single-element array, and the compiler can't + * prove the index is always 0. + */ + if (WARN_ON_ONCE(leader >= nr_cpu_ids)) + continue; + cpu_shard_id[c] = cpu_shard_id[leader]; } } From 8e5d7460502b85dfa6837428e8e7f11aa7ecfaad Mon Sep 17 00:00:00 2001 From: Abhishek Sahu Date: Mon, 27 Apr 2026 05:04:56 +0000 Subject: [PATCH 165/464] NVIDIA: SAUCE: iommu/arm-smmu-v3: Use device ID range for DGX Spark iGPU iommu quirk BugLink: https://bugs.launchpad.net/bugs/2150487 Replace the explicit DGX Spark iGPU device ID list with a range check covering 0x2E00-0x2E3F to accommodate all possible DGX Spark iGPU PCI device IDs without requiring individual additions. The original quirk was introduced in commit ab858638d96a ("NVIDIA: SAUCE: iommu/arm-smmu-v3: Set DGX Spark iGPU default domain type to DMA") and extended with two more IDs in commit 8dc61abaa2eb ("NVIDIA: SAUCE: iommu/arm-smmu-v3: Add two more DGX Spark iGPU IDs for existing iommu quirk"). Using a range avoids further per-ID additions as new DGX Spark variants are introduced. Signed-off-by: Abhishek Sahu Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index de07e00ff5ac0..654ddd68e40cc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3805,8 +3805,7 @@ static int arm_smmu_def_domain_type(struct device *dev) return IOMMU_DOMAIN_IDENTITY; if (pdev->vendor == PCI_VENDOR_ID_NVIDIA && - (pdev->device == 0x2E12 || pdev->device == 0x2E2A || - pdev->device == 0x2E2B)) + pdev->device >= 0x2E00 && pdev->device <= 0x2E3F) return IOMMU_DOMAIN_DMA; } From 8c20761fbdf3d268b2a856b3af8c25a935e91aca Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Fri, 1 May 2026 13:41:52 -0700 Subject: [PATCH 166/464] UBUNTU: [Config] nvidia: Disable default CMA reservation BugLink: https://bugs.launchpad.net/bugs/2150898 Set CONFIG_CMA_SIZE_MBYTES=0 for arm64 linux-nvidia kernels. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- debian.nvidia-bos/config/annotations | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 386aabc4bc8cc..9300aee4dae4f 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -45,8 +45,8 @@ CONFIG_ARM_LFA note<'LP: #2138342'> CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> -CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '32', 'arm64-nvidia-bos': '128', 'arm64-nvidia-bos-64k': '1024'}> -CONFIG_CMA_SIZE_MBYTES note<'LP: #2095028'> +CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '0'}> +CONFIG_CMA_SIZE_MBYTES note<'LP: #2150898'> CONFIG_CORESIGHT policy<{'arm64': 'm'}> CONFIG_CORESIGHT note<'Required for Grace enablement'> From 4212e4f0e84aff20883d48abb29a202851ecd1ff Mon Sep 17 00:00:00 2001 From: Carol L Soto Date: Wed, 6 May 2026 08:17:47 -0700 Subject: [PATCH 167/464] UBUNTU: [Config] nvidia: Defaults for CONFIG_TEST_WORKQUEUE BugLink: https://bugs.launchpad.net/bugs/2150467 Set defaults for CONFIG_TEST_WORKQUEUE. Signed-off-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 9300aee4dae4f..392f5a93126da 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -216,6 +216,9 @@ CONFIG_TCG_ARM_CRB_FFA note<'LP: #2111511'> CONFIG_TCG_TIS_SPI policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_TCG_TIS_SPI note<'Ensures the TPM is available before the IMA driver initializes'> +CONFIG_TEST_WORKQUEUE policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_TEST_WORKQUEUE note<'LP: #2150467'> + CONFIG_UBUNTU_ODM_DRIVERS policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM drivers'> From c38a435046fed3a219bf5403cb5bea94532d5bfe Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 20 Apr 2026 23:29:30 -0700 Subject: [PATCH 168/464] Revert "NVIDIA: VR: SAUCE: firmware: smccc: register as platform driver" BugLink: https://bugs.launchpad.net/bugs/2150652 This reverts commit f1ad1da2ead9f317331195fdba00aebc14408b76. Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 153 ++++---------------------------- 1 file changed, 19 insertions(+), 134 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 90727a66e49a5..ed439daf49319 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -20,10 +20,7 @@ #include #include #include -#include -#include -#define DRIVER_NAME "ARM_LFA" #undef pr_fmt #define pr_fmt(fmt) "Arm LFA: " fmt @@ -287,7 +284,26 @@ static int activate_fw_image(struct image_props *attrs) return lfa_cancel(attrs); } + /* + * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the + * number of firmware images in the LFA agent may change after a + * successful activation attempt. Negate all image flags as well. + */ + attrs = NULL; + list_for_each_entry(attrs, &lfa_fw_images, image_node) { + set_image_flags(attrs, -1, 0b1000, 0, 0); + } + update_fw_images_tree(); + + /* + * Removing non-valid image directories at the end of an activation. + * We can't remove the sysfs attributes while in the respective + * _store() handler, so have to postpone the list removal to a + * workqueue. + */ + INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); + queue_work(fw_images_update_wq, &fw_images_update_work); mutex_unlock(&lfa_lock); return ret; @@ -611,7 +627,6 @@ static int update_fw_images_tree(void) { struct arm_smccc_1_2_regs reg = { 0 }; struct uuid_regs image_uuid; - struct image_props *attrs; char image_id_str[40]; int ret, num_of_components; @@ -621,15 +636,6 @@ static int update_fw_images_tree(void) return -ENODEV; } - /* - * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the - * number of firmware images in the LFA agent may change after a - * successful activation attempt. Negate all image flags as well. - */ - list_for_each_entry(attrs, &lfa_fw_images, image_node) { - set_image_flags(attrs, -1, 0b1000, 0, 0); - } - for (int i = 0; i < num_of_components; i++) { reg.a0 = LFA_1_0_FN_GET_INVENTORY; reg.a1 = i; /* fw_seq_id under consideration */ @@ -647,121 +653,9 @@ static int update_fw_images_tree(void) } } - /* - * Removing non-valid image directories at the end of an activation. - * We can't remove the sysfs attributes while in the respective - * _store() handler, so have to postpone the list removal to a - * workqueue. - */ - INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); - queue_work(fw_images_update_wq, &fw_images_update_work); - - return 0; -} - -#if defined(CONFIG_ACPI) -static void lfa_notify_handler(acpi_handle handle, u32 event, void *data) -{ - struct image_props *attrs = NULL; - int ret; - bool found_activable_image = false; - - /* Get latest FW inventory */ - mutex_lock(&lfa_lock); - ret = update_fw_images_tree(); - mutex_unlock(&lfa_lock); - if (ret != 0) { - pr_err("FW images tree update failed"); - return; - } - - /* - * Go through all FW images in a loop and trigger activation - * of all activable and pending images. - */ - do { - /* Reset activable image flag */ - found_activable_image = false; - list_for_each_entry(attrs, &lfa_fw_images, image_node) { - if (attrs->fw_seq_id == -1) - continue; /* Invalid FW component */ - - if ((!attrs->activation_capable) || (!attrs->activation_pending)) - continue; /* FW component is not activable */ - - /* - * Found an image that is activable. - * As the FW images tree is revised after activation, it is - * not ideal to invoke activation from inside - * list_for_each_entry() loop. - * So, set the flasg and exit loop. - */ - found_activable_image = true; - break; - } - - if (found_activable_image) { - ret = prime_fw_image(attrs); - if (ret) { - pr_err("Firmware prime failed: %s\n", - lfa_error_strings[-ret]); - return; - } - - ret = activate_fw_image(attrs); - if (ret) { - pr_err("Firmware activation failed: %s\n", - lfa_error_strings[-ret]); - return; - } - - pr_info("Firmware %s activation succeeded", attrs->image_name); - } - } while(found_activable_image); - - return; -} - -static int lfa_probe(struct platform_device *pdev) -{ - acpi_status status; - acpi_handle handle = ACPI_HANDLE(&pdev->dev); - if (!handle) - return -ENODEV; - - /* Register notify handler that indicates if LFA updates are available */ - status = acpi_install_notify_handler(handle, - ACPI_DEVICE_NOTIFY, lfa_notify_handler, pdev); - if (ACPI_FAILURE(status)) - return -EIO; - return 0; } -static void lfa_remove(struct platform_device *pdev) { - acpi_handle handle = ACPI_HANDLE(&pdev->dev); - - if (handle) - acpi_remove_notify_handler(handle, - ACPI_DEVICE_NOTIFY, lfa_notify_handler); -} - -static const struct acpi_device_id lfa_acpi_ids[] = { - {"ARML0003"}, - {}, -}; -MODULE_DEVICE_TABLE(acpi, lfa_acpi_ids); - -static struct platform_driver lfa_driver = { - .probe = lfa_probe, - .remove = lfa_remove, - .driver = { - .name = DRIVER_NAME, - .acpi_match_table = ACPI_PTR(lfa_acpi_ids), - }, -}; -#endif - static int __init lfa_init(void) { struct arm_smccc_1_2_regs reg = { 0 }; @@ -785,12 +679,6 @@ static int __init lfa_init(void) pr_info("Live Firmware Activation: detected v%ld.%ld\n", reg.a0 >> 16, reg.a0 & 0xffff); -#if defined(CONFIG_ACPI) - err = platform_driver_register(&lfa_driver); - if (err < 0) - pr_err("Platform driver register failed"); -#endif - lfa_dir = kobject_create_and_add("lfa", firmware_kobj); if (!lfa_dir) return -ENOMEM; @@ -815,9 +703,6 @@ static void __exit lfa_exit(void) mutex_unlock(&lfa_lock); kobject_put(lfa_dir); -#if defined(CONFIG_ACPI) - platform_driver_unregister(&lfa_driver); -#endif } module_exit(lfa_exit); From 32d84a5e824d048dcc24921867a98cb0240834a6 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 20 Apr 2026 23:29:30 -0700 Subject: [PATCH 169/464] Revert "NVIDIA: VR: SAUCE: firmware: smccc: add timeout, touch wdt" BugLink: https://bugs.launchpad.net/bugs/2150652 This reverts commit e7616bcae74b5c2282379e77b56fc93265d10a9d. Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 82 +++++++++------------------------ 1 file changed, 21 insertions(+), 61 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index ed439daf49319..adf4171066437 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -17,9 +17,6 @@ #include #include #include -#include -#include -#include #undef pr_fmt #define pr_fmt(fmt) "Arm LFA: " fmt @@ -40,14 +37,6 @@ #define LFA_PRIME_CALL_AGAIN BIT(0) #define LFA_ACTIVATE_CALL_AGAIN BIT(0) -/* Prime loop limits, TODO: tune after testing */ -#define LFA_PRIME_BUDGET_US 30000000 /* 30s cap */ -#define LFA_PRIME_POLL_DELAY_US 10 /* 10us between polls */ - -/* Activation loop limits, TODO: tune after testing */ -#define LFA_ACTIVATE_BUDGET_US 20000000 /* 20s cap */ -#define LFA_ACTIVATE_POLL_DELAY_US 10 /* 10us between polls */ - /* LFA return values */ #define LFA_SUCCESS 0 #define LFA_NOT_SUPPORTED 1 @@ -228,12 +217,10 @@ static int lfa_cancel(void *data) static int call_lfa_activate(void *data) { struct image_props *attrs = data; - struct arm_smccc_1_2_regs args = { 0 }; - struct arm_smccc_1_2_regs res = { 0 }; - ktime_t end = ktime_add_us(ktime_get(), LFA_ACTIVATE_BUDGET_US); + struct arm_smccc_1_2_regs reg = { 0 }; - args.a0 = LFA_1_0_FN_ACTIVATE; - args.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + reg.a0 = LFA_1_0_FN_ACTIVATE; + reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ /* * As we do not support updates requiring a CPU reset (yet), * we pass 0 in reg.a3 and reg.a4, holding the entry point and context @@ -241,32 +228,21 @@ static int call_lfa_activate(void *data) * cpu_rendezvous_forced is set by the administrator, via sysfs, * cpu_rendezvous is dictated by each firmware component. */ - args.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); + reg.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); for (;;) { - /* Touch watchdog, ACTIVATE shouldn't take longer than watchdog_thresh */ - touch_nmi_watchdog(); - arm_smccc_1_2_invoke(&args, &res); + arm_smccc_1_2_invoke(®, ®); - if ((long)res.a0 < 0) { + if ((long)reg.a0 < 0) { pr_err("ACTIVATE for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-res.a0]); - return res.a0; + attrs->image_name, lfa_error_strings[-reg.a0]); + return reg.a0; } - if (!(res.a1 & LFA_ACTIVATE_CALL_AGAIN)) + if (!(reg.a1 & LFA_ACTIVATE_CALL_AGAIN)) break; /* ACTIVATE successful */ - - /* SMC returned with call_again flag set */ - if (ktime_before(ktime_get(), end)) { - udelay(LFA_ACTIVATE_POLL_DELAY_US); - continue; - } - - pr_err("ACTIVATE for image %s timed out", attrs->image_name); - return -ETIMEDOUT; } - return res.a0; + return reg.a0; } static int activate_fw_image(struct image_props *attrs) @@ -311,9 +287,7 @@ static int activate_fw_image(struct image_props *attrs) static int prime_fw_image(struct image_props *attrs) { - struct arm_smccc_1_2_regs args = { 0 }; - struct arm_smccc_1_2_regs res = { 0 }; - ktime_t end = ktime_add_us(ktime_get(), LFA_PRIME_BUDGET_US); + struct arm_smccc_1_2_regs reg = { 0 }; int ret; mutex_lock(&lfa_lock); @@ -333,41 +307,27 @@ static int prime_fw_image(struct image_props *attrs) } /* - * LFA_PRIME/ACTIVATE will return 1 in res.a1 if the firmware + * LFA_PRIME/ACTIVATE will return 1 in reg.a1 if the firmware * priming/activation is still in progress. In that case * LFA_PRIME/ACTIVATE will need to be called again. - * res.a1 will become 0 once the prime/activate process completes. + * reg.a1 will become 0 once the prime/activate process completes. */ - args.a0 = LFA_1_0_FN_PRIME; - args.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ + reg.a0 = LFA_1_0_FN_PRIME; + reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ for (;;) { - /* Touch watchdog, PRIME shouldn't take longer than watchdog_thresh */ - touch_nmi_watchdog(); - arm_smccc_1_2_invoke(&args, &res); + arm_smccc_1_2_invoke(®, ®); - if ((long)res.a0 < 0) { + if ((long)reg.a0 < 0) { pr_err("LFA_PRIME for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-res.a0]); + attrs->image_name, lfa_error_strings[-reg.a0]); mutex_unlock(&lfa_lock); - return res.a0; + return reg.a0; } - if (!(res.a1 & LFA_PRIME_CALL_AGAIN)) + if (!(reg.a1 & LFA_PRIME_CALL_AGAIN)) { + ret = 0; break; /* PRIME successful */ - - /* SMC returned with call_again flag set */ - if (ktime_before(ktime_get(), end)) { - udelay(LFA_PRIME_POLL_DELAY_US); - continue; } - - pr_err("LFA_PRIME for image %s timed out", attrs->image_name); - mutex_unlock(&lfa_lock); - - ret = lfa_cancel(attrs); - if (ret != 0) - return ret; - return -ETIMEDOUT; } mutex_unlock(&lfa_lock); From 076695dc782ef912e5965afb5dc008a895c53461 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 20 Apr 2026 23:29:30 -0700 Subject: [PATCH 170/464] Revert "NVIDIA: VR: SAUCE: firmware: smccc: add support for Live Firmware Activation (LFA)" BugLink: https://bugs.launchpad.net/bugs/2150652 This reverts commit dc371707478752e2627fe104a1424685fc1cb36f. Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/Kconfig | 8 - drivers/firmware/smccc/Makefile | 1 - drivers/firmware/smccc/lfa_fw.c | 670 -------------------------------- 3 files changed, 679 deletions(-) delete mode 100644 drivers/firmware/smccc/lfa_fw.c diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig index ff7ca49486b05..15e7466179a62 100644 --- a/drivers/firmware/smccc/Kconfig +++ b/drivers/firmware/smccc/Kconfig @@ -23,11 +23,3 @@ config ARM_SMCCC_SOC_ID help Include support for the SoC bus on the ARM SMCCC firmware based platforms providing some sysfs information about the SoC variant. - -config ARM_LFA - tristate "Arm Live Firmware activation support" - depends on HAVE_ARM_SMCCC_DISCOVERY - default y - help - Include support for triggering Live Firmware Activation, which - allows to upgrade certain firmware components without a reboot. diff --git a/drivers/firmware/smccc/Makefile b/drivers/firmware/smccc/Makefile index a6dd01558a94a..40d19144a8607 100644 --- a/drivers/firmware/smccc/Makefile +++ b/drivers/firmware/smccc/Makefile @@ -2,4 +2,3 @@ # obj-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smccc.o kvm_guest.o obj-$(CONFIG_ARM_SMCCC_SOC_ID) += soc_id.o -obj-$(CONFIG_ARM_LFA) += lfa_fw.o diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c deleted file mode 100644 index adf4171066437..0000000000000 --- a/drivers/firmware/smccc/lfa_fw.c +++ /dev/null @@ -1,670 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Copyright (C) 2025 Arm Limited - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#undef pr_fmt -#define pr_fmt(fmt) "Arm LFA: " fmt - -/* LFA v1.0b0 specification */ -#define LFA_1_0_FN_BASE 0xc40002e0 -#define LFA_1_0_FN(n) (LFA_1_0_FN_BASE + (n)) - -#define LFA_1_0_FN_GET_VERSION LFA_1_0_FN(0) -#define LFA_1_0_FN_CHECK_FEATURE LFA_1_0_FN(1) -#define LFA_1_0_FN_GET_INFO LFA_1_0_FN(2) -#define LFA_1_0_FN_GET_INVENTORY LFA_1_0_FN(3) -#define LFA_1_0_FN_PRIME LFA_1_0_FN(4) -#define LFA_1_0_FN_ACTIVATE LFA_1_0_FN(5) -#define LFA_1_0_FN_CANCEL LFA_1_0_FN(6) - -/* CALL_AGAIN flags (returned by SMC) */ -#define LFA_PRIME_CALL_AGAIN BIT(0) -#define LFA_ACTIVATE_CALL_AGAIN BIT(0) - -/* LFA return values */ -#define LFA_SUCCESS 0 -#define LFA_NOT_SUPPORTED 1 -#define LFA_BUSY 2 -#define LFA_AUTH_ERROR 3 -#define LFA_NO_MEMORY 4 -#define LFA_CRITICAL_ERROR 5 -#define LFA_DEVICE_ERROR 6 -#define LFA_WRONG_STATE 7 -#define LFA_INVALID_PARAMETERS 8 -#define LFA_COMPONENT_WRONG_STATE 9 -#define LFA_INVALID_ADDRESS 10 -#define LFA_ACTIVATION_FAILED 11 - -#define LFA_ERROR_STRING(name) \ - [name] = #name - -static const char * const lfa_error_strings[] = { - LFA_ERROR_STRING(LFA_SUCCESS), - LFA_ERROR_STRING(LFA_NOT_SUPPORTED), - LFA_ERROR_STRING(LFA_BUSY), - LFA_ERROR_STRING(LFA_AUTH_ERROR), - LFA_ERROR_STRING(LFA_NO_MEMORY), - LFA_ERROR_STRING(LFA_CRITICAL_ERROR), - LFA_ERROR_STRING(LFA_DEVICE_ERROR), - LFA_ERROR_STRING(LFA_WRONG_STATE), - LFA_ERROR_STRING(LFA_INVALID_PARAMETERS), - LFA_ERROR_STRING(LFA_COMPONENT_WRONG_STATE), - LFA_ERROR_STRING(LFA_INVALID_ADDRESS), - LFA_ERROR_STRING(LFA_ACTIVATION_FAILED) -}; - -enum image_attr_names { - LFA_ATTR_NAME, - LFA_ATTR_CURRENT_VERSION, - LFA_ATTR_PENDING_VERSION, - LFA_ATTR_ACT_CAPABLE, - LFA_ATTR_ACT_PENDING, - LFA_ATTR_MAY_RESET_CPU, - LFA_ATTR_CPU_RENDEZVOUS, - LFA_ATTR_FORCE_CPU_RENDEZVOUS, - LFA_ATTR_ACTIVATE, - LFA_ATTR_CANCEL, - LFA_ATTR_NR_IMAGES -}; - -struct image_props { - struct list_head image_node; - const char *image_name; - int fw_seq_id; - u64 current_version; - u64 pending_version; - bool activation_capable; - bool activation_pending; - bool may_reset_cpu; - bool cpu_rendezvous; - bool cpu_rendezvous_forced; - struct kobject *image_dir; - struct kobj_attribute image_attrs[LFA_ATTR_NR_IMAGES]; -}; -static LIST_HEAD(lfa_fw_images); - -/* A UUID split over two 64-bit registers */ -struct uuid_regs { - u64 uuid_lo; - u64 uuid_hi; -}; - -static const struct fw_image_uuid { - const char *name; - const char *uuid; -} fw_images_uuids[] = { - { - .name = "TF-A BL31 runtime", - .uuid = "47d4086d-4cfe-9846-9b95-2950cbbd5a00", - }, - { - .name = "BL33 non-secure payload", - .uuid = "d6d0eea7-fcea-d54b-9782-9934f234b6e4", - }, - { - .name = "TF-RMM", - .uuid = "6c0762a6-12f2-4b56-92cb-ba8f633606d9", - }, -}; - -static struct kobject *lfa_dir; -static DEFINE_MUTEX(lfa_lock); -static struct workqueue_struct *fw_images_update_wq; -static struct work_struct fw_images_update_work; - -static int update_fw_images_tree(void); - -static void delete_fw_image_node(struct image_props *attrs) -{ - int i; - - for (i = 0; i < LFA_ATTR_NR_IMAGES; i++) - sysfs_remove_file(attrs->image_dir, &attrs->image_attrs[i].attr); - - kobject_put(attrs->image_dir); - list_del(&attrs->image_node); - kfree(attrs); -} - -static void remove_invalid_fw_images(struct work_struct *work) -{ - struct image_props *attrs, *tmp; - - mutex_lock(&lfa_lock); - - /* - * Remove firmware images including directories that are no longer - * present in the LFA agent after updating the existing ones. - */ - list_for_each_entry_safe(attrs, tmp, &lfa_fw_images, image_node) { - if (attrs->fw_seq_id == -1) - delete_fw_image_node(attrs); - } - - mutex_unlock(&lfa_lock); -} - -static void set_image_flags(struct image_props *attrs, int seq_id, - u32 image_flags, u64 reg_current_ver, - u64 reg_pending_ver) -{ - attrs->fw_seq_id = seq_id; - attrs->current_version = reg_current_ver; - attrs->pending_version = reg_pending_ver; - attrs->activation_capable = !!(image_flags & BIT(0)); - attrs->activation_pending = !!(image_flags & BIT(1)); - attrs->may_reset_cpu = !!(image_flags & BIT(2)); - /* cpu_rendezvous_optional bit has inverse logic in the spec */ - attrs->cpu_rendezvous = !(image_flags & BIT(3)); -} - -static unsigned long get_nr_lfa_components(void) -{ - struct arm_smccc_1_2_regs reg = { 0 }; - - reg.a0 = LFA_1_0_FN_GET_INFO; - reg.a1 = 0; /* lfa_info_selector = 0 */ - - arm_smccc_1_2_invoke(®, ®); - if (reg.a0 != LFA_SUCCESS) - return reg.a0; - - return reg.a1; -} - -static int lfa_cancel(void *data) -{ - struct image_props *attrs = data; - struct arm_smccc_1_2_regs reg = { 0 }; - - reg.a0 = LFA_1_0_FN_CANCEL; - reg.a1 = attrs->fw_seq_id; - arm_smccc_1_2_invoke(®, ®); - - /* - * When firmware activation is called with "skip_cpu_rendezvous=1", - * LFA_CANCEL can fail with LFA_BUSY if the activation could not be - * cancelled. - */ - if (reg.a0 == LFA_SUCCESS) { - pr_info("Activation cancelled for image %s\n", - attrs->image_name); - } else { - pr_err("Firmware activation could not be cancelled: %s\n", - lfa_error_strings[-reg.a0]); - return -EINVAL; - } - - return reg.a0; -} - -static int call_lfa_activate(void *data) -{ - struct image_props *attrs = data; - struct arm_smccc_1_2_regs reg = { 0 }; - - reg.a0 = LFA_1_0_FN_ACTIVATE; - reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ - /* - * As we do not support updates requiring a CPU reset (yet), - * we pass 0 in reg.a3 and reg.a4, holding the entry point and context - * ID respectively. - * cpu_rendezvous_forced is set by the administrator, via sysfs, - * cpu_rendezvous is dictated by each firmware component. - */ - reg.a2 = !(attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous); - - for (;;) { - arm_smccc_1_2_invoke(®, ®); - - if ((long)reg.a0 < 0) { - pr_err("ACTIVATE for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-reg.a0]); - return reg.a0; - } - if (!(reg.a1 & LFA_ACTIVATE_CALL_AGAIN)) - break; /* ACTIVATE successful */ - } - - return reg.a0; -} - -static int activate_fw_image(struct image_props *attrs) -{ - int ret; - - mutex_lock(&lfa_lock); - if (attrs->cpu_rendezvous_forced || attrs->cpu_rendezvous) - ret = stop_machine(call_lfa_activate, attrs, cpu_online_mask); - else - ret = call_lfa_activate(attrs); - - if (ret != 0) { - mutex_unlock(&lfa_lock); - return lfa_cancel(attrs); - } - - /* - * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the - * number of firmware images in the LFA agent may change after a - * successful activation attempt. Negate all image flags as well. - */ - attrs = NULL; - list_for_each_entry(attrs, &lfa_fw_images, image_node) { - set_image_flags(attrs, -1, 0b1000, 0, 0); - } - - update_fw_images_tree(); - - /* - * Removing non-valid image directories at the end of an activation. - * We can't remove the sysfs attributes while in the respective - * _store() handler, so have to postpone the list removal to a - * workqueue. - */ - INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); - queue_work(fw_images_update_wq, &fw_images_update_work); - mutex_unlock(&lfa_lock); - - return ret; -} - -static int prime_fw_image(struct image_props *attrs) -{ - struct arm_smccc_1_2_regs reg = { 0 }; - int ret; - - mutex_lock(&lfa_lock); - /* Avoid SMC calls on invalid firmware images */ - if (attrs->fw_seq_id == -1) { - pr_err("Arm LFA: Invalid firmware sequence id\n"); - mutex_unlock(&lfa_lock); - - return -ENODEV; - } - - if (attrs->may_reset_cpu) { - pr_err("CPU reset not supported by kernel driver\n"); - mutex_unlock(&lfa_lock); - - return -EINVAL; - } - - /* - * LFA_PRIME/ACTIVATE will return 1 in reg.a1 if the firmware - * priming/activation is still in progress. In that case - * LFA_PRIME/ACTIVATE will need to be called again. - * reg.a1 will become 0 once the prime/activate process completes. - */ - reg.a0 = LFA_1_0_FN_PRIME; - reg.a1 = attrs->fw_seq_id; /* fw_seq_id under consideration */ - for (;;) { - arm_smccc_1_2_invoke(®, ®); - - if ((long)reg.a0 < 0) { - pr_err("LFA_PRIME for image %s failed: %s\n", - attrs->image_name, lfa_error_strings[-reg.a0]); - mutex_unlock(&lfa_lock); - - return reg.a0; - } - if (!(reg.a1 & LFA_PRIME_CALL_AGAIN)) { - ret = 0; - break; /* PRIME successful */ - } - } - - mutex_unlock(&lfa_lock); - return ret; -} - -static ssize_t name_show(struct kobject *kobj, struct kobj_attribute *attr, - char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_NAME]); - - return sysfs_emit(buf, "%s\n", attrs->image_name); -} - -static ssize_t activation_capable_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_ACT_CAPABLE]); - - return sysfs_emit(buf, "%d\n", attrs->activation_capable); -} - -static ssize_t activation_pending_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_ACT_PENDING]); - struct arm_smccc_1_2_regs reg = { 0 }; - - /* - * Activation pending status can change anytime thus we need to update - * and return its current value - */ - reg.a0 = LFA_1_0_FN_GET_INVENTORY; - reg.a1 = attrs->fw_seq_id; - arm_smccc_1_2_invoke(®, ®); - if (reg.a0 == LFA_SUCCESS) - attrs->activation_pending = !!(reg.a3 & BIT(1)); - - return sysfs_emit(buf, "%d\n", attrs->activation_pending); -} - -static ssize_t may_reset_cpu_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_MAY_RESET_CPU]); - - return sysfs_emit(buf, "%d\n", attrs->may_reset_cpu); -} - -static ssize_t cpu_rendezvous_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_CPU_RENDEZVOUS]); - - return sysfs_emit(buf, "%d\n", attrs->cpu_rendezvous); -} - -static ssize_t force_cpu_rendezvous_store(struct kobject *kobj, - struct kobj_attribute *attr, - const char *buf, size_t count) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_FORCE_CPU_RENDEZVOUS]); - int ret; - - ret = kstrtobool(buf, &attrs->cpu_rendezvous_forced); - if (ret) - return ret; - - return count; -} - -static ssize_t force_cpu_rendezvous_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_FORCE_CPU_RENDEZVOUS]); - - return sysfs_emit(buf, "%d\n", attrs->cpu_rendezvous_forced); -} - -static ssize_t current_version_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_CURRENT_VERSION]); - u32 maj, min; - - maj = attrs->current_version >> 32; - min = attrs->current_version & 0xffffffff; - return sysfs_emit(buf, "%u.%u\n", maj, min); -} - -static ssize_t pending_version_show(struct kobject *kobj, - struct kobj_attribute *attr, char *buf) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_ACT_PENDING]); - struct arm_smccc_1_2_regs reg = { 0 }; - u32 maj, min; - - /* - * Similar to activation pending, this value can change following an - * update, we need to retrieve fresh info instead of stale information. - */ - reg.a0 = LFA_1_0_FN_GET_INVENTORY; - reg.a1 = attrs->fw_seq_id; - arm_smccc_1_2_invoke(®, ®); - if (reg.a0 == LFA_SUCCESS) { - if (reg.a5 != 0 && attrs->activation_pending) - { - attrs->pending_version = reg.a5; - maj = reg.a5 >> 32; - min = reg.a5 & 0xffffffff; - } - } - - return sysfs_emit(buf, "%u.%u\n", maj, min); -} - -static ssize_t activate_store(struct kobject *kobj, struct kobj_attribute *attr, - const char *buf, size_t count) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_ACTIVATE]); - int ret; - - ret = prime_fw_image(attrs); - if (ret) { - pr_err("Firmware prime failed: %s\n", - lfa_error_strings[-ret]); - return -ECANCELED; - } - - ret = activate_fw_image(attrs); - if (ret) { - pr_err("Firmware activation failed: %s\n", - lfa_error_strings[-ret]); - return -ECANCELED; - } - - pr_info("Firmware activation succeeded\n"); - - return count; -} - -static ssize_t cancel_store(struct kobject *kobj, struct kobj_attribute *attr, - const char *buf, size_t count) -{ - struct image_props *attrs = container_of(attr, struct image_props, - image_attrs[LFA_ATTR_CANCEL]); - int ret; - - ret = lfa_cancel(attrs); - if (ret != 0) - return ret; - - return count; -} - -static struct kobj_attribute image_attrs_group[LFA_ATTR_NR_IMAGES] = { - [LFA_ATTR_NAME] = __ATTR_RO(name), - [LFA_ATTR_CURRENT_VERSION] = __ATTR_RO(current_version), - [LFA_ATTR_PENDING_VERSION] = __ATTR_RO(pending_version), - [LFA_ATTR_ACT_CAPABLE] = __ATTR_RO(activation_capable), - [LFA_ATTR_ACT_PENDING] = __ATTR_RO(activation_pending), - [LFA_ATTR_MAY_RESET_CPU] = __ATTR_RO(may_reset_cpu), - [LFA_ATTR_CPU_RENDEZVOUS] = __ATTR_RO(cpu_rendezvous), - [LFA_ATTR_FORCE_CPU_RENDEZVOUS] = __ATTR_RW(force_cpu_rendezvous), - [LFA_ATTR_ACTIVATE] = __ATTR_WO(activate), - [LFA_ATTR_CANCEL] = __ATTR_WO(cancel) -}; - -static void clean_fw_images_tree(void) -{ - struct image_props *attrs, *tmp; - - list_for_each_entry_safe(attrs, tmp, &lfa_fw_images, image_node) - delete_fw_image_node(attrs); -} - -static int update_fw_image_node(char *fw_uuid, int seq_id, - u32 image_flags, u64 reg_current_ver, - u64 reg_pending_ver) -{ - const char *image_name = "(unknown)"; - struct image_props *attrs; - int ret; - - /* - * If a fw_image is already in the images list then we just update - * its flags and seq_id instead of trying to recreate it. - */ - list_for_each_entry(attrs, &lfa_fw_images, image_node) { - if (!strcmp(attrs->image_dir->name, fw_uuid)) { - set_image_flags(attrs, seq_id, image_flags, - reg_current_ver, reg_pending_ver); - return 0; - } - } - - attrs = kzalloc(sizeof(*attrs), GFP_KERNEL); - if (!attrs) - return -ENOMEM; - - for (int i = 0; i < ARRAY_SIZE(fw_images_uuids); i++) { - if (!strcmp(fw_images_uuids[i].uuid, fw_uuid)) - image_name = fw_images_uuids[i].name; - else - image_name = fw_uuid; - } - - attrs->image_dir = kobject_create_and_add(fw_uuid, lfa_dir); - if (!attrs->image_dir) - return -ENOMEM; - - INIT_LIST_HEAD(&attrs->image_node); - attrs->image_name = image_name; - attrs->cpu_rendezvous_forced = 1; - set_image_flags(attrs, seq_id, image_flags, reg_current_ver, - reg_pending_ver); - - /* - * The attributes for each sysfs file are constant (handler functions, - * name and permissions are the same within each directory), but we - * need a per-directory copy regardless, to get a unique handle - * for each directory, so that container_of can do its magic. - * Also this requires an explicit sysfs_attr_init(), since it's a new - * copy, to make LOCKDEP happy. - */ - memcpy(attrs->image_attrs, image_attrs_group, - sizeof(attrs->image_attrs)); - for (int i = 0; i < LFA_ATTR_NR_IMAGES; i++) { - struct attribute *attr = &attrs->image_attrs[i].attr; - - sysfs_attr_init(attr); - ret = sysfs_create_file(attrs->image_dir, attr); - if (ret) { - pr_err("creating sysfs file for uuid %s: %d\n", - fw_uuid, ret); - clean_fw_images_tree(); - - return ret; - } - } - list_add(&attrs->image_node, &lfa_fw_images); - - return ret; -} - -static int update_fw_images_tree(void) -{ - struct arm_smccc_1_2_regs reg = { 0 }; - struct uuid_regs image_uuid; - char image_id_str[40]; - int ret, num_of_components; - - num_of_components = get_nr_lfa_components(); - if (num_of_components <= 0) { - pr_err("Error getting number of LFA components\n"); - return -ENODEV; - } - - for (int i = 0; i < num_of_components; i++) { - reg.a0 = LFA_1_0_FN_GET_INVENTORY; - reg.a1 = i; /* fw_seq_id under consideration */ - arm_smccc_1_2_invoke(®, ®); - if (reg.a0 == LFA_SUCCESS) { - image_uuid.uuid_lo = reg.a1; - image_uuid.uuid_hi = reg.a2; - - snprintf(image_id_str, sizeof(image_id_str), "%pUb", - &image_uuid); - ret = update_fw_image_node(image_id_str, i, - reg.a3, reg.a4, reg.a5); - if (ret) - return ret; - } - } - - return 0; -} - -static int __init lfa_init(void) -{ - struct arm_smccc_1_2_regs reg = { 0 }; - int err; - - reg.a0 = LFA_1_0_FN_GET_VERSION; - arm_smccc_1_2_invoke(®, ®); - if (reg.a0 == -LFA_NOT_SUPPORTED) { - pr_info("Live Firmware activation: no firmware agent found\n"); - return -ENODEV; - } - - fw_images_update_wq = alloc_workqueue("fw_images_update_wq", - WQ_UNBOUND | WQ_MEM_RECLAIM, 1); - if (!fw_images_update_wq) { - pr_err("Live Firmware Activation: Failed to allocate workqueue.\n"); - - return -ENOMEM; - } - - pr_info("Live Firmware Activation: detected v%ld.%ld\n", - reg.a0 >> 16, reg.a0 & 0xffff); - - lfa_dir = kobject_create_and_add("lfa", firmware_kobj); - if (!lfa_dir) - return -ENOMEM; - - mutex_lock(&lfa_lock); - err = update_fw_images_tree(); - if (err != 0) - kobject_put(lfa_dir); - - mutex_unlock(&lfa_lock); - return err; -} -module_init(lfa_init); - -static void __exit lfa_exit(void) -{ - flush_workqueue(fw_images_update_wq); - destroy_workqueue(fw_images_update_wq); - - mutex_lock(&lfa_lock); - clean_fw_images_tree(); - mutex_unlock(&lfa_lock); - - kobject_put(lfa_dir); -} -module_exit(lfa_exit); - -MODULE_DESCRIPTION("ARM Live Firmware Activation (LFA)"); -MODULE_LICENSE("GPL"); From 463268b89c870022520e437544f297789d3c1ca3 Mon Sep 17 00:00:00 2001 From: Andre Przywara Date: Tue, 17 Mar 2026 11:33:27 +0100 Subject: [PATCH 171/464] NVIDIA: VR: SAUCE: dt-bindings: arm: Add Live Firmware Activation binding BugLink: https://bugs.launchpad.net/bugs/2150652 The Arm Live Firmware Activation spec [1] describes updating firmware images during runtime, without requiring a reboot. Update images might be deployed out-of-band, for instance via a BMC, in this case the OS needs to be notified about the availability of a new image. This binding describes an interrupt that could be triggered by the platform, to notify about any changes. [1] https://developer.arm.com/documentation/den0147/latest/ Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- .../devicetree/bindings/arm/arm,lfa.yaml | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 Documentation/devicetree/bindings/arm/arm,lfa.yaml diff --git a/Documentation/devicetree/bindings/arm/arm,lfa.yaml b/Documentation/devicetree/bindings/arm/arm,lfa.yaml new file mode 100644 index 0000000000000..92f0564fd672e --- /dev/null +++ b/Documentation/devicetree/bindings/arm/arm,lfa.yaml @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/arm/arm,lfa.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Arm Live Firmware Activation (LFA) + +maintainers: + - Andre Przywara + - Sudeep Holla + +description: + The Arm Live Firmware Activation (LFA) specification [1] describes a + firmware interface to activate an updated firmware at runtime, without + requiring a reboot. Updates might be supplied out-of-band, for instance + via a BMC, in which case the platform needs to notify an OS about pending + image updates. + [1] https://developer.arm.com/documentation/den0147/latest/ + +properties: + compatible: + const: arm,lfa + + interrupts: + maxItems: 1 + description: notification interrupt for changed firmware image status + +required: + - compatible + - interrupts + +additionalProperties: false + +examples: + - | + #include + + firmware { + arm-lfa { + compatible = "arm,lfa"; + interrupts = ; + }; + }; +... From 84914689394ee6c30a1d73e37d56d2e9e1416e49 Mon Sep 17 00:00:00 2001 From: Salman Nabi Date: Tue, 17 Mar 2026 11:33:28 +0100 Subject: [PATCH 172/464] NVIDIA: VR: SAUCE: firmware: smccc: Add support for Live Firmware Activation (LFA) BugLink: https://bugs.launchpad.net/bugs/2150652 The Arm Live Firmware Activation (LFA) is a specification [1] to describe activating firmware components without a reboot. Those components (like TF-A's BL31, EDK-II, TF-RMM, secure paylods) would be updated the usual way: via fwupd, FF-A or other secure storage methods, or via some IMPDEF Out-Of-Bound method. The user can then activate this new firmware, at system runtime, without requiring a reboot. The specification covers the SMCCC interface to list and query available components and eventually trigger the activation. Add a new directory under /sys/firmware to present firmware components capable of live activation. Each of them is a directory under lfa/, and is identified via its GUID. The activation will be triggered by echoing "1" into the "activate" file: ========================================== /sys/firmware/lfa # ls -l . 6c* .: total 0 drwxr-xr-x 2 0 0 0 Jan 19 11:33 47d4086d-4cfe-9846-9b95-2950cbbd5a00 drwxr-xr-x 2 0 0 0 Jan 19 11:33 6c0762a6-12f2-4b56-92cb-ba8f633606d9 drwxr-xr-x 2 0 0 0 Jan 19 11:33 d6d0eea7-fcea-d54b-9782-9934f234b6e4 6c0762a6-12f2-4b56-92cb-ba8f633606d9: total 0 --w------- 1 0 0 4096 Jan 19 11:33 activate -r--r--r-- 1 0 0 4096 Jan 19 11:33 activation_capable -r--r--r-- 1 0 0 4096 Jan 19 11:33 activation_pending --w------- 1 0 0 4096 Jan 19 11:33 cancel -r--r--r-- 1 0 0 4096 Jan 19 11:33 cpu_rendezvous -r--r--r-- 1 0 0 4096 Jan 19 11:33 current_version -rw-r--r-- 1 0 0 4096 Jan 19 11:33 force_cpu_rendezvous -r--r--r-- 1 0 0 4096 Jan 19 11:33 may_reset_cpu -r--r--r-- 1 0 0 4096 Jan 19 11:33 name -r--r--r-- 1 0 0 4096 Jan 19 11:33 pending_version /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # grep . * grep: activate: Permission denied activation_capable:1 activation_pending:1 grep: cancel: Permission denied cpu_rendezvous:1 current_version:0.0 force_cpu_rendezvous:1 may_reset_cpu:0 name:TF-RMM pending_version:0.0 /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # echo 1 > activate [ 2825.797871] Arm LFA: firmware activation succeeded. /sys/firmware/lfa/6c0762a6-12f2-4b56-92cb-ba8f633606d9 # ========================================== [1] https://developer.arm.com/documentation/den0147/latest/ Signed-off-by: Salman Nabi Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/Kconfig | 10 + drivers/firmware/smccc/Makefile | 1 + drivers/firmware/smccc/lfa_fw.c | 721 ++++++++++++++++++++++++++++++++ 3 files changed, 732 insertions(+) create mode 100644 drivers/firmware/smccc/lfa_fw.c diff --git a/drivers/firmware/smccc/Kconfig b/drivers/firmware/smccc/Kconfig index 15e7466179a62..7fd646d515f86 100644 --- a/drivers/firmware/smccc/Kconfig +++ b/drivers/firmware/smccc/Kconfig @@ -23,3 +23,13 @@ config ARM_SMCCC_SOC_ID help Include support for the SoC bus on the ARM SMCCC firmware based platforms providing some sysfs information about the SoC variant. + +config ARM_LFA + tristate "Arm Live Firmware activation support" + depends on HAVE_ARM_SMCCC_DISCOVERY && ARM64 + default y + help + Include support for triggering a Live Firmware Activation (LFA), + which allows to upgrade certain firmware components without a reboot. + This is described in the Arm DEN0147 specification, and relies on + a firmware agent running in EL3. diff --git a/drivers/firmware/smccc/Makefile b/drivers/firmware/smccc/Makefile index 40d19144a8607..a6dd01558a94a 100644 --- a/drivers/firmware/smccc/Makefile +++ b/drivers/firmware/smccc/Makefile @@ -2,3 +2,4 @@ # obj-$(CONFIG_HAVE_ARM_SMCCC_DISCOVERY) += smccc.o kvm_guest.o obj-$(CONFIG_ARM_SMCCC_SOC_ID) += soc_id.o +obj-$(CONFIG_ARM_LFA) += lfa_fw.o diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c new file mode 100644 index 0000000000000..284b7c18d3d09 --- /dev/null +++ b/drivers/firmware/smccc/lfa_fw.c @@ -0,0 +1,721 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 Arm Limited + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#undef pr_fmt +#define pr_fmt(fmt) "Arm LFA: " fmt + +/* LFA v1.0b0 specification */ +#define LFA_1_0_FN_BASE 0xc40002e0 +#define LFA_1_0_FN(n) (LFA_1_0_FN_BASE + (n)) + +#define LFA_1_0_FN_GET_VERSION LFA_1_0_FN(0) +#define LFA_1_0_FN_CHECK_FEATURE LFA_1_0_FN(1) +#define LFA_1_0_FN_GET_INFO LFA_1_0_FN(2) +#define LFA_1_0_FN_GET_INVENTORY LFA_1_0_FN(3) +#define LFA_1_0_FN_PRIME LFA_1_0_FN(4) +#define LFA_1_0_FN_ACTIVATE LFA_1_0_FN(5) +#define LFA_1_0_FN_CANCEL LFA_1_0_FN(6) + +/* CALL_AGAIN flags (returned by SMC) */ +#define LFA_PRIME_CALL_AGAIN BIT(0) +#define LFA_ACTIVATE_CALL_AGAIN BIT(0) + +/* LFA return values */ +#define LFA_SUCCESS 0 +#define LFA_NOT_SUPPORTED 1 +#define LFA_BUSY 2 +#define LFA_AUTH_ERROR 3 +#define LFA_NO_MEMORY 4 +#define LFA_CRITICAL_ERROR 5 +#define LFA_DEVICE_ERROR 6 +#define LFA_WRONG_STATE 7 +#define LFA_INVALID_PARAMETERS 8 +#define LFA_COMPONENT_WRONG_STATE 9 +#define LFA_INVALID_ADDRESS 10 +#define LFA_ACTIVATION_FAILED 11 + +/* + * Not error codes described by the spec, but used internally when + * PRIME/ACTIVATE calls return with the CALL_AGAIN bit set. + */ +#define LFA_TIMED_OUT 32 +#define LFA_CALL_AGAIN 33 + +#define LFA_ERROR_STRING(name) \ + [name] = #name + +static const char * const lfa_error_strings[] = { + LFA_ERROR_STRING(LFA_SUCCESS), + LFA_ERROR_STRING(LFA_NOT_SUPPORTED), + LFA_ERROR_STRING(LFA_BUSY), + LFA_ERROR_STRING(LFA_AUTH_ERROR), + LFA_ERROR_STRING(LFA_NO_MEMORY), + LFA_ERROR_STRING(LFA_CRITICAL_ERROR), + LFA_ERROR_STRING(LFA_DEVICE_ERROR), + LFA_ERROR_STRING(LFA_WRONG_STATE), + LFA_ERROR_STRING(LFA_INVALID_PARAMETERS), + LFA_ERROR_STRING(LFA_COMPONENT_WRONG_STATE), + LFA_ERROR_STRING(LFA_INVALID_ADDRESS), + LFA_ERROR_STRING(LFA_ACTIVATION_FAILED) +}; + +enum image_attr_names { + LFA_ATTR_NAME, + LFA_ATTR_CURRENT_VERSION, + LFA_ATTR_PENDING_VERSION, + LFA_ATTR_ACT_CAPABLE, + LFA_ATTR_ACT_PENDING, + LFA_ATTR_MAY_RESET_CPU, + LFA_ATTR_CPU_RENDEZVOUS, + LFA_ATTR_FORCE_CPU_RENDEZVOUS, + LFA_ATTR_ACTIVATE, + LFA_ATTR_CANCEL, + LFA_ATTR_NR_IMAGES +}; + +struct fw_image { + struct kobject kobj; + const char *image_name; + int fw_seq_id; + u64 current_version; + u64 pending_version; + bool activation_capable; + bool activation_pending; + bool may_reset_cpu; + bool cpu_rendezvous; + bool cpu_rendezvous_forced; + struct kobj_attribute image_attrs[LFA_ATTR_NR_IMAGES]; +}; + +static struct fw_image *kobj_to_fw_image(struct kobject *kobj) +{ + return container_of(kobj, struct fw_image, kobj); +} + +/* A UUID split over two 64-bit registers */ +struct uuid_regs { + u64 uuid_lo; + u64 uuid_hi; +}; + +/* A list of known GUIDs, to be shown in the "name" sysfs file. */ +static const struct fw_image_uuid { + const char *name; + const char *uuid; +} fw_images_uuids[] = { + { + .name = "TF-A BL31 runtime", + .uuid = "47d4086d-4cfe-9846-9b95-2950cbbd5a00", + }, + { + .name = "BL33 non-secure payload", + .uuid = "d6d0eea7-fcea-d54b-9782-9934f234b6e4", + }, + { + .name = "TF-RMM", + .uuid = "6c0762a6-12f2-4b56-92cb-ba8f633606d9", + }, +}; + +static struct kset *lfa_kset; +static struct workqueue_struct *fw_images_update_wq; +static struct work_struct fw_images_update_work; +static struct attribute *image_default_attrs[LFA_ATTR_NR_IMAGES + 1]; + +static const struct attribute_group image_attr_group = { + .attrs = image_default_attrs, +}; + +static const struct attribute_group *image_default_groups[] = { + &image_attr_group, + NULL +}; + +static int update_fw_images_tree(void); + +static const char *lfa_error_string(int error) +{ + if (error > 0) + return lfa_error_strings[LFA_SUCCESS]; + + error = -error; + if (error < ARRAY_SIZE(lfa_error_strings)) + return lfa_error_strings[error]; + if (error == -LFA_TIMED_OUT) + return "timed out"; + + return lfa_error_strings[LFA_DEVICE_ERROR]; +} + +static void image_release(struct kobject *kobj) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + kfree(image); +} + +static const struct kobj_type image_ktype = { + .release = image_release, + .sysfs_ops = &kobj_sysfs_ops, + .default_groups = image_default_groups, +}; + +static void delete_fw_image_node(struct fw_image *image) +{ + kobject_del(&image->kobj); + kobject_put(&image->kobj); +} + +static void remove_invalid_fw_images(struct work_struct *work) +{ + struct kobject *kobj, *tmp; + struct list_head images_to_delete = LIST_HEAD_INIT(images_to_delete); + + /* + * Remove firmware images including directories that are no longer + * present in the LFA agent after updating the existing ones. + * Delete list images before calling kobject_del() and kobject_put() on + * them. Kobject_del() uses kset->list_lock itself which can cause lock + * recursion, and kobject_put() may sleep. + */ + spin_lock(&lfa_kset->list_lock); + list_for_each_entry_safe(kobj, tmp, &lfa_kset->list, entry) { + struct fw_image *image = kobj_to_fw_image(kobj); + + if (image->fw_seq_id == -1) + list_move_tail(&kobj->entry, &images_to_delete); + } + spin_unlock(&lfa_kset->list_lock); + + /* + * Now safely remove the sysfs kobjects for the deleted list items + */ + list_for_each_entry_safe(kobj, tmp, &images_to_delete, entry) { + struct fw_image *image = kobj_to_fw_image(kobj); + + delete_fw_image_node(image); + } +} + +static void set_image_flags(struct fw_image *image, int seq_id, + u32 image_flags, u64 reg_current_ver, + u64 reg_pending_ver) +{ + image->fw_seq_id = seq_id; + image->current_version = reg_current_ver; + image->pending_version = reg_pending_ver; + image->activation_capable = !!(image_flags & BIT(0)); + image->activation_pending = !!(image_flags & BIT(1)); + image->may_reset_cpu = !!(image_flags & BIT(2)); + /* cpu_rendezvous_optional bit has inverse logic in the spec */ + image->cpu_rendezvous = !(image_flags & BIT(3)); +} + +static unsigned long get_nr_lfa_components(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_GET_INFO; + reg.a1 = 0; /* lfa_info_selector = 0 */ + + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 != LFA_SUCCESS) + return reg.a0; + + return reg.a1; +} + +static int lfa_cancel(void *data) +{ + struct fw_image *image = data; + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_CANCEL; + reg.a1 = image->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + + /* + * When firmware activation is called with "skip_cpu_rendezvous=1", + * LFA_CANCEL can fail with LFA_BUSY if the activation could not be + * cancelled. + */ + if (reg.a0 == LFA_SUCCESS) { + pr_info("Activation cancelled for image %s\n", + image->image_name); + } else { + pr_err("Activation not cancelled for image %s: %s\n", + image->image_name, lfa_error_string(reg.a0)); + return -EINVAL; + } + + return reg.a0; +} + +static const char *get_image_name(const struct fw_image *image) +{ + if (image->image_name && image->image_name[0] != '\0') + return image->image_name; + + return kobject_name(&image->kobj); +} + +/* + * Try a single activation call. The smc_lock writer lock must be held, + * and it must be called from inside stop_machine() when CPU rendezvous is + * required. + */ +static int call_lfa_activate(void *data) +{ + struct fw_image *image = data; + struct arm_smccc_1_2_regs reg = { 0 }, res; + + reg.a0 = LFA_1_0_FN_ACTIVATE; + reg.a1 = image->fw_seq_id; + /* + * As we do not support updates requiring a CPU reset (yet), + * we pass 0 in reg.a3 and reg.a4, holding the entry point and + * context ID respectively. + * cpu_rendezvous_forced is set by the administrator, via sysfs, + * cpu_rendezvous is dictated by each firmware component. + */ + reg.a2 = !(image->cpu_rendezvous_forced || image->cpu_rendezvous); + arm_smccc_1_2_invoke(®, &res); + + if ((long)res.a0 < 0) + return (long)res.a0; + + if (res.a1 & LFA_ACTIVATE_CALL_AGAIN) + return -LFA_CALL_AGAIN; + + return 0; +} + +static int activate_fw_image(struct fw_image *image) +{ + struct kobject *kobj; + int ret; + +retry: + if (image->cpu_rendezvous_forced || image->cpu_rendezvous) + ret = stop_machine(call_lfa_activate, image, cpu_online_mask); + else + ret = call_lfa_activate(image); + + if (!ret) { + /* + * Invalidate fw_seq_ids (-1) for all images as the seq_ids + * and the number of firmware images in the LFA agent may + * change after a successful activation attempt. + * Negate all image flags as well. + */ + spin_lock(&lfa_kset->list_lock); + list_for_each_entry(kobj, &lfa_kset->list, entry) { + struct fw_image *image = kobj_to_fw_image(kobj); + + set_image_flags(image, -1, 0b1000, 0, 0); + } + spin_unlock(&lfa_kset->list_lock); + + update_fw_images_tree(); + + /* + * Removing non-valid image directories at the end of an + * activation. + * We can't remove the sysfs attributes while in the respective + * _store() handler, so have to postpone the list removal to a + * workqueue. + */ + queue_work(fw_images_update_wq, &fw_images_update_work); + + return 0; + } + + if (ret == -LFA_CALL_AGAIN) + goto retry; + + lfa_cancel(image); + + pr_err("LFA_ACTIVATE for image %s failed: %s\n", + get_image_name(image), lfa_error_string(ret)); + + return ret; +} + +static int prime_fw_image(struct fw_image *image) +{ + struct arm_smccc_1_2_regs reg = { 0 }, res; + + if (image->may_reset_cpu) { + pr_err("CPU reset not supported by kernel driver\n"); + + return -EINVAL; + } + + reg.a0 = LFA_1_0_FN_PRIME; +retry: + /* + * LFA_PRIME will return 1 in reg.a1 if the firmware priming + * is still in progress. In that case LFA_PRIME will need to + * be called again. + * reg.a1 will become 0 once the prime process completes. + */ + reg.a1 = image->fw_seq_id; + arm_smccc_1_2_invoke(®, &res); + if ((long)res.a0 < 0) { + pr_err("LFA_PRIME for image %s failed: %s\n", + get_image_name(image), + lfa_error_string(res.a0)); + + return res.a0; + } + + if (res.a1 & LFA_PRIME_CALL_AGAIN) + goto retry; + + return 0; +} + +static ssize_t name_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%s\n", image->image_name); +} + +static ssize_t activation_capable_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%d\n", image->activation_capable); +} + +static void update_fw_image_pending(struct fw_image *image) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + reg.a1 = image->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + + if (reg.a0 == LFA_SUCCESS) + image->activation_pending = !!(reg.a3 & BIT(1)); +} + +static ssize_t activation_pending_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + /* + * Activation pending status can change anytime thus we need to update + * and return its current value + */ + update_fw_image_pending(image); + + return sysfs_emit(buf, "%d\n", image->activation_pending); +} + +static ssize_t may_reset_cpu_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%d\n", image->may_reset_cpu); +} + +static ssize_t cpu_rendezvous_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%d\n", image->cpu_rendezvous); +} + +static ssize_t force_cpu_rendezvous_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + int ret; + + ret = kstrtobool(buf, &image->cpu_rendezvous_forced); + if (ret) + return ret; + + return count; +} + +static ssize_t force_cpu_rendezvous_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%d\n", image->cpu_rendezvous_forced); +} + +static ssize_t current_version_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + u32 maj, min; + + maj = image->current_version >> 32; + min = image->current_version & 0xffffffff; + + return sysfs_emit(buf, "%u.%u\n", maj, min); +} + +static ssize_t pending_version_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + struct arm_smccc_1_2_regs reg = { 0 }; + + /* + * Similar to activation pending, this value can change following an + * update, we need to retrieve fresh info instead of stale information. + */ + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + reg.a1 = image->fw_seq_id; + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == LFA_SUCCESS) { + if (reg.a5 != 0 && image->activation_pending) { + u32 maj, min; + + image->pending_version = reg.a5; + maj = reg.a5 >> 32; + min = reg.a5 & 0xffffffff; + + return sysfs_emit(buf, "%u.%u\n", maj, min); + } + } + + return sysfs_emit(buf, "N/A\n"); +} + +static ssize_t activate_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + int ret; + + ret = prime_fw_image(image); + if (ret) + return -ECANCELED; + + ret = activate_fw_image(image); + if (ret) + return -ECANCELED; + + pr_info("%s: successfully activated\n", get_image_name(image)); + + return count; +} + +static ssize_t cancel_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + int ret; + + ret = lfa_cancel(image); + if (ret != 0) + return ret; + + return count; +} + +static struct kobj_attribute image_attrs_group[LFA_ATTR_NR_IMAGES] = { + [LFA_ATTR_NAME] = __ATTR_RO(name), + [LFA_ATTR_CURRENT_VERSION] = __ATTR_RO(current_version), + [LFA_ATTR_PENDING_VERSION] = __ATTR_RO(pending_version), + [LFA_ATTR_ACT_CAPABLE] = __ATTR_RO(activation_capable), + [LFA_ATTR_ACT_PENDING] = __ATTR_RO(activation_pending), + [LFA_ATTR_MAY_RESET_CPU] = __ATTR_RO(may_reset_cpu), + [LFA_ATTR_CPU_RENDEZVOUS] = __ATTR_RO(cpu_rendezvous), + [LFA_ATTR_FORCE_CPU_RENDEZVOUS] = __ATTR_RW(force_cpu_rendezvous), + [LFA_ATTR_ACTIVATE] = __ATTR_WO(activate), + [LFA_ATTR_CANCEL] = __ATTR_WO(cancel) +}; + +static void init_image_default_attrs(void) +{ + for (int i = 0; i < LFA_ATTR_NR_IMAGES; i++) + image_default_attrs[i] = &image_attrs_group[i].attr; + image_default_attrs[LFA_ATTR_NR_IMAGES] = NULL; +} + +static void clean_fw_images_tree(void) +{ + struct kobject *kobj, *tmp; + struct list_head images_to_delete; + + INIT_LIST_HEAD(&images_to_delete); + + spin_lock(&lfa_kset->list_lock); + list_for_each_entry_safe(kobj, tmp, &lfa_kset->list, entry) { + list_move_tail(&kobj->entry, &images_to_delete); + } + spin_unlock(&lfa_kset->list_lock); + + list_for_each_entry_safe(kobj, tmp, &images_to_delete, entry) { + struct fw_image *image = kobj_to_fw_image(kobj); + + delete_fw_image_node(image); + } +} + +static int update_fw_image_node(char *fw_uuid, int seq_id, + u32 image_flags, u64 reg_current_ver, + u64 reg_pending_ver) +{ + const char *image_name = ""; + struct fw_image *image; + struct kobject *kobj; + int i; + + /* + * If a fw_image is already in the images list then we just update + * its flags and seq_id instead of trying to recreate it. + */ + spin_lock(&lfa_kset->list_lock); + list_for_each_entry(kobj, &lfa_kset->list, entry) { + if (!strcmp(kobject_name(kobj), fw_uuid)) { + struct fw_image *image = kobj_to_fw_image(kobj); + + set_image_flags(image, seq_id, image_flags, + reg_current_ver, reg_pending_ver); + spin_unlock(&lfa_kset->list_lock); + + return 0; + } + } + spin_unlock(&lfa_kset->list_lock); + + image = kzalloc_obj(*image); + if (!image) + return -ENOMEM; + + for (i = 0; i < ARRAY_SIZE(fw_images_uuids); i++) { + if (!strcmp(fw_images_uuids[i].uuid, fw_uuid)) + image_name = fw_images_uuids[i].name; + } + + image->kobj.kset = lfa_kset; + image->image_name = image_name; + image->cpu_rendezvous_forced = true; + set_image_flags(image, seq_id, image_flags, reg_current_ver, + reg_pending_ver); + if (kobject_init_and_add(&image->kobj, &image_ktype, NULL, + "%s", fw_uuid)) { + kobject_put(&image->kobj); + + return -ENOMEM; + } + + return 0; +} + +static int update_fw_images_tree(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }, res; + struct uuid_regs image_uuid; + char image_id_str[40]; + int ret, num_of_components; + + num_of_components = get_nr_lfa_components(); + if (num_of_components <= 0) { + pr_err("Error getting number of LFA components\n"); + return -ENODEV; + } + + reg.a0 = LFA_1_0_FN_GET_INVENTORY; + for (int i = 0; i < num_of_components; i++) { + reg.a1 = i; /* fw_seq_id to be queried */ + arm_smccc_1_2_invoke(®, &res); + if (res.a0 == LFA_SUCCESS) { + image_uuid.uuid_lo = res.a1; + image_uuid.uuid_hi = res.a2; + + snprintf(image_id_str, sizeof(image_id_str), "%pUb", + &image_uuid); + ret = update_fw_image_node(image_id_str, i, res.a3, + res.a4, res.a5); + if (ret) + return ret; + } + } + + return 0; +} + +static int __init lfa_init(void) +{ + struct arm_smccc_1_2_regs reg = { 0 }; + int err; + + reg.a0 = LFA_1_0_FN_GET_VERSION; + arm_smccc_1_2_invoke(®, ®); + if (reg.a0 == -LFA_NOT_SUPPORTED) { + pr_info("Live Firmware activation: no firmware agent found\n"); + return -ENODEV; + } + + pr_info("Live Firmware Activation: detected v%ld.%ld\n", + reg.a0 >> 16, reg.a0 & 0xffff); + + fw_images_update_wq = alloc_workqueue("fw_images_update_wq", + WQ_UNBOUND | WQ_MEM_RECLAIM, 1); + if (!fw_images_update_wq) { + pr_err("Live Firmware Activation: Failed to allocate workqueue.\n"); + + return -ENOMEM; + } + INIT_WORK(&fw_images_update_work, remove_invalid_fw_images); + + init_image_default_attrs(); + lfa_kset = kset_create_and_add("lfa", NULL, firmware_kobj); + if (!lfa_kset) + return -ENOMEM; + + err = update_fw_images_tree(); + if (err != 0) { + kset_unregister(lfa_kset); + destroy_workqueue(fw_images_update_wq); + } + + return err; +} +module_init(lfa_init); + +static void __exit lfa_exit(void) +{ + flush_workqueue(fw_images_update_wq); + destroy_workqueue(fw_images_update_wq); + clean_fw_images_tree(); + kset_unregister(lfa_kset); +} +module_exit(lfa_exit); + +MODULE_DESCRIPTION("ARM Live Firmware Activation (LFA)"); +MODULE_LICENSE("GPL"); From a60824f146d0d9f2ca38b9f692a5106a32480ff2 Mon Sep 17 00:00:00 2001 From: Vedashree Vidwans Date: Tue, 17 Mar 2026 11:33:29 +0100 Subject: [PATCH 173/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Move image rescanning BugLink: https://bugs.launchpad.net/bugs/2150652 After an image activation, the list of firmware images might change, so we have to re-iterate them through the SMC interface. Move the corresponding code from the activate_fw_image() function into update_fw_images_tree(), where it could be reused more easily, for instance when triggered by an interrupt. Signed-off-by: Vedashree Vidwans [Andre: split off from another patch, rebased] Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 46 ++++++++++++++++----------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 284b7c18d3d09..4831abf2b60ee 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -310,7 +310,6 @@ static int call_lfa_activate(void *data) static int activate_fw_image(struct fw_image *image) { - struct kobject *kobj; int ret; retry: @@ -320,31 +319,8 @@ static int activate_fw_image(struct fw_image *image) ret = call_lfa_activate(image); if (!ret) { - /* - * Invalidate fw_seq_ids (-1) for all images as the seq_ids - * and the number of firmware images in the LFA agent may - * change after a successful activation attempt. - * Negate all image flags as well. - */ - spin_lock(&lfa_kset->list_lock); - list_for_each_entry(kobj, &lfa_kset->list, entry) { - struct fw_image *image = kobj_to_fw_image(kobj); - - set_image_flags(image, -1, 0b1000, 0, 0); - } - spin_unlock(&lfa_kset->list_lock); - update_fw_images_tree(); - /* - * Removing non-valid image directories at the end of an - * activation. - * We can't remove the sysfs attributes while in the respective - * _store() handler, so have to postpone the list removal to a - * workqueue. - */ - queue_work(fw_images_update_wq, &fw_images_update_work); - return 0; } @@ -640,6 +616,7 @@ static int update_fw_images_tree(void) { struct arm_smccc_1_2_regs reg = { 0 }, res; struct uuid_regs image_uuid; + struct kobject *kobj; char image_id_str[40]; int ret, num_of_components; @@ -649,6 +626,19 @@ static int update_fw_images_tree(void) return -ENODEV; } + /* + * Invalidate fw_seq_ids (-1) for all images as the seq_ids and the + * number of firmware images in the LFA agent may change after a + * successful activation attempt. Negate all image flags as well. + */ + spin_lock(&lfa_kset->list_lock); + list_for_each_entry(kobj, &lfa_kset->list, entry) { + struct fw_image *image = kobj_to_fw_image(kobj); + + set_image_flags(image, -1, 0b1000, 0, 0); + } + spin_unlock(&lfa_kset->list_lock); + reg.a0 = LFA_1_0_FN_GET_INVENTORY; for (int i = 0; i < num_of_components; i++) { reg.a1 = i; /* fw_seq_id to be queried */ @@ -666,6 +656,14 @@ static int update_fw_images_tree(void) } } + /* + * Removing non-valid image directories at the end of an activation. + * We can't remove the sysfs attributes while in the respective + * _store() handler, so have to postpone the list removal to a + * workqueue. + */ + queue_work(fw_images_update_wq, &fw_images_update_work); + return 0; } From 9b8e3d7430699a3ad1ccc3421f9b119fc5d1aacd Mon Sep 17 00:00:00 2001 From: Vedashree Vidwans Date: Tue, 17 Mar 2026 11:33:30 +0100 Subject: [PATCH 174/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Add timeout and trigger watchdog BugLink: https://bugs.launchpad.net/bugs/2150652 Enhance PRIME/ACTIVATION functions to touch watchdog and implement timeout mechanism. This update ensures that any potential hangs are detected promptly and that the LFA process is allocated sufficient execution time before the watchdog timer expires. These changes improve overall system reliability by reducing the risk of undetected process stalls and unexpected watchdog resets. Signed-off-by: Vedashree Vidwans Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 43 ++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 4831abf2b60ee..d1b5cd29b8a09 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -5,11 +5,14 @@ #include #include +#include #include #include #include +#include #include #include +#include #include #include #include @@ -38,6 +41,11 @@ #define LFA_PRIME_CALL_AGAIN BIT(0) #define LFA_ACTIVATE_CALL_AGAIN BIT(0) +#define LFA_PRIME_BUDGET_MS 30000 /* 30s cap */ +#define LFA_PRIME_DELAY_MS 10 /* 10ms between polls */ +#define LFA_ACTIVATE_BUDGET_MS 10000 /* 10s cap */ +#define LFA_ACTIVATE_DELAY_MS 10 /* 10ms between polls */ + /* LFA return values */ #define LFA_SUCCESS 0 #define LFA_NOT_SUPPORTED 1 @@ -287,6 +295,7 @@ static int call_lfa_activate(void *data) struct fw_image *image = data; struct arm_smccc_1_2_regs reg = { 0 }, res; + touch_nmi_watchdog(); reg.a0 = LFA_1_0_FN_ACTIVATE; reg.a1 = image->fw_seq_id; /* @@ -310,6 +319,7 @@ static int call_lfa_activate(void *data) static int activate_fw_image(struct fw_image *image) { + ktime_t end = ktime_add_ms(ktime_get(), LFA_ACTIVATE_BUDGET_MS); int ret; retry: @@ -324,8 +334,15 @@ static int activate_fw_image(struct fw_image *image) return 0; } - if (ret == -LFA_CALL_AGAIN) - goto retry; + if (ret == -LFA_CALL_AGAIN) { + /* SMC returned with call_again flag set */ + if (ktime_before(ktime_get(), end)) { + msleep_interruptible(LFA_ACTIVATE_DELAY_MS); + goto retry; + } + + ret = -LFA_TIMED_OUT; + } lfa_cancel(image); @@ -338,6 +355,8 @@ static int activate_fw_image(struct fw_image *image) static int prime_fw_image(struct fw_image *image) { struct arm_smccc_1_2_regs reg = { 0 }, res; + ktime_t end = ktime_add_ms(ktime_get(), LFA_PRIME_BUDGET_MS); + int ret; if (image->may_reset_cpu) { pr_err("CPU reset not supported by kernel driver\n"); @@ -345,6 +364,8 @@ static int prime_fw_image(struct fw_image *image) return -EINVAL; } + touch_nmi_watchdog(); + reg.a0 = LFA_1_0_FN_PRIME; retry: /* @@ -363,8 +384,22 @@ static int prime_fw_image(struct fw_image *image) return res.a0; } - if (res.a1 & LFA_PRIME_CALL_AGAIN) - goto retry; + if (res.a1 & LFA_PRIME_CALL_AGAIN) { + /* SMC returned with call_again flag set */ + if (ktime_before(ktime_get(), end)) { + msleep_interruptible(LFA_PRIME_DELAY_MS); + goto retry; + } + + pr_err("LFA_PRIME for image %s timed out", + get_image_name(image)); + + ret = lfa_cancel(image); + if (ret != 0) + return ret; + + return -ETIMEDOUT; + } return 0; } From e466cf8f8a6aa4646b4fe239dff29864be7d2de0 Mon Sep 17 00:00:00 2001 From: Vedashree Vidwans Date: Tue, 17 Mar 2026 11:33:31 +0100 Subject: [PATCH 175/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Register ACPI notification BugLink: https://bugs.launchpad.net/bugs/2150652 The Arm LFA spec describes an ACPI notification mechanism, where the platform (firmware) can notify an LFA client about newly available firmware imag updates ("pending images" in LFA terms). Add a faux device after discovering the existence of an LFA agent via the SMCCC discovery mechnism, and use that device to check for the ACPI notification description. Register this when one is provided. The notification just conveys the fact that at least one firmware image has now a pending update, it doesn't say which, also there could be more than one pending. Loop through all images to find every which needs to be activated, and trigger the activation. We need to do this is a loop, since an activation might change the number and the status of available images. Signed-off-by: Vedashree Vidwans [Andre: convert from platform driver to faux device] Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 147 ++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index d1b5cd29b8a09..f20ea45cdbd93 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -3,11 +3,14 @@ * Copyright (C) 2025 Arm Limited */ +#include #include #include #include +#include #include #include +#include #include #include #include @@ -17,11 +20,13 @@ #include #include #include +#include #include #include #include +#define DRIVER_NAME "ARM_LFA" #undef pr_fmt #define pr_fmt(fmt) "Arm LFA: " fmt @@ -702,6 +707,139 @@ static int update_fw_images_tree(void) return 0; } +/* + * Go through all FW images in a loop and trigger activation + * of all activatible and pending images. + * We have to restart enumeration after every triggered activation, + * since the firmware images might have changed during the activation. + */ +static int activate_pending_image(void) +{ + struct kobject *kobj; + bool found_pending = false; + struct fw_image *image; + int ret; + + spin_lock(&lfa_kset->list_lock); + list_for_each_entry(kobj, &lfa_kset->list, entry) { + image = kobj_to_fw_image(kobj); + + if (image->fw_seq_id == -1) + continue; /* Invalid FW component */ + + update_fw_image_pending(image); + if (image->activation_capable && image->activation_pending) { + found_pending = true; + break; + } + } + spin_unlock(&lfa_kset->list_lock); + + if (!found_pending) + return -ENOENT; + + ret = prime_fw_image(image); + if (ret) + return ret; + + ret = activate_fw_image(image); + if (ret) + return ret; + + pr_info("%s: automatic activation succeeded\n", get_image_name(image)); + + return 0; +} + +#ifdef CONFIG_ACPI +static void lfa_acpi_notify_handler(acpi_handle handle, u32 event, void *data) +{ + int ret; + + while (!(ret = activate_pending_image())) + ; + + if (ret != -ENOENT) + pr_warn("notified image activation failed: %d\n", ret); +} + +static int lfa_register_acpi(struct device *dev) +{ + struct acpi_device *acpi_dev; + acpi_handle handle; + acpi_status status; + + acpi_dev = acpi_dev_get_first_match_dev("ARML0003", NULL, -1); + if (!acpi_dev) + return -ENODEV; + handle = acpi_device_handle(acpi_dev); + if (!handle) { + acpi_dev_put(acpi_dev); + return -ENODEV; + } + + /* Register notify handler that indicates LFA updates are available */ + status = acpi_install_notify_handler(handle, ACPI_DEVICE_NOTIFY, + lfa_acpi_notify_handler, NULL); + if (ACPI_FAILURE(status)) { + acpi_dev_put(acpi_dev); + return -EIO; + } + + ACPI_COMPANION_SET(dev, acpi_dev); + + return 0; +} + +static void lfa_remove_acpi(struct device *dev) +{ + struct acpi_device *acpi_dev = ACPI_COMPANION(dev); + acpi_handle handle = acpi_device_handle(acpi_dev); + + if (handle) + acpi_remove_notify_handler(handle, + ACPI_DEVICE_NOTIFY, + lfa_acpi_notify_handler); + acpi_dev_put(acpi_dev); +} +#else /* !CONFIG_ACPI */ +static int lfa_register_acpi(struct device *dev) +{ + return -ENODEV; +} + +static void lfa_remove_acpi(struct device *dev) +{ +} +#endif + +static int lfa_faux_probe(struct faux_device *fdev) +{ + int ret; + + if (!acpi_disabled) { + ret = lfa_register_acpi(&fdev->dev); + if (ret != -ENODEV) { + if (!ret) + pr_info("registered LFA ACPI notification\n"); + return ret; + } + } + + return 0; +} + +static void lfa_faux_remove(struct faux_device *fdev) +{ + lfa_remove_acpi(&fdev->dev); +} + +static struct faux_device *lfa_dev; +static struct faux_device_ops lfa_device_ops = { + .probe = lfa_faux_probe, + .remove = lfa_faux_remove, +}; + static int __init lfa_init(void) { struct arm_smccc_1_2_regs reg = { 0 }; @@ -731,6 +869,14 @@ static int __init lfa_init(void) if (!lfa_kset) return -ENOMEM; + /* + * This faux device is just used for the optional notification + * mechanism, to register the ACPI notification or interrupt. + * If the firmware tables do not contain this information, the + * driver will still work. + */ + lfa_dev = faux_device_create("arm-lfa", NULL, &lfa_device_ops); + err = update_fw_images_tree(); if (err != 0) { kset_unregister(lfa_kset); @@ -747,6 +893,7 @@ static void __exit lfa_exit(void) destroy_workqueue(fw_images_update_wq); clean_fw_images_tree(); kset_unregister(lfa_kset); + faux_device_destroy(lfa_dev); } module_exit(lfa_exit); From a1e85751fae37f524407db561c29403cbac484ff Mon Sep 17 00:00:00 2001 From: Andre Przywara Date: Tue, 17 Mar 2026 11:33:32 +0100 Subject: [PATCH 176/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Add auto_activate sysfs file BugLink: https://bugs.launchpad.net/bugs/2150652 The Arm LFA spec places control over the actual activation process in the hands of the non-secure host OS. An platform initiated interrupt or notification signals the availability of an updateable firmware image, but does not necessarily need to trigger it automatically. Add a sysfs control file that guards such automatic activation. If an administrator wants to allow automatic platform initiated updates, they can activate that by echoing a "1" into the auto_activate file in the respective sysfs directory. Any incoming notification would then result in the activation triggered. Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 34 ++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index f20ea45cdbd93..5dc531e462ebc 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -101,6 +101,7 @@ enum image_attr_names { LFA_ATTR_FORCE_CPU_RENDEZVOUS, LFA_ATTR_ACTIVATE, LFA_ATTR_CANCEL, + LFA_ATTR_AUTO_ACTIVATE, LFA_ATTR_NR_IMAGES }; @@ -115,6 +116,7 @@ struct fw_image { bool may_reset_cpu; bool cpu_rendezvous; bool cpu_rendezvous_forced; + bool auto_activate; struct kobj_attribute image_attrs[LFA_ATTR_NR_IMAGES]; }; @@ -561,6 +563,28 @@ static ssize_t cancel_store(struct kobject *kobj, struct kobj_attribute *attr, return count; } +static ssize_t auto_activate_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + int ret; + + ret = kstrtobool(buf, &image->auto_activate); + if (ret) + return ret; + + return count; +} + +static ssize_t auto_activate_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct fw_image *image = kobj_to_fw_image(kobj); + + return sysfs_emit(buf, "%d\n", image->auto_activate); +} + static struct kobj_attribute image_attrs_group[LFA_ATTR_NR_IMAGES] = { [LFA_ATTR_NAME] = __ATTR_RO(name), [LFA_ATTR_CURRENT_VERSION] = __ATTR_RO(current_version), @@ -571,7 +595,8 @@ static struct kobj_attribute image_attrs_group[LFA_ATTR_NR_IMAGES] = { [LFA_ATTR_CPU_RENDEZVOUS] = __ATTR_RO(cpu_rendezvous), [LFA_ATTR_FORCE_CPU_RENDEZVOUS] = __ATTR_RW(force_cpu_rendezvous), [LFA_ATTR_ACTIVATE] = __ATTR_WO(activate), - [LFA_ATTR_CANCEL] = __ATTR_WO(cancel) + [LFA_ATTR_CANCEL] = __ATTR_WO(cancel), + [LFA_ATTR_AUTO_ACTIVATE] = __ATTR_RW(auto_activate), }; static void init_image_default_attrs(void) @@ -640,6 +665,7 @@ static int update_fw_image_node(char *fw_uuid, int seq_id, image->kobj.kset = lfa_kset; image->image_name = image_name; image->cpu_rendezvous_forced = true; + image->auto_activate = false; set_image_flags(image, seq_id, image_flags, reg_current_ver, reg_pending_ver); if (kobject_init_and_add(&image->kobj, &image_ktype, NULL, @@ -709,7 +735,8 @@ static int update_fw_images_tree(void) /* * Go through all FW images in a loop and trigger activation - * of all activatible and pending images. + * of all activatible and pending images, but only if automatic + * activation for that image is allowed. * We have to restart enumeration after every triggered activation, * since the firmware images might have changed during the activation. */ @@ -728,7 +755,8 @@ static int activate_pending_image(void) continue; /* Invalid FW component */ update_fw_image_pending(image); - if (image->activation_capable && image->activation_pending) { + if (image->activation_capable && image->activation_pending && + image->auto_activate) { found_pending = true; break; } From e937978de59d745b2bccf12ee3db9ad33cc8866f Mon Sep 17 00:00:00 2001 From: Andre Przywara Date: Tue, 17 Mar 2026 11:33:33 +0100 Subject: [PATCH 177/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Register DT interrupt BugLink: https://bugs.launchpad.net/bugs/2150652 The Arm Live Firmware Activation spec describes an asynchronous notification mechanism, where the platform can notify the host OS about newly pending image updates. In the absence of the ACPI notification mechanism also a simple devicetree node can describe an interrupt. Add code to find the respective DT node and register the specified interrupt, to trigger the activation if needed. Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 45 +++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 5dc531e462ebc..ecd056901b8d8 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include #include #include @@ -841,6 +843,43 @@ static void lfa_remove_acpi(struct device *dev) } #endif +static irqreturn_t lfa_irq_handler(int irq, void *dev_id) +{ + return IRQ_WAKE_THREAD; +} + +static irqreturn_t lfa_irq_handler_thread(int irq, void *dev_id) +{ + int ret; + + while (!(ret = activate_pending_image())) + ; + + if (ret != -ENOENT) + pr_warn("notified image activation failed: %d\n", ret); + + return IRQ_HANDLED; +} + +static int lfa_register_dt(struct device *dev) +{ + struct device_node *np; + unsigned int irq; + + np = of_find_compatible_node(NULL, NULL, "arm,lfa"); + if (!np) + return -ENODEV; + + irq = irq_of_parse_and_map(np, 0); + of_node_put(np); + if (!irq) + return -ENODEV; + + return devm_request_threaded_irq(dev, irq, lfa_irq_handler, + lfa_irq_handler_thread, + IRQF_COND_ONESHOT, NULL, NULL); +} + static int lfa_faux_probe(struct faux_device *fdev) { int ret; @@ -854,6 +893,12 @@ static int lfa_faux_probe(struct faux_device *fdev) } } + ret = lfa_register_dt(&fdev->dev); + if (!ret) + pr_info("registered LFA DT notification interrupt\n"); + if (ret != -ENODEV) + return ret; + return 0; } From 4b7831f50242685ba534c6a058579668b4361b4e Mon Sep 17 00:00:00 2001 From: Andre Przywara Date: Tue, 17 Mar 2026 11:33:34 +0100 Subject: [PATCH 178/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: introduce SMC access lock BugLink: https://bugs.launchpad.net/bugs/2150652 After a successful live activation, the list of firmware images might change, which also affects the sequence IDs. We store the sequence ID in a data structure and connect it to its GUID, which is the identifier used to access certain image properties from userland. When an activation is happening, the sequence ID associations might change at any point, so we must be sure to not use any previously learned sequence ID during this time. Protect the association between a sequence ID and a firmware image (its GUID, really) by a reader/writer lock. In this case it's a R/W semaphore, so it can sleep and we can hold it for longer, also concurrent SMC calls are not blocked on each other, it's just an activation that blocks calls. Signed-off-by: Andre Przywara (backported from https://lore.kernel.org/all/20260317103336.1273582-1-andre.przywara@arm.com/) Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 38 +++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index ecd056901b8d8..663ba79f07135 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,16 @@ static struct workqueue_struct *fw_images_update_wq; static struct work_struct fw_images_update_work; static struct attribute *image_default_attrs[LFA_ATTR_NR_IMAGES + 1]; +/* + * A successful image activation might change the number of available images, + * leading to a re-order and thus re-assignment of the sequence IDs. + * The lock protects the connection between a firmware image (through its + * user visible UUID) and the sequence IDs. Anyone doing an SMC call with + * a sequence ID needs to take the readers lock. Doing an activation requires + * the writer lock, as that process might change the assocications. + */ +struct rw_semaphore smc_lock; + static const struct attribute_group image_attr_group = { .attrs = image_default_attrs, }; @@ -253,6 +264,7 @@ static unsigned long get_nr_lfa_components(void) reg.a0 = LFA_1_0_FN_GET_INFO; reg.a1 = 0; /* lfa_info_selector = 0 */ + /* No need for the smc_lock, since no sequence IDs are involved. */ arm_smccc_1_2_invoke(®, ®); if (reg.a0 != LFA_SUCCESS) return reg.a0; @@ -265,9 +277,11 @@ static int lfa_cancel(void *data) struct fw_image *image = data; struct arm_smccc_1_2_regs reg = { 0 }; + down_read(&smc_lock); reg.a0 = LFA_1_0_FN_CANCEL; reg.a1 = image->fw_seq_id; arm_smccc_1_2_invoke(®, ®); + up_read(&smc_lock); /* * When firmware activation is called with "skip_cpu_rendezvous=1", @@ -332,6 +346,7 @@ static int activate_fw_image(struct fw_image *image) int ret; retry: + down_write(&smc_lock); if (image->cpu_rendezvous_forced || image->cpu_rendezvous) ret = stop_machine(call_lfa_activate, image, cpu_online_mask); else @@ -339,10 +354,13 @@ static int activate_fw_image(struct fw_image *image) if (!ret) { update_fw_images_tree(); + up_write(&smc_lock); return 0; } + up_write(&smc_lock); + if (ret == -LFA_CALL_AGAIN) { /* SMC returned with call_again flag set */ if (ktime_before(ktime_get(), end)) { @@ -383,8 +401,11 @@ static int prime_fw_image(struct fw_image *image) * be called again. * reg.a1 will become 0 once the prime process completes. */ + down_read(&smc_lock); reg.a1 = image->fw_seq_id; arm_smccc_1_2_invoke(®, &res); + up_read(&smc_lock); + if ((long)res.a0 < 0) { pr_err("LFA_PRIME for image %s failed: %s\n", get_image_name(image), @@ -429,7 +450,7 @@ static ssize_t activation_capable_show(struct kobject *kobj, return sysfs_emit(buf, "%d\n", image->activation_capable); } -static void update_fw_image_pending(struct fw_image *image) +static void _update_fw_image_pending(struct fw_image *image) { struct arm_smccc_1_2_regs reg = { 0 }; @@ -441,6 +462,13 @@ static void update_fw_image_pending(struct fw_image *image) image->activation_pending = !!(reg.a3 & BIT(1)); } +static void update_fw_image_pending(struct fw_image *image) +{ + down_read(&smc_lock); + _update_fw_image_pending(image); + up_read(&smc_lock); +} + static ssize_t activation_pending_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf) { @@ -515,9 +543,11 @@ static ssize_t pending_version_show(struct kobject *kobj, * Similar to activation pending, this value can change following an * update, we need to retrieve fresh info instead of stale information. */ + down_read(&smc_lock); reg.a0 = LFA_1_0_FN_GET_INVENTORY; reg.a1 = image->fw_seq_id; arm_smccc_1_2_invoke(®, ®); + up_read(&smc_lock); if (reg.a0 == LFA_SUCCESS) { if (reg.a5 != 0 && image->activation_pending) { u32 maj, min; @@ -749,6 +779,7 @@ static int activate_pending_image(void) struct fw_image *image; int ret; + down_read(&smc_lock); spin_lock(&lfa_kset->list_lock); list_for_each_entry(kobj, &lfa_kset->list, entry) { image = kobj_to_fw_image(kobj); @@ -756,7 +787,7 @@ static int activate_pending_image(void) if (image->fw_seq_id == -1) continue; /* Invalid FW component */ - update_fw_image_pending(image); + _update_fw_image_pending(image); if (image->activation_capable && image->activation_pending && image->auto_activate) { found_pending = true; @@ -764,6 +795,7 @@ static int activate_pending_image(void) } } spin_unlock(&lfa_kset->list_lock); + up_read(&smc_lock); if (!found_pending) return -ENOENT; @@ -950,6 +982,8 @@ static int __init lfa_init(void) */ lfa_dev = faux_device_create("arm-lfa", NULL, &lfa_device_ops); + init_rwsem(&smc_lock); + err = update_fw_images_tree(); if (err != 0) { kset_unregister(lfa_kset); From e06e9ca63e356f641d346dfdbfb8d232b35e59a8 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 23 Apr 2026 04:39:29 -0700 Subject: [PATCH 179/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: handle LFA_BUSY in PRIME and ACTIVATE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BugLink: https://bugs.launchpad.net/bugs/2150652 DEN0147 §2.6: LFA_ACTIVATE can return LFA_BUSY when the firmware postpones the activation. Although the rwsem in this driver prevents concurrent ACTIVATE calls from kernel space, an external agent or internal firmware state may still produce LFA_BUSY. Add an explicit retry loop (same budget and delay as CALL_AGAIN) so the code does not silently treat a retriable condition as a terminal failure. Catching LFA_BUSY explicitly also surfaces potential firmware or driver bugs. DEN0147 §2.5: LFA_PRIME returning LFA_BUSY means another CPU is running LFA_PRIME concurrently. This driver never issues parallel PRIME, so this is unexpected; log pr_warn and return so the caller can surface the anomaly rather than swallowing it in the generic error path. Signed-off-by: Vedashree Vidwans Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 663ba79f07135..3c6a59c74bc96 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -371,6 +371,23 @@ static int activate_fw_image(struct fw_image *image) ret = -LFA_TIMED_OUT; } + /* + * DEN0147 §2.6: LFA_BUSY means activation was postponed by firmware + * and must be retried. Although the rwsem prevents concurrent ACTIVATE + * from this driver, an external agent or firmware-internal state may + * still return LFA_BUSY. Handle it explicitly so it is not silently + * treated as a terminal failure — catching it here may also indicate a + * bug in the driver or firmware. + */ + if (ret == -LFA_BUSY) { + if (ktime_before(ktime_get(), end)) { + msleep_interruptible(LFA_ACTIVATE_DELAY_MS); + goto retry; + } + + ret = -LFA_TIMED_OUT; + } + lfa_cancel(image); pr_err("LFA_ACTIVATE for image %s failed: %s\n", @@ -406,6 +423,17 @@ static int prime_fw_image(struct fw_image *image) arm_smccc_1_2_invoke(®, &res); up_read(&smc_lock); + /* + * DEN0147 §2.5: LFA_BUSY from PRIME means another CPU is concurrently + * running LFA_PRIME. This driver never issues parallel PRIME, so this + * is unexpected and likely indicates a firmware or driver bug. + */ + if ((long)res.a0 == -LFA_BUSY) { + pr_warn("LFA_PRIME for image %s returned LFA_BUSY (concurrent PRIME unexpected; possible firmware or driver bug)\n", + get_image_name(image)); + return res.a0; + } + if ((long)res.a0 < 0) { pr_err("LFA_PRIME for image %s failed: %s\n", get_image_name(image), From 1e0d035086aa001096f64a88da167c0f9924c4fb Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Mon, 4 May 2026 04:26:06 -0700 Subject: [PATCH 180/464] NVIDIA: VR: SAUCE: firmware: smccc: lfa: Emit a uevent on inventory updates BugLink: https://bugs.launchpad.net/bugs/2150652 Firmware image directories are plain kobjects under /sys/firmware. udev coldplug does not enumerate them as devices, so rules matching the per-image LFA kobjects do not run reliably at boot. LFA already creates the arm-lfa faux device. Emit KOBJ_CHANGE from that device after the firmware image tree is refreshed, so user space can use the existing driver-core device as the notification anchor for runtime inventory updates. The same udev rule then also covers coldplug via the device add event, e.g.: ACTION=="add|change", SUBSYSTEM=="faux", KERNEL=="arm-lfa", \ RUN+="/usr/local/sbin/lfa-auto-activate" Signed-off-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/smccc/lfa_fw.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/smccc/lfa_fw.c b/drivers/firmware/smccc/lfa_fw.c index 3c6a59c74bc96..e8e7073af91b2 100644 --- a/drivers/firmware/smccc/lfa_fw.c +++ b/drivers/firmware/smccc/lfa_fw.c @@ -154,6 +154,7 @@ static const struct fw_image_uuid { }; static struct kset *lfa_kset; +static struct faux_device *lfa_dev; static struct workqueue_struct *fw_images_update_wq; static struct work_struct fw_images_update_work; static struct attribute *image_default_attrs[LFA_ATTR_NR_IMAGES + 1]; @@ -241,6 +242,21 @@ static void remove_invalid_fw_images(struct work_struct *work) delete_fw_image_node(image); } + + /* + * Notify user space only after the firmware image tree has been + * fully reconciled, so that consumers reading /sys/firmware/lfa/ + * see a settled inventory. Anchor the event on the arm-lfa faux + * device because the per-image kobjects under /sys/firmware/ are + * not enumerated by udev coldplug. + */ + if (lfa_dev) { + int ret = kobject_uevent(&lfa_dev->dev.kobj, KOBJ_CHANGE); + + if (ret) + pr_warn("failed to send firmware inventory change uevent: %d\n", + ret); + } } static void set_image_flags(struct fw_image *image, int seq_id, @@ -967,7 +983,6 @@ static void lfa_faux_remove(struct faux_device *fdev) lfa_remove_acpi(&fdev->dev); } -static struct faux_device *lfa_dev; static struct faux_device_ops lfa_device_ops = { .probe = lfa_faux_probe, .remove = lfa_faux_remove, From 4a2d4acede3995cc85391341355db77884644050 Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Mon, 20 Apr 2026 09:24:01 +0200 Subject: [PATCH 181/464] NVIDIA: VR: SAUCE: sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity BugLink: https://bugs.launchpad.net/bugs/2150671 On asymmetric CPU capacity systems, the wakeup path uses select_idle_capacity(), which scans the span of sd_asym_cpucapacity rather than sd_llc. The has_idle_cores hint however lives on sd_llc->shared, so the wakeup-time read of has_idle_cores operates on an LLC-scoped blob while the actual scan/decision spans the wider asym domain; nr_busy_cpus also lives in the same shared sched_domain data, but it's never used in the asym CPU capacity scenario. Therefore, move the sched_domain_shared object to sd_asym_cpucapacity whenever the CPU has a SD_ASYM_CPUCAPACITY_FULL ancestor and that ancestor is non-overlapping (i.e., not built from SD_NUMA). In that case the scope of has_idle_cores matches the scope of the wakeup scan. Fall back to attaching the shared object to sd_llc in three cases: 1) plain symmetric systems (no SD_ASYM_CPUCAPACITY_FULL anywhere); 2) CPUs in an exclusive cpuset that carves out a symmetric capacity island: has_asym is system-wide but those CPUs have no SD_ASYM_CPUCAPACITY_FULL ancestor in their hierarchy and follow the symmetric LLC path in select_idle_sibling(); 3) exotic topologies where SD_ASYM_CPUCAPACITY_FULL lands on an SD_NUMA-built domain. init_sched_domain_shared() keys the shared blob off cpumask_first(span), which on overlapping NUMA domains would alias unrelated spans onto the same blob. Keep the shared object on the LLC there; select_idle_capacity() gracefully skips the has_idle_cores preference when sd->shared is NULL. While at it, also rename the per-CPU sd_llc_shared to sd_balance_shared, as it is no longer strictly tied to the LLC. Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak (backported from https://lore.kernel.org/all/20260428051720.3180182-1-arighi@nvidia.com) [ arighi: - backport full logic to attach sd->shared in build_sched_domains() - do not rename sd_llc_shared to reduce the risk of conflicts ] Signed-off-by: Andrea Righi Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- kernel/sched/fair.c | 6 ++- kernel/sched/topology.c | 101 +++++++++++++++++++++++++++++++++------- 2 files changed, 89 insertions(+), 18 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3bce48ad0bc5a..70b402765efb5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12619,7 +12619,8 @@ static void set_cpu_sd_state_busy(int cpu) goto unlock; sd->nohz_idle = 0; - atomic_inc(&sd->shared->nr_busy_cpus); + if (sd->shared) + atomic_inc(&sd->shared->nr_busy_cpus); unlock: rcu_read_unlock(); } @@ -12648,7 +12649,8 @@ static void set_cpu_sd_state_idle(int cpu) goto unlock; sd->nohz_idle = 1; - atomic_dec(&sd->shared->nr_busy_cpus); + if (sd->shared) + atomic_dec(&sd->shared->nr_busy_cpus); unlock: rcu_read_unlock(); } diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 2864f43bff6df..b5cc76ccb4e14 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -680,16 +680,38 @@ static void update_top_cache_domain(int cpu) int id = cpu; int size = 1; + sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); + /* + * The shared object is attached to sd_asym_cpucapacity only when the + * asym domain is non-overlapping (i.e., not built from SD_NUMA). + * On overlapping (NUMA) asym domains we fall back to letting the + * SD_SHARE_LLC path own the shared object, so sd->shared may be NULL + * here. + */ + if (sd && sd->shared) + sds = sd->shared; + + rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + sd = highest_flag_domain(cpu, SD_SHARE_LLC); if (sd) { id = cpumask_first(sched_domain_span(sd)); size = cpumask_weight(sched_domain_span(sd)); - sds = sd->shared; + + /* + * If sd_asym_cpucapacity didn't claim the shared object, + * sd_llc must have one linked. + */ + if (!sds) { + WARN_ON_ONCE(!sd->shared); + sds = sd->shared; + } } rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; + rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); sd = lowest_flag_domain(cpu, SD_CLUSTER); @@ -708,9 +730,6 @@ static void update_top_cache_domain(int cpu) sd = highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd); - - sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); - rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } /* @@ -1640,13 +1659,12 @@ sd_init(struct sched_domain_topology_level *tl, { struct sd_data *sdd = &tl->data; struct sched_domain *sd = *per_cpu_ptr(sdd->sd, cpu); - int sd_id, sd_weight, sd_flags = 0; + int sd_weight, sd_flags = 0; struct cpumask *sd_span; sd_span = sched_domain_span(sd); cpumask_and(sd_span, cpu_map, tl->mask(tl, cpu)); sd_weight = cpumask_weight(sd_span); - sd_id = cpumask_first(sd_span); if (tl->sd_flags) sd_flags = (*tl->sd_flags)(); @@ -1725,16 +1743,6 @@ sd_init(struct sched_domain_topology_level *tl, sd->cache_nice_tries = 1; } - /* - * For all levels sharing cache; connect a sched_domain_shared - * instance. - */ - if (sd->flags & SD_SHARE_LLC) { - sd->shared = *per_cpu_ptr(sdd->sds, sd_id); - atomic_inc(&sd->shared->ref); - atomic_set(&sd->shared->nr_busy_cpus, sd_weight); - } - sd->private = sdd; return sd; @@ -2546,6 +2554,16 @@ static bool topology_span_sane(const struct cpumask *cpu_map) return true; } +static void init_sched_domain_shared(struct sched_domain *sd) +{ + struct sd_data *sdd = sd->private; + int sd_id = cpumask_first(sched_domain_span(sd)); + + sd->shared = *per_cpu_ptr(sdd->sds, sd_id); + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2603,6 +2621,57 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att } } + for_each_cpu(i, cpu_map) { + struct sched_domain *sd_asym = NULL; + bool asym_claimed = false; + + sd = *per_cpu_ptr(d.sd, i); + if (!sd) + continue; + + /* + * In case of ASYM_CPUCAPACITY, attach sd->shared to + * sd_asym_cpucapacity for wakeup stat tracking. + * + * Caveats: + * + * 1) has_asym is system-wide, but a given CPU may still + * lack an SD_ASYM_CPUCAPACITY_FULL ancestor (e.g., an + * exclusive cpuset carving out a symmetric capacity island). + * Such CPUs must fall through to the LLC seeding path below. + * + * 2) Skip the asym attach if the asym ancestor is an + * overlapping domain (SD_NUMA). On those topologies let the + * LLC path own the shared object instead. + * + * XXX: This assumes SD_ASYM_CPUCAPACITY_FULL domain + * always has more than one group else it is prone to + * degeneration. + */ + sd_asym = sd; + while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL)) + sd_asym = sd_asym->parent; + + if (sd_asym && !(sd_asym->flags & SD_NUMA)) { + init_sched_domain_shared(sd_asym); + asym_claimed = true; + } + + /* First, find the topmost SD_SHARE_LLC domain */ + sd = *per_cpu_ptr(d.sd, i); + while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) + sd = sd->parent; + + if (sd->flags & SD_SHARE_LLC) { + /* + * Initialize the sd->shared for SD_SHARE_LLC unless + * the asym path above already claimed it. + */ + if (!asym_claimed) + init_sched_domain_shared(sd); + } + } + /* * Calculate an allowed NUMA imbalance such that LLCs do not get * imbalanced. From b76f5eb20edcb7a2f9339cdf11b91ec9b150bc15 Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Wed, 11 Mar 2026 18:43:19 +0100 Subject: [PATCH 182/464] NVIDIA: VR: SAUCE: sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection BugLink: https://bugs.launchpad.net/bugs/2150671 On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone. Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active, always prefer fully-idle SMT cores over partially-idle ones. Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup. On an SMT system with asymmetric CPU capacities, SMT-aware idle selection has been shown to improve throughput by around 15-18% for CPU-bound workloads, running an amount of tasks equal to the amount of SMT cores. Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Christian Loehle Cc: Koba Ko Reviewed-by: K Prateek Nayak Reported-by: Felix Abecassis Signed-off-by: Andrea Righi (cherry picked from https://lore.kernel.org/all/20260428051720.3180182-1-arighi@nvidia.com) Signed-off-by: Andrea Righi Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- kernel/sched/fair.c | 70 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 65 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 70b402765efb5..73c8ff94a5795 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7757,6 +7757,22 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool return idle_cpu; } +/* + * Idle-capacity scan ranks transformed util_fits_cpu() outcomes; lower values + * are more preferred (see select_idle_capacity()). + */ +enum asym_fits_state { + /* In descending order of preference */ + ASYM_IDLE_CORE_UCLAMP_MISFIT = -4, + ASYM_IDLE_CORE_COMPLETE_MISFIT, + ASYM_IDLE_THREAD_FITS, + ASYM_IDLE_THREAD_UCLAMP_MISFIT, + ASYM_IDLE_COMPLETE_MISFIT, + + /* asym_fits_cpu() bias for an idle core. */ + ASYM_IDLE_CORE_BIAS = -3, +}; + /* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which * the task fits. If no CPU is big enough, but there are idle ones, try to @@ -7765,8 +7781,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) { + bool prefers_idle_core = sched_smt_active() && test_idle_cores(target); unsigned long task_util, util_min, util_max, best_cap = 0; - int fits, best_fits = 0; + int fits, best_fits = ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu = -1; struct cpumask *cpus; @@ -7778,6 +7795,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max = uclamp_eff_value(p, UCLAMP_MAX); for_each_cpu_wrap(cpu, cpus, target) { + bool preferred_core = !prefers_idle_core || is_core_idle(cpu); unsigned long cpu_cap = capacity_of(cpu); if (!available_idle_cpu(cpu) && !sched_idle_cpu(cpu)) @@ -7786,7 +7804,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits = util_fits_cpu(task_util, util_min, util_max, cpu); /* This CPU fits with all requirements */ - if (fits > 0) + if (fits > 0 && preferred_core) return cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -7794,9 +7812,33 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) */ else if (fits < 0) cpu_cap = get_actual_cpu_capacity(cpu); + /* + * fits > 0 implies we are not on a preferred core + * but the util fits CPU capacity. Set fits to ASYM_IDLE_THREAD_FITS + * so the effective range becomes + * [ASYM_IDLE_THREAD_FITS, ASYM_IDLE_COMPLETE_MISFIT] where: + * ASYM_IDLE_COMPLETE_MISFIT - does not fit + * ASYM_IDLE_THREAD_UCLAMP_MISFIT - fits with the exception of UCLAMP_MIN + * ASYM_IDLE_THREAD_FITS - fits with the exception of preferred_core + */ + else if (fits > 0) + fits = ASYM_IDLE_THREAD_FITS; + + /* + * If we are on a preferred core, translate the range of fits + * of [ASYM_IDLE_THREAD_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT] to + * [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_COMPLETE_MISFIT]. + * This ensures that an idle core is always given priority over + * (partially) busy core. + * + * A fully fitting idle core would have returned early and hence + * fits > 0 for preferred_core need not be dealt with. + */ + if (preferred_core) + fits += ASYM_IDLE_CORE_BIAS; /* - * First, select CPU which fits better (-1 being better than 0). + * First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) || @@ -7807,6 +7849,19 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) } } + /* + * A value in the [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_BIAS] + * range means the chosen CPU is in a fully idle SMT core. Values above + * ASYM_IDLE_CORE_BIAS mean we never ranked such a CPU best. + * + * The asym-capacity wakeup path returns from select_idle_sibling() + * after this function and never runs select_idle_cpu(), so the usual + * select_idle_cpu() tail that clears idle cores must live here when the + * idle-core preference did not win. + */ + if (prefers_idle_core && best_fits > ASYM_IDLE_CORE_BIAS) + set_idle_cores(target, false); + return best_cpu; } @@ -7815,12 +7870,17 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) { /* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. + * + * When SMT is active, also require that the core has no busy + * siblings. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); + return (!sched_smt_active() || is_core_idle(cpu)) && + (util_fits_cpu(util, util_min, util_max, cpu) > 0); + } return true; } From 90cc25da99419bcc92161c77c8c8dc0faad69f23 Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Wed, 25 Mar 2026 16:39:32 +0100 Subject: [PATCH 183/464] NVIDIA: VR: SAUCE: sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity BugLink: https://bugs.launchpad.net/bugs/2150671 When SD_ASYM_CPUCAPACITY load balancing considers pulling a misfit task, capacity_of(dst_cpu) can overstate available compute if the SMT sibling is busy: the core does not deliver its full nominal capacity. If SMT is active and dst_cpu is not on a fully idle core, skip this destination so we do not migrate a misfit expecting a capacity upgrade we cannot actually provide. Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Christian Loehle Cc: Koba Ko Cc: K Prateek Nayak Reported-by: Felix Abecassis Signed-off-by: Andrea Righi (cherry picked from https://lore.kernel.org/all/20260428051720.3180182-1-arighi@nvidia.com) Signed-off-by: Andrea Righi Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- kernel/sched/fair.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 73c8ff94a5795..ad0ac6e72f96b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9392,6 +9392,7 @@ struct lb_env { int dst_cpu; struct rq *dst_rq; + bool dst_core_idle; struct cpumask *dst_grpmask; int new_dst_cpu; @@ -10637,10 +10638,16 @@ static bool update_sd_pick_busiest(struct lb_env *env, * We can use max_capacity here as reduction in capacity on some * CPUs in the group should either be possible to resolve * internally or be covered by avg_load imbalance (eventually). + * + * When SMT is active, only pull a misfit to dst_cpu if it is on a + * fully idle core; otherwise the effective capacity of the core is + * reduced and we may not actually provide more capacity than the + * source. */ if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && (sgs->group_type == group_misfit_task) && - (!capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) || + (!env->dst_core_idle || + !capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) || sds->local_stat.group_type != group_has_spare)) return false; @@ -11206,6 +11213,8 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd unsigned long sum_util = 0; bool sg_overloaded = 0, sg_overutilized = 0; + env->dst_core_idle = !sched_smt_active() || is_core_idle(env->dst_cpu); + do { struct sg_lb_stats *sgs = &tmp_sgs; int local_group; From 4447b0574d8722e7dbac34f84ca11aa31d3bb016 Mon Sep 17 00:00:00 2001 From: K Prateek Nayak Date: Tue, 21 Apr 2026 16:52:46 +0530 Subject: [PATCH 184/464] NVIDIA: VR: SAUCE: sched/fair: Add SIS_UTIL support to select_idle_capacity() BugLink: https://bugs.launchpad.net/bugs/2150671 Add to select_idle_capacity() the same SIS_UTIL-controlled idle-scan mechanism, already used by select_idle_cpu(): when sched_feat(SIS_UTIL) is enabled and the LLC domain has sched_domain_shared data, derive the per-attempt scan limit from sd->shared->nr_idle_scan. That bounds the walk on large LLCs and allows an early return once the scan limit is reached, if we already picked a sufficiently strong idle-core candidate (best_fits == ASYM_IDLE_CORE_UCLAMP_MISFIT). Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak (cherry picked from https://lore.kernel.org/all/20260428051720.3180182-1-arighi@nvidia.com) Signed-off-by: Andrea Righi Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- kernel/sched/fair.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ad0ac6e72f96b..def97f2d6a470 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7786,6 +7786,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) int fits, best_fits = ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu = -1; struct cpumask *cpus; + int nr = INT_MAX; cpus = this_cpu_cpumask_var_ptr(select_rq_mask); cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); @@ -7794,10 +7795,28 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_min = uclamp_eff_value(p, UCLAMP_MIN); util_max = uclamp_eff_value(p, UCLAMP_MAX); + if (sched_feat(SIS_UTIL) && sd->shared) { + /* + * Same nr_idle_scan hint as select_idle_cpu(), nr only limits + * the scan when not preferring an idle core. + */ + nr = READ_ONCE(sd->shared->nr_idle_scan) + 1; + /* overloaded domain is unlikely to have idle cpu/core */ + if (nr == 1) + return -1; + } + for_each_cpu_wrap(cpu, cpus, target) { bool preferred_core = !prefers_idle_core || is_core_idle(cpu); unsigned long cpu_cap = capacity_of(cpu); + /* + * Good-enough early exit (mirrors select_idle_cpu() logic). + */ + if (!prefers_idle_core && + --nr <= 0 && best_fits == ASYM_IDLE_CORE_UCLAMP_MISFIT) + return best_cpu; + if (!available_idle_cpu(cpu) && !sched_idle_cpu(cpu)) continue; From 13a3d67dc587d7da030a8274c98013992c610e94 Mon Sep 17 00:00:00 2001 From: Nicolai Buchwitz Date: Mon, 23 Feb 2026 09:54:42 +0100 Subject: [PATCH 185/464] net: microchip: lan743x: add ethtool nway_reset support BugLink: https://bugs.launchpad.net/bugs/2152064 Wire phylink_ethtool_nway_reset() as the .nway_reset ethtool operation, allowing userspace to restart PHY autonegotiation via 'ethtool -r'. Signed-off-by: Nicolai Buchwitz Reviewed-by: Russel King (Oracle) Link: https://patch.msgid.link/20260223085442.42852-1-nb@tipi-net.de Signed-off-by: Paolo Abeni (cherry picked from commit 8636385b9f0175220318bc23f6926b3fd013b131) Signed-off-by: David Thompson Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/microchip/lan743x_ethtool.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/ethernet/microchip/lan743x_ethtool.c b/drivers/net/ethernet/microchip/lan743x_ethtool.c index 8a3c1ecc7866c..9195419ecee02 100644 --- a/drivers/net/ethernet/microchip/lan743x_ethtool.c +++ b/drivers/net/ethernet/microchip/lan743x_ethtool.c @@ -1079,6 +1079,13 @@ static int lan743x_ethtool_set_eee(struct net_device *netdev, return phylink_ethtool_set_eee(adapter->phylink, eee); } +static int lan743x_ethtool_nway_reset(struct net_device *netdev) +{ + struct lan743x_adapter *adapter = netdev_priv(netdev); + + return phylink_ethtool_nway_reset(adapter->phylink); +} + static int lan743x_ethtool_set_link_ksettings(struct net_device *netdev, const struct ethtool_link_ksettings *cmd) @@ -1369,6 +1376,7 @@ const struct ethtool_ops lan743x_ethtool_ops = { .set_rxfh = lan743x_ethtool_set_rxfh, .get_rxfh_fields = lan743x_ethtool_get_rxfh_fields, .get_ts_info = lan743x_ethtool_get_ts_info, + .nway_reset = lan743x_ethtool_nway_reset, .get_eee = lan743x_ethtool_get_eee, .set_eee = lan743x_ethtool_set_eee, .get_link_ksettings = lan743x_ethtool_get_link_ksettings, From 1a7060e03adfef9f561cddf1f69ad6a9312056fb Mon Sep 17 00:00:00 2001 From: Thangaraj Samynathan Date: Wed, 18 Mar 2026 12:02:28 +0530 Subject: [PATCH 186/464] net: lan743x: fix SGMII detection on PCI1xxxx B0+ during warm reset BugLink: https://bugs.launchpad.net/bugs/2152064 A warm reset on boards using an EEPROM-only strap configuration (where no MAC address is set in the image) can cause the driver to incorrectly revert to RGMII mode. This occurs because the ENET_CONFIG_LOAD_STARTED bit may not persist or behave as expected. Update pci11x1x_strap_get_status() to use revision-specific validation: - For PCI11x1x A0: Continue using the legacy check (config load started or reset protection) to validate the SGMII strap. - For PCI11x1x B0 and later: Use the newly available STRAP_READ_USE_SGMII_EN_ bit in the upper strap register to validate the lower SGMII_EN bit. This ensures the SGMII interface is correctly identified even after a warm reboot. Signed-off-by: Thangaraj Samynathan Link: https://patch.msgid.link/20260318063228.17110-1-thangaraj.s@microchip.com Signed-off-by: Jakub Kicinski (cherry picked from commit e783e40fb689381caca31e03d28c39e10c82e722) Signed-off-by: David Thompson Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/microchip/lan743x_main.c | 15 +++++++++++---- drivers/net/ethernet/microchip/lan743x_main.h | 1 + 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c index f0b5dd752f084..b4cabde6625a2 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.c +++ b/drivers/net/ethernet/microchip/lan743x_main.c @@ -28,6 +28,12 @@ #define RFE_RD_FIFO_TH_3_DWORDS 0x3 +static bool pci11x1x_is_a0(struct lan743x_adapter *adapter) +{ + u32 dev_rev = adapter->csr.id_rev & ID_REV_CHIP_REV_MASK_; + return dev_rev == ID_REV_CHIP_REV_PCI11X1X_A0_; +} + static void pci11x1x_strap_get_status(struct lan743x_adapter *adapter) { u32 chip_rev; @@ -47,10 +53,11 @@ static void pci11x1x_strap_get_status(struct lan743x_adapter *adapter) cfg_load = lan743x_csr_read(adapter, ETH_SYS_CONFIG_LOAD_STARTED_REG); lan743x_hs_syslock_release(adapter); hw_cfg = lan743x_csr_read(adapter, HW_CFG); - - if (cfg_load & GEN_SYS_LOAD_STARTED_REG_ETH_ || - hw_cfg & HW_CFG_RST_PROTECT_) { - strap = lan743x_csr_read(adapter, STRAP_READ); + strap = lan743x_csr_read(adapter, STRAP_READ); + if ((pci11x1x_is_a0(adapter) && + (cfg_load & GEN_SYS_LOAD_STARTED_REG_ETH_ || + hw_cfg & HW_CFG_RST_PROTECT_)) || + (strap & STRAP_READ_USE_SGMII_EN_)) { if (strap & STRAP_READ_SGMII_EN_) adapter->is_sgmii_en = true; else diff --git a/drivers/net/ethernet/microchip/lan743x_main.h b/drivers/net/ethernet/microchip/lan743x_main.h index 02a28b7091630..160d94a7cee66 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.h +++ b/drivers/net/ethernet/microchip/lan743x_main.h @@ -27,6 +27,7 @@ #define ID_REV_CHIP_REV_MASK_ (0x0000FFFF) #define ID_REV_CHIP_REV_A0_ (0x00000000) #define ID_REV_CHIP_REV_B0_ (0x00000010) +#define ID_REV_CHIP_REV_PCI11X1X_A0_ (0x000000A0) #define ID_REV_CHIP_REV_PCI11X1X_B0_ (0x000000B0) #define FPGA_REV (0x04) From bdc7cbefd8531c6d1ed469f569d4be00b2733cbd Mon Sep 17 00:00:00 2001 From: Thangaraj Samynathan Date: Fri, 10 Apr 2026 14:27:10 +0530 Subject: [PATCH 187/464] net: lan743x: rename chip_rev to fpga_rev BugLink: https://bugs.launchpad.net/bugs/2152064 The variable chip_rev stores the value read from the FPGA_REV register and represents the FPGA revision. Rename it to fpga_rev to better reflect its meaning. No functional change intended. Signed-off-by: Thangaraj Samynathan Link: https://patch.msgid.link/20260410085710.9246-1-thangaraj.s@microchip.com Signed-off-by: Jakub Kicinski (cherry picked from commit 469faa546e7a82be85114e322cec6438790870ff) Signed-off-by: David Thompson Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/microchip/lan743x_main.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c index b4cabde6625a2..f3332417162e6 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.c +++ b/drivers/net/ethernet/microchip/lan743x_main.c @@ -36,7 +36,7 @@ static bool pci11x1x_is_a0(struct lan743x_adapter *adapter) static void pci11x1x_strap_get_status(struct lan743x_adapter *adapter) { - u32 chip_rev; + u32 fpga_rev; u32 cfg_load; u32 hw_cfg; u32 strap; @@ -63,9 +63,9 @@ static void pci11x1x_strap_get_status(struct lan743x_adapter *adapter) else adapter->is_sgmii_en = false; } else { - chip_rev = lan743x_csr_read(adapter, FPGA_REV); - if (chip_rev) { - if (chip_rev & FPGA_SGMII_OP) + fpga_rev = lan743x_csr_read(adapter, FPGA_REV); + if (fpga_rev) { + if (fpga_rev & FPGA_SGMII_OP) adapter->is_sgmii_en = true; else adapter->is_sgmii_en = false; From 97a981a2238172e81dc14ec7296404e68c36330d Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:24:17 -0700 Subject: [PATCH 188/464] Revert "NVIDIA: VR: SAUCE: perf vendor events arm64: Add Tegra410 Olympus PMU events" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit cf682dc04b5c460266809f96c9122abd728cbfe4. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../arch/arm64/common-and-microarch.json | 90 --- tools/perf/pmu-events/arch/arm64/mapfile.csv | 1 - .../arch/arm64/nvidia/t410/branch.json | 45 -- .../arch/arm64/nvidia/t410/brbe.json | 6 - .../arch/arm64/nvidia/t410/bus.json | 48 -- .../arch/arm64/nvidia/t410/exception.json | 62 -- .../arch/arm64/nvidia/t410/fp_operation.json | 78 -- .../arch/arm64/nvidia/t410/general.json | 15 - .../arch/arm64/nvidia/t410/l1d_cache.json | 122 --- .../arch/arm64/nvidia/t410/l1i_cache.json | 114 --- .../arch/arm64/nvidia/t410/l2d_cache.json | 134 ---- .../arch/arm64/nvidia/t410/ll_cache.json | 107 --- .../arch/arm64/nvidia/t410/memory.json | 46 -- .../arch/arm64/nvidia/t410/metrics.json | 722 ------------------ .../arch/arm64/nvidia/t410/misc.json | 646 ---------------- .../arch/arm64/nvidia/t410/retired.json | 94 --- .../arch/arm64/nvidia/t410/spe.json | 42 - .../arm64/nvidia/t410/spec_operation.json | 230 ------ .../arch/arm64/nvidia/t410/stall.json | 145 ---- .../arch/arm64/nvidia/t410/tlb.json | 158 ---- 20 files changed, 2905 deletions(-) delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json delete mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json diff --git a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json index 6af15776ff170..468cb085d8796 100644 --- a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json +++ b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json @@ -179,11 +179,6 @@ "EventName": "BUS_CYCLES", "BriefDescription": "Bus cycle" }, - { - "EventCode": "0x001E", - "EventName": "CHAIN", - "BriefDescription": "Chain a pair of event counters." - }, { "PublicDescription": "Level 1 data cache allocation without refill", "EventCode": "0x1F", @@ -1517,26 +1512,11 @@ "EventName": "L2D_CACHE_REFILL_PRFM", "BriefDescription": "Level 2 data cache refill, software preload" }, - { - "EventCode": "0x8150", - "EventName": "L3D_CACHE_RW", - "BriefDescription": "Level 3 data cache demand access." - }, - { - "EventCode": "0x8151", - "EventName": "L3D_CACHE_PRFM", - "BriefDescription": "Level 3 data cache software prefetch" - }, { "EventCode": "0x8152", "EventName": "L3D_CACHE_MISS", "BriefDescription": "Level 3 data cache demand access miss" }, - { - "EventCode": "0x8153", - "EventName": "L3D_CACHE_REFILL_PRFM", - "BriefDescription": "Level 3 data cache refill, software prefetch." - }, { "EventCode": "0x8154", "EventName": "L1D_CACHE_HWPRF", @@ -1547,11 +1527,6 @@ "EventName": "L2D_CACHE_HWPRF", "BriefDescription": "Level 2 data cache hardware prefetch." }, - { - "EventCode": "0x8156", - "EventName": "L3D_CACHE_HWPRF", - "BriefDescription": "Level 3 data cache hardware prefetch." - }, { "EventCode": "0x8158", "EventName": "STALL_FRONTEND_MEMBOUND", @@ -1707,11 +1682,6 @@ "EventName": "L2D_CACHE_REFILL_HWPRF", "BriefDescription": "Level 2 data cache refill, hardware prefetch." }, - { - "EventCode": "0x81BE", - "EventName": "L3D_CACHE_REFILL_HWPRF", - "BriefDescription": "Level 3 data cache refill, hardware prefetch." - }, { "EventCode": "0x81C0", "EventName": "L1I_CACHE_HIT_RD", @@ -1742,31 +1712,11 @@ "EventName": "L1I_CACHE_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by software preload" }, - { - "EventCode": "0x81DC", - "EventName": "L1D_CACHE_HIT_RW_FPRFM", - "BriefDescription": "Level 1 data cache demand access first hit, fetched by software prefetch." - }, { "EventCode": "0x81E0", "EventName": "L1I_CACHE_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by hardware prefetcher" }, - { - "EventCode": "0x81EC", - "EventName": "L1D_CACHE_HIT_RW_FHWPRF", - "BriefDescription": "Level 1 data cache demand access first hit, fetched by hardware prefetcher." - }, - { - "EventCode": "0x81F0", - "EventName": "L1I_CACHE_HIT_RD_FPRF", - "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by prefetch." - }, - { - "EventCode": "0x81FC", - "EventName": "L1D_CACHE_HIT_RW_FPRF", - "BriefDescription": "Level 1 data cache demand access first hit, fetched by prefetch." - }, { "EventCode": "0x8200", "EventName": "L1I_CACHE_HIT", @@ -1817,26 +1767,11 @@ "EventName": "L1I_LFB_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by software preload" }, - { - "EventCode": "0x825C", - "EventName": "L1D_LFB_HIT_RW_FPRFM", - "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by software prefetch." - }, { "EventCode": "0x8260", "EventName": "L1I_LFB_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by hardware prefetcher" }, - { - "EventCode": "0x826C", - "EventName": "L1D_LFB_HIT_RW_FHWPRF", - "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by hardware prefetcher." - }, - { - "EventCode": "0x827C", - "EventName": "L1D_LFB_HIT_RW_FPRF", - "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by prefetch." - }, { "EventCode": "0x8280", "EventName": "L1I_CACHE_PRF", @@ -1872,11 +1807,6 @@ "EventName": "LL_CACHE_REFILL", "BriefDescription": "Last level cache refill" }, - { - "EventCode": "0x828E", - "EventName": "L3D_CACHE_REFILL_PRF", - "BriefDescription": "Level 3 data cache refill, prefetch." - }, { "EventCode": "0x8320", "EventName": "L1D_CACHE_REFILL_PERCYC", @@ -1942,16 +1872,6 @@ "EventName": "FP_FP8_MIN_SPEC", "BriefDescription": "Floating-point operation speculatively_executed, smallest type is 8-bit floating-point." }, - { - "EventCode": "0x8480", - "EventName": "FP_SP_FIXED_MIN_OPS_SPEC", - "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." - }, - { - "EventCode": "0x8482", - "EventName": "FP_HP_FIXED_MIN_OPS_SPEC", - "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." - }, { "EventCode": "0x8483", "EventName": "FP_BF16_FIXED_MIN_OPS_SPEC", @@ -1962,16 +1882,6 @@ "EventName": "FP_FP8_FIXED_MIN_OPS_SPEC", "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is 8-bit floating-point." }, - { - "EventCode": "0x8488", - "EventName": "FP_SP_SCALE_MIN_OPS_SPEC", - "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." - }, - { - "EventCode": "0x848A", - "EventName": "FP_HP_SCALE_MIN_OPS_SPEC", - "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." - }, { "EventCode": "0x848B", "EventName": "FP_BF16_SCALE_MIN_OPS_SPEC", diff --git a/tools/perf/pmu-events/arch/arm64/mapfile.csv b/tools/perf/pmu-events/arch/arm64/mapfile.csv index 7f0eaa7020485..bb3fa8a33496a 100644 --- a/tools/perf/pmu-events/arch/arm64/mapfile.csv +++ b/tools/perf/pmu-events/arch/arm64/mapfile.csv @@ -46,4 +46,3 @@ 0x00000000500f0000,v1,ampere/emag,core 0x00000000c00fac30,v1,ampere/ampereone,core 0x00000000c00fac40,v1,ampere/ampereonex,core -0x000000004e0f0100,v1,nvidia/t410,core diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json deleted file mode 100644 index 532bc59dc573e..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json +++ /dev/null @@ -1,45 +0,0 @@ -[ - { - "ArchStdEvent": "BR_MIS_PRED", - "PublicDescription": "The Event counts Branches which are speculatively executed and mis-predicted." - }, - { - "ArchStdEvent": "BR_PRED", - "PublicDescription": "The Event counts all speculatively executed Branches." - }, - { - "EventCode": "0x017e", - "EventName": "BR_PRED_BTB_CTX_UPDATE", - "PublicDescription": "Branch context table update." - }, - { - "EventCode": "0x0188", - "EventName": "BR_MIS_PRED_DIR_RESOLVED", - "PublicDescription": "Number of Branch misprediction due to direction misprediction." - }, - { - "EventCode": "0x0189", - "EventName": "BR_MIS_PRED_DIR_UNCOND_RESOLVED", - "PublicDescription": "Number of Branch misprediction due to direction misprediction for unconditional Branches." - }, - { - "EventCode": "0x018a", - "EventName": "BR_MIS_PRED_DIR_UNCOND_DIRECT_RESOLVED", - "PublicDescription": "Number of Branch misprediction due to direction misprediction for unconditional direct Branches." - }, - { - "EventCode": "0x018b", - "EventName": "BR_PRED_MULTI_RESOLVED", - "PublicDescription": "Number of resolved branch which made prediction by polymorphic indirect predictor." - }, - { - "EventCode": "0x018c", - "EventName": "BR_MIS_PRED_MULTI_RESOLVED", - "PublicDescription": "Number of branch misprediction which made prediction by polymorphic indirect predictor." - }, - { - "EventCode": "0x01e4", - "EventName": "BR_RGN_RECLAIM", - "PublicDescription": "The Event counts the Indirect predictor entries flushed by region reclamation." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json deleted file mode 100644 index 7c43a01c9707d..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json +++ /dev/null @@ -1,6 +0,0 @@ -[ - { - "ArchStdEvent": "BRB_FILTRATE", - "PublicDescription": "The Event counts each valid Branch record captured in the Branch record buffer. Branch records that are not captured because they are removed by filtering are not counted." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json deleted file mode 100644 index c4cee0be1242e..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json +++ /dev/null @@ -1,48 +0,0 @@ -[ - { - "ArchStdEvent": "BUS_ACCESS", - "PublicDescription": "The Event counts the number of Data-beat access between the CPU and the external bus. This count includes access due to Read, Write, and Snoop. Each beat of Data is counted individually." - }, - { - "ArchStdEvent": "BUS_CYCLES", - "PublicDescription": "The Event counts bus cycles in the CPU. Bus cycles represent a clock cycle in which a transaction could be sent or received on the interface from the CPU to the external bus. Since that interface is driven at the same clock speed as the CPU, this Event increments at the rate of CPU clock. Regardless of the WFE/WFI state of the PE, this Event increment on each processor clock." - }, - { - "ArchStdEvent": "BUS_ACCESS_RD", - "PublicDescription": "The Event counts memory Read transactions seen on the external bus. Each beat of Data is counted individually." - }, - { - "ArchStdEvent": "BUS_ACCESS_WR", - "PublicDescription": "The Event counts memory Write transactions seen on the external bus. Each beat of Data is counted individually." - }, - { - "EventCode": "0x0154", - "EventName": "BUS_REQUEST_REQ", - "PublicDescription": "Bus request, request." - }, - { - "EventCode": "0x0155", - "EventName": "BUS_REQUEST_RETRY", - "PublicDescription": "Bus request, retry." - }, - { - "EventCode": "0x0198", - "EventName": "L2_CHI_CBUSY0", - "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 0." - }, - { - "EventCode": "0x0199", - "EventName": "L2_CHI_CBUSY1", - "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 1." - }, - { - "EventCode": "0x019a", - "EventName": "L2_CHI_CBUSY2", - "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 2." - }, - { - "EventCode": "0x019b", - "EventName": "L2_CHI_CBUSY3", - "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 3." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json deleted file mode 100644 index 2f31fb2e67a25..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json +++ /dev/null @@ -1,62 +0,0 @@ -[ - { - "ArchStdEvent": "EXC_TAKEN", - "PublicDescription": "The Event counts any taken architecturally visible exceptions such as IRQ, FIQ, SError, and other synchronous exceptions. Exceptions are counted whether or not they are taken locally." - }, - { - "ArchStdEvent": "EXC_RETURN", - "PublicDescription": "The Event counts any architecturally executed exception return Instructions. For example: AArch64: ERET." - }, - { - "ArchStdEvent": "EXC_UNDEF", - "PublicDescription": "The Event counts the number of synchronous exceptions which are taken locally that are due to attempting to execute an Instruction that is UNDEFINED. Attempting to execute Instruction bit patterns that have not been allocated. Attempting to execute Instructions when they are disabled. Attempting to execute Instructions at an inappropriate Exception level. Attempting to execute an Instruction when the value of PSTATE.IL is 1." - }, - { - "ArchStdEvent": "EXC_SVC", - "PublicDescription": "The Event counts SVC exceptions taken locally." - }, - { - "ArchStdEvent": "EXC_PABORT", - "PublicDescription": "The Event counts synchronous exceptions that are taken locally and caused by Instruction Aborts." - }, - { - "ArchStdEvent": "EXC_DABORT", - "PublicDescription": "The Event counts exceptions that are taken locally and are caused by Data aborts or SErrors. Conditions that could cause those exceptions are attempting to read or write memory where the MMU generates a fault, attempting to read or write memory with a misaligned address, Interrupts from the nSEI inputs and internally generated SErrors." - }, - { - "ArchStdEvent": "EXC_IRQ", - "PublicDescription": "The Event counts IRQ exceptions including the virtual IRQs that are taken locally." - }, - { - "ArchStdEvent": "EXC_FIQ", - "PublicDescription": "The Event counts FIQ exceptions including the virtual FIQs that are taken locally." - }, - { - "ArchStdEvent": "EXC_SMC", - "PublicDescription": "The Event counts SMC exceptions take to EL3." - }, - { - "ArchStdEvent": "EXC_HVC", - "PublicDescription": "The Event counts HVC exceptions taken to EL2." - }, - { - "ArchStdEvent": "EXC_TRAP_PABORT", - "PublicDescription": "The Event counts exceptions which are traps not taken locally and are caused by Instruction Aborts. For example, attempting to execute an Instruction with a misaligned PC." - }, - { - "ArchStdEvent": "EXC_TRAP_DABORT", - "PublicDescription": "The Event counts exceptions which are traps not taken locally and are caused by Data Aborts or SError Interrupts. Conditions that could cause those exceptions are: * Attempting to read or write memory where the MMU generates a fault, * Attempting to read or write memory with a misaligned address, * Interrupts from the SEI input. * internally generated SErrors." - }, - { - "ArchStdEvent": "EXC_TRAP_OTHER", - "PublicDescription": "The Event counts the number of synchronous trap exceptions which are not taken locally and are not SVC, SMC, HVC, Data aborts, Instruction Aborts, or Interrupts." - }, - { - "ArchStdEvent": "EXC_TRAP_IRQ", - "PublicDescription": "The Event counts IRQ exceptions including the virtual IRQs that are not taken locally." - }, - { - "ArchStdEvent": "EXC_TRAP_FIQ", - "PublicDescription": "The Event counts FIQs which are not taken locally but taken from EL0, EL1, or EL2 to EL3 (which would be the normal behavior for FIQs when not executing in EL3)." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json deleted file mode 100644 index 0b6d047207518..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json +++ /dev/null @@ -1,78 +0,0 @@ -[ - { - "ArchStdEvent": "FP_HP_SPEC", - "PublicDescription": "The Event counts speculatively executed half precision floating point operations." - }, - { - "ArchStdEvent": "FP_SP_SPEC", - "PublicDescription": "The Event counts speculatively executed single precision floating point operations." - }, - { - "ArchStdEvent": "FP_DP_SPEC", - "PublicDescription": "The Event counts speculatively executed double precision floating point operations." - }, - { - "ArchStdEvent": "FP_SCALE_OPS_SPEC", - "PublicDescription": "The Event counts speculatively executed scalable single precision floating point operations." - }, - { - "ArchStdEvent": "FP_FIXED_OPS_SPEC", - "PublicDescription": "The Event counts speculatively executed non-scalable single precision floating point operations." - }, - { - "ArchStdEvent": "FP_HP_SCALE_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes the counter to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_HP_FIXED_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The even to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_SP_SCALE_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_SP_FIXED_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_DP_SCALE_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the largest type was double-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_DP_FIXED_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the largest type was double-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_BF16_FIXED_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was BFloat16 floating-point. Where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_BF16_SCALE_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was BFloat16 floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_FP8_FIXED_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was 8-bit floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_FP8_SCALE_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was 8-bit floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_HP_FIXED_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_HP_SCALE_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_SP_FIXED_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an Instruction where the smallest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." - }, - { - "ArchStdEvent": "FP_SP_SCALE_MIN_OPS_SPEC", - "PublicDescription": "The Event increments by v for each speculatively executed scalable element arithmetic operation, due to an Instruction where the smallest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or Instruction which causes The Event to increment. The Event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json deleted file mode 100644 index bbeb4a75d2618..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json +++ /dev/null @@ -1,15 +0,0 @@ -[ - { - "ArchStdEvent": "CPU_CYCLES", - "PublicDescription": "The Event counts CPU clock cycles, when the PE is not in WFE/WFI. The clock measured by this Event is defined as the physical clock driving the CPU logic." - }, - { - "ArchStdEvent": "CNT_CYCLES", - "PublicDescription": "The Event increments at a constant frequency equal to the rate of increment of the System Counter, CNTPCT_EL0. This Event does not increment when the PE is in WFE/WFI." - }, - { - "EventCode": "0x01e1", - "EventName": "CPU_SLOT", - "PublicDescription": "Entitled CPU slots. The Event counts the number of slots. When in ST mode, this Event shall increment by PMMIR_EL1.SLOTS quantities, and when in SMT partitioned resource mode (regardless of in WFI state or otherwise), this Event is incremented by PMMIR_EL1.SLOTS/2 quantities." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json deleted file mode 100644 index 7c9b2fc4b38cb..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json +++ /dev/null @@ -1,122 +0,0 @@ -[ - { - "ArchStdEvent": "L1D_CACHE_REFILL", - "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Load or Store operations or preload Instructions or hardware cache prefetching that missed in the L1 D-cache. This Event only counts one Event per cache line. Since the caches are Write-back only for this processor, there are no Write-through cache accesses." - }, - { - "ArchStdEvent": "L1D_CACHE", - "PublicDescription": "The Event counts L1 D-cache accesses from any Load/Store operations, software preload or hardware prefetch operation. Atomic operations that resolve in the CPUs caches (near atomic operations) counts as both a Write access and Read access. Each access to a cache line is counted including the multiple accesses caused by single Instructions such as LDM or STM. Each access to other L1 Data or unified memory structures, for example refill buffers, Write buffers, and Write-back buffers, are also counted. This Event counts the sum of L1D_CACHE_RD, L1D_CACHE_WR, L1D_CACHE_PRFM and L1D_CACHE_HWPRF." - }, - { - "ArchStdEvent": "L1D_CACHE_WB", - "PublicDescription": "The Event counts Write-backs of dirty Data from the L1 D-cache to the L2 cache. This occurs when either a dirty cache line is evicted from L1 D-cache and allocated in the L2 cache or dirty Data is written to the L2 and possibly to the next level of cache. This Event counts both victim cache line evictions and cache Write-backs from snoops or cache maintenance operations. The following cache operations are not counted: * Invalidations which do not result in Data being transferred out of the L1 (such as evictions of clean Data), * Full line Writes which write to L2 without writing L1, such as Write streaming mode. This Event is the sum of the L1D_CACHE_WB_CLEAN and L1D_CACHE_WB_VICTIM Events." - }, - { - "ArchStdEvent": "L1D_CACHE_LMISS_RD", - "PublicDescription": "The Event counts cache line refills into the L1 D-cache from any memory Read operations, that incurred additional latency. Counts same as L1D_CACHE_REFILL_RD on this CPU." - }, - { - "ArchStdEvent": "L1D_CACHE_RD", - "PublicDescription": "The Event counts L1 D-cache accesses from any Load operation. Atomic Load operations that resolve in the CPUs caches counts as both a Write access and Read access." - }, - { - "ArchStdEvent": "L1D_CACHE_WR", - "PublicDescription": "The Event counts L1 D-cache accesses generated by Store operations. This Event also counts accesses caused by a DC ZVA (D-cache zero, specified by virtual address) Instruction. Near atomic operations that resolve in the CPUs caches count as a Write access and Read access. This Event is a subset of the L1D_CACHE Event, except this Event only counts memory Write operations." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_RD", - "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Load Instructions where the memory Read operation misses in the L1 D-cache. This Event only counts one Event per cache line. This Event is a subset of the L1D_CACHE_REFILL Event, but only counts memory Read operations. This Event does not count reads caused by cache maintenance operations or preload Instructions." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_WR", - "PublicDescription": "The Event counts L1 D-cache refills caused by speculatively executed Store Instructions where the memory Write operation misses in the L1 D-cache. This Event only counts one Event per cache line. This Event is a subset of the L1D_CACHE_REFILL Event, but only counts memory Write operations." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_INNER", - "PublicDescription": "The Event counts L1 D-cache refills (L1D_CACHE_REFILL) where the cache line Data came from caches inside the immediate Cluster of the Core (L2 cache)." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_OUTER", - "PublicDescription": "The Event counts L1 D-cache refills (L1D_CACHE_REFILL) for which the cache line Data came from outside the immediate Cluster of the Core, like an SLC in the system interconnect or DRAM or remote socket." - }, - { - "ArchStdEvent": "L1D_CACHE_WB_VICTIM", - "PublicDescription": "The Event counts dirty cache line evictions from the L1 D-cache caused by a new cache line allocation. This Event does not count evictions caused by cache maintenance operations. This Event is a subset of the L1D_CACHE_WB Event, but the Event only counts Write-backs that are a result of the line being allocated for an access made by the CPU." - }, - { - "ArchStdEvent": "L1D_CACHE_WB_CLEAN", - "PublicDescription": "The Event counts Write-backs from the L1 D-cache that are a result of a coherency operation made by another CPU. Event count includes cache maintenance operations. This Event is a subset of the L1D_CACHE_WB Event." - }, - { - "ArchStdEvent": "L1D_CACHE_INVAL", - "PublicDescription": "The Event counts each explicit invalidation of a cache line in the L1 D-cache caused by: * Cache Maintenance Operations (CMO) that operate by a virtual address. * Broadcast cache coherency operations from another CPU in the system. This Event does not count for the following conditions: * A cache refill invalidates a cache line. * A CMO which is executed on that CPU and invalidates a cache line specified by Set/Way. Note that CMOs that operate by Set/Way cannot be broadcast from one CPU to another." - }, - { - "ArchStdEvent": "L1D_CACHE_RW", - "PublicDescription": "The Event counts L1 Data demand cache accesses from any Load or Store operation. Near atomic operations that resolve in the CPUs caches counts as both a Write access and Read access. This Event is implemented as L1D_CACHE_RD + L1D_CACHE_WR" - }, - { - "ArchStdEvent": "L1D_CACHE_PRFM", - "PublicDescription": "The Event counts L1 D-cache accesses from software preload or prefetch Instructions." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_PRFM", - "PublicDescription": "The Event counts L1 D-cache refills where the cache line access was generated by software preload or prefetch Instructions." - }, - { - "ArchStdEvent": "L1D_CACHE_HWPRF", - "PublicDescription": "The Event counts L1 D-cache accesses from any Load/Store operations generated by the hardware prefetcher." - }, - { - "ArchStdEvent": "L1D_CACHE_MISS", - "PublicDescription": "The Event counts each demand access counted by L1D_CACHE_RW that misses in the L1 Data or unified cache, causing an access to outside of the L1 caches of this PE." - }, - { - "ArchStdEvent": "L1D_CACHE_REFILL_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch access counted by L1D_CACHE_HWPRF that causes a refill of the L1 D-cache from outside of the L1 D-cache." - }, - { - "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRFM", - "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched in response to a prefetch Instruction. That is, the L1D_CACHE_REFILL_PRFM Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "ArchStdEvent": "L1D_CACHE_HIT_RW_FHWPRF", - "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched by a hardware prefetcher. That is, the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRF", - "PublicDescription": "The Event counts each demand access first hit counted by L1D_CACHE_HIT_RW where the cache line was fetched in response to a prefetch Instruction or by a hardware prefetcher. That is, the L1D_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "ArchStdEvent": "L1D_LFB_HIT_RW_FHWPRF", - "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF, where the cache line was fetched by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "ArchStdEvent": "L1D_LFB_HIT_RW_FPRFM", - "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF where the cache line was fetched in response to a prefetch Instruction. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and theL1D_CACHE_REFILL_PRFM Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "ArchStdEvent": "L1D_LFB_HIT_RW_FPRF", - "PublicDescription": "The Event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW where the cache line was fetched in response to a prefetch Instruction or by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x0204", - "EventName": "L1D_CACHE_REFILL_OUTER_LLC", - "PublicDescription": "The Event counts L1D_CACHE_REFILL from L3 D-cache." - }, - { - "EventCode": "0x0205", - "EventName": "L1D_CACHE_REFILL_OUTER_DRAM", - "PublicDescription": "The Event counts L1D_CACHE_REFILL from local memory." - }, - { - "EventCode": "0x0206", - "EventName": "L1D_CACHE_REFILL_OUTER_REMOTE", - "PublicDescription": "The Event counts L1D_CACHE_REFILL from a remote memory." - }, - { - "EventCode": "0x01f5", - "EventName": "L1D_CACHE_REFILL_RW", - "PublicDescription": "L1 D-cache refill, demand Read and Write. The Event counts demand Read and Write accesses that causes a refill of the L1 D-cache of this PE, from outside of this cache." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json deleted file mode 100644 index a1faa284b80f6..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json +++ /dev/null @@ -1,114 +0,0 @@ -[ - { - "ArchStdEvent": "L1I_CACHE_REFILL", - "PublicDescription": "The Event counts cache line refills in the L1 I-cache caused by a missed Instruction fetch (Demand, hardware prefetch and software preload accesses). Instruction fetches may include accessing multiple Instructions, but the single cache line allocation is counted once." - }, - { - "ArchStdEvent": "L1I_CACHE", - "PublicDescription": "The Event counts Instruction fetches (Demand, hardware prefetch and software preload accesses) which access the L1 Instruction Cache. Instruction Cache accesses caused by cache maintenance operations are not counted." - }, - { - "ArchStdEvent": "L1I_CACHE_LMISS", - "PublicDescription": "The Event counts cache line refills into the L1 I-cache, that incurred additional latency. Counts the same as L1I_CACHE_REFILL in this CPU." - }, - { - "ArchStdEvent": "L1I_CACHE_RD", - "PublicDescription": "The Event counts demand Instruction fetches which access the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_CACHE_PRFM", - "PublicDescription": "The Event counts Instruction fetches generated by software preload or prefetch Instructions which access the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_CACHE_HWPRF", - "PublicDescription": "The Event counts Instruction fetches which access the L1 I-cache generated by the hardware prefetcher." - }, - { - "ArchStdEvent": "L1I_CACHE_REFILL_PRFM", - "PublicDescription": "The Event counts cache line refills in the L1 I-cache caused by a missed Instruction fetch generated by software preload or prefetch Instructions. Instruction fetches may include accessing multiple Instructions, but the single cache line allocation is counted once." - }, - { - "ArchStdEvent": "L1I_CACHE_HIT_RD", - "PublicDescription": "The Event counts demand Instruction fetches that access the L1 I-cache and hit in the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_CACHE_HIT", - "PublicDescription": "The Event counts Instruction fetches that access the L1 I-cache (Demand, hardware prefetch and software preload accesses) and hit in the L1 I-cache. I-cache accesses caused by cache maintenance operations are not counted." - }, - { - "ArchStdEvent": "L1I_CACHE_HIT_PRFM", - "PublicDescription": "The Event counts Instruction fetches generated by software preload or prefetch Instructions that access the L1 I-cache and hit in the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_LFB_HIT_RD", - "PublicDescription": "The Event counts demand Instruction fetches that access the L1 I-cache and hit in a line that is in the process of being loaded into the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_CACHE_REFILL_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch access counted by L1I_CACHE_HWPRF that causes a refill of the Level 1I-cache from outside of the L1 I-cache." - }, - { - "ArchStdEvent": "L1I_CACHE_HIT_RD_FPRF", - "PublicDescription": "The Event counts each demand fetch first hit counted by L1I_CACHE_HIT_RD where the cache line was fetched in response to a software preload or by a hardware prefetcher. That is, the L1I_CACHE_REFILL_PRF Event was generated when the cache line was fetched into the cache. Only the first hit by a demand access is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x0174", - "EventName": "L1I_HWPRF_REQ_DROP", - "PublicDescription": "L1 I-cache hardware prefetch dropped." - }, - { - "EventCode": "0x01ea", - "EventName": "L1I_CFC_ENTRIES", - "PublicDescription": "The Event counts the CFC (Cache Fill Control) entries. The CFC is the fill buffer for I-cache." - }, - { - "EventCode": "0x0228", - "EventName": "L1I_CACHE_HIT_PRFM_FPRF", - "PublicDescription": "L1 I-cache software prefetch access first hit, fetched by hardware or software prefetch. The Event counts each software preload access first hit where the cache line was fetched in response to a hadware prefetcher or software preload Instruction. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x0212", - "EventName": "L1I_CACHE_HIT_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch access that hits an L1 I-cache." - }, - { - "EventCode": "0x022a", - "EventName": "L1I_CACHE_HIT_HWPRF_FPRF", - "PublicDescription": "L1 I-cache hardware prefetch access first hit, fetched by hardware or software prefetch. The Event counts each hardware prefetch access first hit where the cache line was fetched in response to a hardware or prefetch Instruction. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x0215", - "EventName": "L1I_LFB_HIT", - "PublicDescription": "L1 Line fill buffer hit. The Event counts each Demand or software preload or hardware prefetch induced Instruction fetch that hits an L1 I-cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." - }, - { - "EventCode": "0x0216", - "EventName": "L1I_LFB_HIT_PRFM", - "PublicDescription": "The Event counts each software prefetch access that hits a cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." - }, - { - "EventCode": "0x0219", - "EventName": "L1I_LFB_HIT_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch access that hits a cache line that is in the process of being loaded into the L1 Instruction, and so does not generate a new refill, but has to wait for the previous refill to complete." - }, - { - "EventCode": "0x0221", - "EventName": "L1I_PRFM_REQ", - "PublicDescription": "L1 I-cache software prefetch requests." - }, - { - "EventCode": "0x0222", - "EventName": "L1I_HWPRF_REQ", - "PublicDescription": "L1 I-cache hardware prefetch requests." - }, - { - "EventCode": "0x01e3", - "EventName": "L1I_CACHE_REFILL_RD", - "PublicDescription": "L1 I-cache refill, Read. The Event counts demand Instruction fetch that causes a refill of the L1 I-cache of this PE, from outside of this cache." - }, - { - "EventCode": "0x01ef", - "EventName": "L1I_CACHE_INVAL", - "PublicDescription": "L1 I-cache invalidate. The Event counts each explicit invalidation of a cache line in the L1 I-cache caused by: * Broadcast cache coherency operations from another CPU in the system. * Invalidation dues to capacity eviction in L2 D-cache. This Event does not count for the following conditions: * A cache refill invalidates a cache line. * A CMO which is executed on that CPU Core and invalidates a cache line specified by Set/Way. * Cache Maintenance Operations (CMO) that operate by a virtual address. Note that * CMOs that operate by Set/Way cannot be broadcast from one CPU Core to another. * The CMO is treated as No-op for the purposes of L1 I-cache line invalidation, as this Core implements fully coherent I-cache." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json deleted file mode 100644 index ac3f8095a9979..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json +++ /dev/null @@ -1,134 +0,0 @@ -[ - { - "ArchStdEvent": "L2D_CACHE", - "PublicDescription": "The Event counts accesses to the L2 cache due to Data accesses. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This Event also counts Write back of dirty Data from L1 D-cache to the L2 cache. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is the sum of the L2D_CACHE_RD, L2D_CACHE_WR, L2D_CACHE_PRFM, and L2D_CACHE_HWPRF Events." - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL", - "PublicDescription": "The Event counts cache line refills into the L2 cache. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented. This Event is the sum of L2D_CACHE_REFILL_RD, L2D_CACHE_REFILL_WR, L2D_CACHE_REFILL_HWPRF, and L2D_CACHE_REFILL_PRFM." - }, - { - "ArchStdEvent": "L2D_CACHE_WB", - "PublicDescription": "The Event counts Write-backs of Data from the L2 cache to outside the CPU. This includes snoops to the L2 (from other CPUs) which return Data even if the snoops cause an invalidation. L2 cache line invalidations which do not write Data outside the CPU and snoops which return Data from an L1 cache are not counted. Data would not be written outside the cache when invalidating a clean cache line. This Event is the sum of the L2D_CACHE_WB_VICTIM and L2D_CACHE_WB_CLEAN Events." - }, - { - "ArchStdEvent": "L2D_CACHE_RD", - "PublicDescription": "The Event counts L2 D-cache accesses due to memory Read operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is a subset of the L2D_CACHE Event, but this Event only counts memory Read operations." - }, - { - "ArchStdEvent": "L2D_CACHE_WR", - "PublicDescription": "The Event counts L2 cache accesses due to memory Write operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This Event is a subset of the L2D_CACHE Event, but this Event only counts memory Write operations." - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL_RD", - "PublicDescription": "The Event counts refills for memory accesses due to memory Read operation counted by L2D_CACHE_RD. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented. This Event is a subset of the L2D_CACHE_REFILL Event. This Event does not count L2 refills caused by stashes into L2. This count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL_WR", - "PublicDescription": "The Event counts refills for memory accesses due to memory Write operation counted by L2D_CACHE_WR. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." - }, - { - "ArchStdEvent": "L2D_CACHE_WB_VICTIM", - "PublicDescription": "The Event counts evictions from the L2 cache because of a line being allocated into the L2 cache. This Event is a subset of the L2D_CACHE_WB Event." - }, - { - "ArchStdEvent": "L2D_CACHE_WB_CLEAN", - "PublicDescription": "The Event counts Write-backs from the L2 cache that are a result of any of the following: * Cache maintenance operations, * Snoop responses or, * Direct cache transfers to another CPU due to a forwarding snoop request. This Event is a subset of the L2D_CACHE_WB Event." - }, - { - "ArchStdEvent": "L2D_CACHE_INVAL", - "PublicDescription": "The Event counts each explicit invalidation of a cache line in the L2 cache by cache maintenance operations that operate by a virtual address, or by external coherency operations. This Event does not count if either: * A cache refill invalidates a cache line, or * A cache Maintenance Operation (CMO), which invalidates a cache line specified by Set/Way, is executed on that CPU. CMOs that operate by Set/Way cannot be broadcast from one CPU to another." - }, - { - "ArchStdEvent": "L2D_CACHE_LMISS_RD", - "PublicDescription": "The Event counts cache line refills into the L2 unified cache from any memory Read operations that incurred additional latency. Counts the same as L2D_CACHE_REFILL_RD in this CPU" - }, - { - "ArchStdEvent": "L2D_CACHE_RW", - "PublicDescription": "The Event counts L2 cache demand accesses from any Load/Store operations. L2 cache is a unified cache for Data and Instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This CPU includes I-cache accesses in this counter as L2I equivalent Event was not implemented. This Event is the sum of the L2D_CACHE_RD and L2D_CACHE_WR Events." - }, - { - "ArchStdEvent": "L2D_CACHE_PRFM", - "PublicDescription": "The Event counts L2 D-cache accesses generated by software preload or prefetch Instructions with target = L1/L2/L3 cache. Note that a software preload or prefetch Instructions with (target = L1/L2/L3) that hits in L1D will not result in an L2 D-cache access. Therefore, such a software preload or prefetch Instructions will not be counted by this Event." - }, - { - "ArchStdEvent": "L2D_CACHE_MISS", - "PublicDescription": "The Event counts cache line misses in the L2 cache. L2 cache is a unified cache for Data and Instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. Counts same as L2D_CACHE_REFILL_RD in this CPU" - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL_PRFM", - "PublicDescription": "The Event counts refills due to accesses generated as a result of software preload or prefetch Instructions as counted by L2D_CACHE_PRFM. This CPU includes I-cache refills in this counter as L2I equivalent Event was not implemented." - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch access counted by L2D_CACHE_HWPRF that causes a refill of the L2 cache, or any L1 Data, or Instruction of this PE, from outside of those caches. This does not include prefetch requests pending waiting for a refill in LFB and a new demand request to the same cache line hitting the LFB entry. All such refills are counted as L2D_LFB_HIT_RWL1PRF_FHWPRF." - }, - { - "ArchStdEvent": "L2D_CACHE_REFILL_PRF", - "PublicDescription": "The Event counts each access to L2 Cache due to a prefetch Instruction, or hardware prefetch that causes a refill of the L2 or any Level 1, from outside of those caches." - }, - { - "ArchStdEvent": "L2D_CACHE_HWPRF", - "PublicDescription": "The Event counts the L2 D-cache access caused by L1 or L2 hardware prefetcher." - }, - { - "EventCode": "0x0108", - "EventName": "L2D_CACHE_IF_REFILL", - "PublicDescription": "L2 D-cache refill, Instruction fetch. The Event counts demand Instruction fetch that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x0109", - "EventName": "L2D_CACHE_TBW_REFILL", - "PublicDescription": "L2 D-cache refill, Page table walk. The Event counts demand translation table walk that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x010a", - "EventName": "L2D_CACHE_PF_REFILL", - "PublicDescription": "L2 D-cache refill, prefetch. The Event counts L1 or L2 hardware or software prefetch accesses that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x0201", - "EventName": "L2D_CACHE_BACKSNOOP_L1D_VIRT_ALIASING", - "PublicDescription": "The Event counts when the L2 D-cache sends an invalidating back-snoop to the L1 D for an access initiated by the L1 D, where the corresponding line is already present in the L1 D-cache. The L2 D-cache line tags the PE that refilled the line. It also retains specific bits of the VA to identify virtually aliased addresses. The L1 D request requiring a back-snoop can originate either from the same PE that refilled the L2 D line or from a different PE. In either case, this Event only counts those back snoop where the requested VA mismatch the VA stored in the L2 D tag. This Event is counted only by PE that initiated the original request necessitating a back-snoop. Note : The L1 D is VIPT, it identifies this access as a miss. Conversely, as L2 is PIPT, it identifies this as a hit. L2 D utilizes the back-snoop mechanism to refill L1 D with the snooped Data." - }, - { - "EventCode": "0x0179", - "EventName": "L2D_CACHE_HIT_RWL1PRF_FHWPRF", - "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by hardware prefetch.. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x020c", - "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRFM", - "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software prefetch. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x020e", - "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRF", - "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software or hardware prefetch. The Event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch or software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x010b", - "EventName": "L2D_LFB_HIT_RWL1PRF_FHWPRF", - "PublicDescription": "L2 line fill buffer demand Read, demand Write or L1 prefetch first hit, fetched by hardware prefetch. The Event counts each of the following access that hit the line-fill buffer when the same cache line is already being fetched due to an L2 hardware prefetcher. * Demand Read or Write * L1I-HWPRF * L1D-HWPRF * L1I PRFM * L1D PRFM These accesses hit a cache line that is currently being loaded into the L2 cache as a result of a hardware prefetcher to the same line. Consequently, this access does not initiate a new refill but waits for the completion of the previous refill. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x01b9", - "EventName": "L2D_CACHE_REFILL_L1PRF", - "PublicDescription": "L2 D-cache refill, L1 hardware or software prefetch. The Event counts each access counted by L2D_CACHE_L1PRF that causes a refill of the L2 cache or any L1 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x020a", - "EventName": "L2D_CACHE_REFILL_RWL1PRF", - "PublicDescription": "L2 D-cache refill, demand Read, demand Write or L1 hardware or software prefetch. The Event counts each access counted by L2D_CACHE_RWL1PRF that causes a refill of the L2 cache, or any L1 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x01b8", - "EventName": "L2D_CACHE_L1PRF", - "PublicDescription": "L2 D-cache access, L1 hardware or software prefetch. The Event counts L1 Hardware or software prefetch access to L2 D-cache." - }, - { - "EventCode": "0x0208", - "EventName": "L2D_CACHE_RWL1PRF", - "PublicDescription": "L2 D-cache access, demand Read, demand Write or L1 hardware or software prefetch. The Event counts each access to L2 D-cache due to the following: * Demand Read or Write. * L1 Hardware or software prefetch." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json deleted file mode 100644 index 661cbed4ee347..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json +++ /dev/null @@ -1,107 +0,0 @@ -[ - { - "ArchStdEvent": "LL_CACHE_RD", - "PublicDescription": "This is an alias to the Event L3D_CACHE_RD (0x00a0)." - }, - { - "ArchStdEvent": "LL_CACHE_MISS_RD", - "PublicDescription": "This is an alias to the Event L3D_CACHE_REFILL_RD (0x00a2)." - }, - { - "ArchStdEvent": "L3D_CACHE_ALLOCATE", - "PublicDescription": "The Event counts each memory Write operation that writes an entire line into the L3 Data without fetching Data from outside the L3 Data. These are allocations of cache lines in the L3 Data that are not refills counted by L3D_CACHE_REFILL. For example: A Write-back of an entire cache line from an L2 cache to the L3 D-cache. * A Write of an entire cache line from a coalescing Write buffer. * An operation such as DC ZVA. This counter does not count that writes an entire line to beyond level 3. Thus this counter does not count the streaming Writes to beyond L3 cache." - }, - { - "ArchStdEvent": "L3D_CACHE_REFILL", - "PublicDescription": "The Event counts each access counted by L3D_CACHE that causes a refill of the L3 Data,or any L1 Data, Instruction or L2 cache of this PE, from outside of those caches. This includes the refill due to hardware prefetch and software prefetch accesses. This Event is a sum of L3D_CACHE_MISS, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL_HWPRF Event. A refill includes any access that causes Data to be fetched from outside of the L1 to L3 caches, even if the Data is ultimately not allocated into the L3 D-cache." - }, - { - "ArchStdEvent": "L3D_CACHE", - "PublicDescription": "The Event counts each memory Read operation or memory Write operation that causes a cache access to the Level 3. This Event is a sum of the following Events: * L3D_CACHE_RD(0x00a0) * L3D_CACHE_ALLOCATE(0x0029) * L3D_CACHE_PRFM(0x8151) * L3D_CACHE_HWPRF(0x8156) * L2D_CACHE_WB(0x0018)" - }, - { - "ArchStdEvent": "L3D_CACHE_RD", - "PublicDescription": "The Event counts each Memory Read operation to L3 D-cache from Instruction Fetch, Load/Store, and MMU translation table accesses. This does not include HWPRF or PRFM accesses. This include L1 and L2 prefetcher accesses to L3 D-cache." - }, - { - "ArchStdEvent": "L3D_CACHE_REFILL_RD", - "PublicDescription": "The Event counts each access counted by both L3D_CACHE_RD and L3D_CACHE_REFILL. That is, every refill of the L3 cache counted by L3D_CACHE_REFILL that is caused by a Memory Read operation. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." - }, - { - "ArchStdEvent": "L3D_CACHE_LMISS_RD", - "PublicDescription": "The Event counts each memory Read operation to the L3 cache counted by L3D_CACHE that incurs additional latency because it returns Data from outside of the L1 to L3 caches. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." - }, - { - "ArchStdEvent": "L3D_CACHE_RW", - "PublicDescription": "The Event counts each access counted by L3D_CACHE that is due to a demand memory Read operation or demand memory Write operation. This Event is a sum of L3D_CACHE_RD(0x00a0), L3D_CACHE_ALLOCATE(0x0029) and L2D_CACHE_WB(0x0018). Note that this counter does not count that writes an entire line to beyond level 3. Thus this counter does not count the streaming Writes to beyond L3 cache." - }, - { - "ArchStdEvent": "L3D_CACHE_PRFM", - "PublicDescription": "The Event counts each access counted by L3D_CACHE that is due to a prefetch Instruction. This includes L3 Data accesses due to the L1, L2, or L3 prefetch Instruction." - }, - { - "ArchStdEvent": "L3D_CACHE_MISS", - "PublicDescription": "The Event counts each demand Read access counted by L3D_CACHE_RD that misses in the L1 to L3 Data, causing an access to outside of the L3 cache. The L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same Event in the hardware." - }, - { - "ArchStdEvent": "L3D_CACHE_REFILL_PRFM", - "PublicDescription": "The Event counts each access counted by L3D_CACHE_PRFM that causes a refill of the L3 cache, or any L1 or L2 Data, from outside of those caches." - }, - { - "ArchStdEvent": "L3D_CACHE_HWPRF", - "PublicDescription": "The Event counts each access to L3 cache that is due to a hardware prefetcher. This includes L3D accesses due to the Level-1 or Level-2 or Level-3 hardware prefetcher." - }, - { - "ArchStdEvent": "L3D_CACHE_REFILL_HWPRF", - "PublicDescription": "The Event counts each hardware prefetch counted by L3D_CACHE_HWPRF that causes a refill of the L3 Data or unified cache, or any L1 or L2 Data, Instruction, or unified cache of this PE, from outside of those caches." - }, - { - "ArchStdEvent": "L3D_CACHE_REFILL_PRF", - "PublicDescription": "The Event counts each access to L3 cache due to a prefetch Instruction, or hardware prefetch that causes a refill of the L3 Data, or any L1 or L2 Data, from outside of those caches." - }, - { - "EventCode": "0x01f7", - "EventName": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", - "PublicDescription": "L3 cache demand Read, demand Write , L1 prefetch L2 prefetch first hit, fetched by software or hardware prefetch. The Event counts each demand Read, demand Write , L1 hardware or software prefetch request and L2 hardware or software prefetch that hit an L3 D-cache line that was refilled into L3 D-cache in response to an L3 hardware prefetch or software prefetch. Only the first hit is counted. After this Event is generated for a cache line, the Event is not generated again for the same cache line while it remains in the cache." - }, - { - "EventCode": "0x01e9", - "EventName": "L3D_CACHE_REFILL_RWL1PRFL2PRF", - "PublicDescription": "L3 cache refill, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch. The Event counts each access counted by L3D_CACHE_RWL1PRFL2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x01e8", - "EventName": "L3D_CACHE_RWL1PRFL2PRF", - "PublicDescription": "L3 cache access, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch. The Event counts each access to L3 D-cache due to the following: * Demand Read or Write. * L1 Hardware or software prefetch. * L2 Hardware or software prefetch." - }, - { - "EventCode": "0x0225", - "EventName": "L3D_CACHE_REFILL_IF", - "PublicDescription": "L3 cache refill, Instruction fetch. The Event counts demand Instruction fetch that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x0226", - "EventName": "L3D_CACHE_REFILL_MM", - "PublicDescription": "L3 cache refill, translation table walk access. The Event counts demand translation table access that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x0227", - "EventName": "L3D_CACHE_REFILL_L1PRF", - "PublicDescription": "The Event counts each access counted by L3D_CACHE_L1PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x01f6", - "EventName": "L3D_CACHE_REFILL_L2PRF", - "PublicDescription": "The Event counts each access counted by L3D_CACHE_L2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." - }, - { - "EventCode": "0x022c", - "EventName": "L3D_CACHE_L1PRF", - "PublicDescription": "The Event counts the L3 D-cache access due to L1 hardware prefetch of software prefetch request. The L1 hardware prefetch or software prefetch request that miss the L1I, L1D and L2 D-cache are counted by this counter" - }, - { - "EventCode": "0x022d", - "EventName": "L3D_CACHE_L2PRF", - "PublicDescription": "The Event counts the L3 D-cache access due to L2 hardware prefetch of software prefetch request. The L2 hardware prefetch or software prefetch request that miss the L2 D-cache are counted by this counter" - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json deleted file mode 100644 index 64fced85a9881..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json +++ /dev/null @@ -1,46 +0,0 @@ -[ - { - "ArchStdEvent": "MEM_ACCESS", - "PublicDescription": "The Event counts memory accesses issued by the CPU Load/Store unit, where those accesses are issued due to Load or Store operations. This Event counts memory accesses no matter whether the Data is received from any level of cache hierarchy or external memory. If memory accesses are broken up into smaller transactions than what were specified in the Load or Store Instructions, then the Event counts those smaller memory transactions. Memory accesses generated by the following Instructions or activity are not counted: Instruction fetches, cache maintenance Instructions, Translation table walks or prefetches, Memory prefetch operations. This Event counts the sum of the MEM_ACCESS_RD and MEM_ACCESS_WR Events." - }, - { - "ArchStdEvent": "MEMORY_ERROR", - "PublicDescription": "The Event counts any detected correctable or uncorrectable physical memory errors (ECC or parity) in protected CPUs RAMs. On the Core, this Event counts errors in the caches (including Data and tag rams). Any detected memory error (from either a speculative and abandoned access, or an architecturally executed access) is counted. Note that errors are only detected when the actual protected memory is accessed by an operation." - }, - { - "ArchStdEvent": "REMOTE_ACCESS", - "PublicDescription": "Counter counts each external bus Read access that causes an access to a remote device. That is, a socket that does not contain the PE." - }, - { - "ArchStdEvent": "MEM_ACCESS_RD", - "PublicDescription": "The Event counts memory accesses issued by the CPU due to Load operations. The Event counts any memory Load access, no matter whether the Data is received from any level of cache hierarchy or external memory. The Event also counts atomic Load operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the Event counts those smaller transactions. The following Instructions are not counted: 1) Instruction fetches, 2) Cache maintenance Instructions, 3) Translation table walks or prefetches, 4) Memory prefetch operations. This Event is a subset of the MEM_ACCESS Event but the Event only counts memory-Read operations." - }, - { - "ArchStdEvent": "MEM_ACCESS_WR", - "PublicDescription": "The Event counts memory accesses issued by the CPU due to Store operations. The Event counts any memory Store access, no matter whether the Data is located in any level of cache or external memory. The Event also counts atomic Load and Store operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the Event counts those smaller transactions." - }, - { - "ArchStdEvent": "LDST_ALIGN_LAT", - "PublicDescription": "The Event counts the number of memory Read and Write accesses in a cycle that incurred additional latency, due to the alignment of the address and the size of Data being accessed, which results in Store crossing a single cache line. This Event is implemented as the sum of LD_ALIGN_LAT and ST_ALIGN_LAT on this CPU." - }, - { - "ArchStdEvent": "LD_ALIGN_LAT", - "PublicDescription": "The Event counts the number of memory Read accesses in a cycle that incurred additional latency, due to the alignment of the address and size of Data being accessed, which results in Load crossing a single cache line." - }, - { - "ArchStdEvent": "ST_ALIGN_LAT", - "PublicDescription": "The Event counts the number of memory Write access in a cycle that incurred additional latency, due to the alignment of the address and size of Data being accessed incurred additional latency." - }, - { - "ArchStdEvent": "INST_FETCH_PERCYC", - "PublicDescription": "The Event counts number of Instruction fetches outstanding per cycle, which will provide an average latency of Instruction fetch." - }, - { - "ArchStdEvent": "MEM_ACCESS_RD_PERCYC", - "PublicDescription": "The Event counts the number of outstanding Loads or memory Read accesses per cycle." - }, - { - "ArchStdEvent": "INST_FETCH", - "PublicDescription": "The Event counts Instruction memory accesses that the PE makes." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json deleted file mode 100644 index 18c2fd58ee9ec..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json +++ /dev/null @@ -1,722 +0,0 @@ -[ - { - "MetricName": "backend_bound", - "MetricExpr": "100 * (STALL_SLOT_BACKEND / CPU_SLOT)", - "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the backend of the processor.", - "ScaleUnit": "1percent of slots", - "MetricGroup": "TopdownL1" - }, - { - "MetricName": "backend_busy_bound", - "MetricExpr": "100 * (STALL_BACKEND_BUSY / STALL_BACKEND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to issue queues being full to accept operations for execution.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_cache_l1d_bound", - "MetricExpr": "100 * (STALL_BACKEND_L1D / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L1 D-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_cache_l2d_bound", - "MetricExpr": "100 * (STALL_BACKEND_MEM / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L2 D-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_core_bound", - "MetricExpr": "100 * (STALL_BACKEND_CPUBOUND / STALL_BACKEND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints not related to Instruction fetch latency issues caused by memory access components.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_core_rename_bound", - "MetricExpr": "100 * (STALL_BACKEND_RENAME / STALL_BACKEND_CPUBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend as the rename unit registers are unavailable.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_mem_bound", - "MetricExpr": "100 * (STALL_BACKEND_MEMBOUND / STALL_BACKEND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints related to memory access latency issues caused by memory access components.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_mem_cache_bound", - "MetricExpr": "100 * ((STALL_BACKEND_L1D + STALL_BACKEND_MEM) / STALL_BACKEND_MEMBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory latency issues caused by D-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_mem_store_bound", - "MetricExpr": "100 * (STALL_BACKEND_ST / STALL_BACKEND_MEMBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory Write pending caused by Stores stalled in the pre-commit stage.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_mem_tlb_bound", - "MetricExpr": "100 * (STALL_BACKEND_TLB / STALL_BACKEND_MEMBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by Data TLB misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Backend" - }, - { - "MetricName": "backend_stalled_cycles", - "MetricExpr": "100 * (STALL_BACKEND / CPU_CYCLES)", - "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the backend unit of the processor.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Cycle_Accounting" - }, - { - "MetricName": "bad_speculation", - "MetricExpr": "100 - (frontend_bound + retiring + backend_bound)", - "BriefDescription": "This metric is the percentage of total slots that executed operations and didn't retire due to a pipeline flush. This indicates cycles that were utilized but inefficiently.", - "ScaleUnit": "1percent of slots", - "MetricGroup": "TopdownL1" - }, - { - "MetricName": "bus_bandwidth", - "MetricExpr": "BUS_ACCESS * 32 / duration_time ", - "BriefDescription": "This metrics measures the bus-bandwidth of the Data transferred between this PE's L2 with unCore in the system", - "ScaleUnit": "1Bytes/sec" - }, - { - "MetricName": "barrier_percentage", - "MetricExpr": "100 * ((ISB_SPEC + DSB_SPEC + DMB_SPEC) / INST_SPEC)", - "BriefDescription": "This metric measures Instruction and Data barrier operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "branch_direct_ratio", - "MetricExpr": "BR_IMMED_RETIRED / BR_RETIRED", - "BriefDescription": "This metric measures the ratio of direct branches retired to the total number of Branches architecturally executed.", - "ScaleUnit": "1per branch", - "MetricGroup": "Branch_Effectiveness" - }, - { - "MetricName": "branch_indirect_ratio", - "MetricExpr": "BR_IND_RETIRED / BR_RETIRED", - "BriefDescription": "This metric measures the ratio of indirect Branches retired, including function returns, to the total number of Branches architecturally executed.", - "ScaleUnit": "1per branch", - "MetricGroup": "Branch_Effectiveness" - }, - { - "MetricName": "branch_misprediction_ratio", - "MetricExpr": "BR_MIS_PRED_RETIRED / BR_RETIRED", - "BriefDescription": "This metric measures the ratio of Branches mispredicted to the total number of Branches architecturally executed. This gives an indication of the effectiveness of the Branch prediction unit.", - "ScaleUnit": "1per branch", - "MetricGroup": "Miss_Ratio;Branch_Effectiveness" - }, - { - "MetricName": "branch_mpki", - "MetricExpr": "1000 * (BR_MIS_PRED_RETIRED / INST_RETIRED)", - "BriefDescription": "This metric measures the number of Branch mispredictions per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;Branch_Effectiveness" - }, - { - "MetricName": "branch_percentage", - "MetricExpr": "100 * ((BR_IMMED_SPEC + BR_INDIRECT_SPEC) / INST_SPEC)", - "BriefDescription": "This metric measures Branch operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "branch_return_ratio", - "MetricExpr": "BR_RETURN_RETIRED / BR_RETIRED", - "BriefDescription": "This metric measures the ratio of Branches retired that are function returns to the total number of Branches architecturally executed.", - "ScaleUnit": "1per branch", - "MetricGroup": "Branch_Effectiveness" - }, - { - "MetricName": "cpu_cycles_fraction_in_st_mode", - "MetricExpr": "((CPU_SLOT/CPU_CYCLES) - 5) / 5", - "BriefDescription": "This metric counts fraction of the CPU cycles spent in ST mode during program execution.", - "ScaleUnit": "1fraction of cycles", - "MetricGroup": "SMT" - }, - { - "MetricName": "cpu_cycles_in_smt_mode", - "MetricExpr": "(1 - cpu_cycles_fraction_in_st_mode) * CPU_CYCLES", - "BriefDescription": "This metric counts CPU cycles in SMT mode during program execution.", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "SMT" - }, - { - "MetricName": "cpu_cycles_in_st_mode", - "MetricExpr": "cpu_cycles_fraction_in_st_mode * CPU_CYCLES", - "BriefDescription": "This metric counts CPU cycles in ST mode during program execution.", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "SMT" - }, - { - "MetricName": "crypto_percentage", - "MetricExpr": "100 * (CRYPTO_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures crypto operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "dtlb_mpki", - "MetricExpr": "1000 * (DTLB_WALK / INST_RETIRED)", - "BriefDescription": "This metric measures the number of Data TLB Walks per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;DTLB_Effectiveness" - }, - { - "MetricName": "dtlb_walk_average_latency", - "MetricExpr": "DTLB_WALK_PERCYC / DTLB_WALK", - "BriefDescription": "This metric measures the average latency of Data TLB walks in CPU cycles", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "Average_Latency" - }, - { - "MetricName": "dtlb_walk_ratio", - "MetricExpr": "DTLB_WALK / L1D_TLB", - "BriefDescription": "This metric measures the ratio of Data TLB Walks to the total number of Data TLB accesses. This gives an indication of the effectiveness of the Data TLB accesses.", - "ScaleUnit": "1per TLB access", - "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" - }, - { - "MetricName": "fp_ops_per_cycle", - "MetricExpr": "(FP_SCALE_OPS_SPEC + FP_FIXED_OPS_SPEC) / CPU_CYCLES", - "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by any Instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", - "ScaleUnit": "1operations per cycle", - "MetricGroup": "FP_Arithmetic_Intensity" - }, - { - "MetricName": "fp16_percentage", - "MetricExpr": "100 * (FP_HP_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures half-precision floating point operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "FP_Precision_Mix" - }, - { - "MetricName": "fp32_percentage", - "MetricExpr": "100 * (FP_SP_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures single-precision floating point operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "FP_Precision_Mix" - }, - { - "MetricName": "fp64_percentage", - "MetricExpr": "100 * (FP_DP_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures double-precision floating point operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "FP_Precision_Mix" - }, - { - "MetricName": "frontend_bound", - "MetricExpr": "100 * (STALL_SLOT_FRONTEND_WITHOUT_MISPRED / CPU_SLOT)", - "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the frontend of the processor.", - "ScaleUnit": "1percent of slots", - "MetricGroup": "TopdownL1" - }, - { - "MetricName": "frontend_cache_l1i_bound", - "MetricExpr": "100 * (STALL_FRONTEND_L1I / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L1 I-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_cache_l2i_bound", - "MetricExpr": "100 * (STALL_FRONTEND_MEM / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L2 I-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_core_bound", - "MetricExpr": "100 * (STALL_FRONTEND_CPUBOUND / STALL_FRONTEND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints not related to Instruction fetch latency issues caused by memory access components.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_core_flow_bound", - "MetricExpr": "100 * (STALL_FRONTEND_FLOW / STALL_FRONTEND_CPUBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the decode unit is awaiting input from the Branch prediction unit.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_core_flush_bound", - "MetricExpr": "100 * (STALL_FRONTEND_FLUSH / STALL_FRONTEND_CPUBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the processor is recovering from a pipeline flush caused by bad speculation or other machine resteers.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_mem_bound", - "MetricExpr": "100 * (STALL_FRONTEND_MEMBOUND / STALL_FRONTEND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints related to the Instruction fetch latency issues caused by memory access components.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_mem_cache_bound", - "MetricExpr": "100 * ((STALL_FRONTEND_L1I + STALL_FRONTEND_MEM) / STALL_FRONTEND_MEMBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to Instruction fetch latency issues caused by I-cache misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_mem_tlb_bound", - "MetricExpr": "100 * (STALL_FRONTEND_TLB / STALL_FRONTEND_MEMBOUND)", - "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to Instruction fetch latency issues caused by Instruction TLB misses.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Topdown_Frontend" - }, - { - "MetricName": "frontend_stalled_cycles", - "MetricExpr": "100 * (STALL_FRONTEND / CPU_CYCLES)", - "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the frontend unit of the processor.", - "ScaleUnit": "1percent of cycles", - "MetricGroup": "Cycle_Accounting" - }, - { - "MetricName": "instruction_fetch_average_latency", - "MetricExpr": "INST_FETCH_PERCYC / INST_FETCH", - "BriefDescription": "This metric measures the average latency of Instruction fetches in CPU cycles", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "Average_Latency" - }, - { - "MetricName": "integer_dp_percentage", - "MetricExpr": "100 * (DP_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures scalar integer operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "ipc", - "MetricExpr": "INST_RETIRED / CPU_CYCLES", - "BriefDescription": "This metric measures the number of Instructions retired per cycle.", - "ScaleUnit": "1per cycle", - "MetricGroup": "General" - }, - { - "MetricName": "itlb_mpki", - "MetricExpr": "1000 * (ITLB_WALK / INST_RETIRED)", - "BriefDescription": "This metric measures the number of Instruction TLB Walks per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;ITLB_Effectiveness" - }, - { - "MetricName": "itlb_walk_average_latency", - "MetricExpr": "ITLB_WALK_PERCYC / ITLB_WALK", - "BriefDescription": "This metric measures the average latency of Instruction TLB walks in CPU cycles", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "Average_Latency" - }, - { - "MetricName": "itlb_walk_ratio", - "MetricExpr": "ITLB_WALK / L1I_TLB", - "BriefDescription": "This metric measures the ratio of Instruction TLB Walks to the total number of Instruction TLB accesses. This gives an indication of the effectiveness of the Instruction TLB accesses.", - "ScaleUnit": "1per TLB access", - "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" - }, - { - "MetricName": "l1d_cache_miss_ratio", - "MetricExpr": "L1D_CACHE_REFILL / L1D_CACHE", - "BriefDescription": "This metric measures the ratio of L1 D-cache accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache.", - "ScaleUnit": "1per cache access", - "MetricGroup": "Miss_Ratio;L1D_Cache_Effectiveness" - }, - { - "MetricName": "l1d_cache_mpki", - "MetricExpr": "1000 * (L1D_CACHE_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L1 D-cache accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;L1D_Cache_Effectiveness" - }, - { - "MetricName": "l1d_cache_rw_miss_ratio", - "MetricExpr": "l1d_demand_misses / l1d_demand_accesses", - "BriefDescription": "This metric measures the ratio of L1 D-cache Read accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache for demand Load or Store traffic.", - "ScaleUnit": "1per cache access", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_demand_accesses", - "MetricExpr": "L1D_CACHE_RW", - "BriefDescription": "This metric measures the count of L1 D-cache accesses incurred on Load or Store by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_demand_misses", - "MetricExpr": "L1D_CACHE_REFILL_RW", - "BriefDescription": "This metric measures the count of L1 D-cache misses incurred on a Load or Store by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_prf_accuracy", - "MetricExpr": "100 * (l1d_useful_prf / l1d_refilled_prf)", - "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", - "ScaleUnit": "1percent of prefetch", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_prf_coverage", - "MetricExpr": "100 * (l1d_useful_prf / (l1d_demand_misses + l1d_refilled_prf))", - "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", - "ScaleUnit": "1percent of cache access", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_refilled_prf", - "MetricExpr": "L1D_CACHE_REFILL_HWPRF + L1D_CACHE_REFILL_PRFM + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", - "BriefDescription": "This metric measures the count of cache lines refilled by L1 Data prefetcher (hardware prefetches or software preload) into L1 D-cache.", - "ScaleUnit": "1count", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1d_tlb_miss_ratio", - "MetricExpr": "L1D_TLB_REFILL / L1D_TLB", - "BriefDescription": "This metric measures the ratio of L1 Data TLB accesses missed to the total number of L1 Data TLB accesses. This gives an indication of the effectiveness of the L1 Data TLB.", - "ScaleUnit": "1per TLB access", - "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" - }, - { - "MetricName": "l1d_tlb_mpki", - "MetricExpr": "1000 * (L1D_TLB_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L1 Data TLB accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;DTLB_Effectiveness" - }, - { - "MetricName": "l1d_useful_prf", - "MetricExpr": "L1D_CACHE_HIT_RW_FPRF + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", - "BriefDescription": "This metric measures the count of cache lines refilled by L1 Data prefetcher (hardware prefetches or software preload) into L1 D-cache which are further used by Load or Store from the Instruction stream of the program.", - "ScaleUnit": "1count", - "MetricGroup": "L1I_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_cache_miss_ratio", - "MetricExpr": "L1I_CACHE_REFILL / L1I_CACHE", - "BriefDescription": "This metric measures the ratio of L1 I-cache accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache.", - "ScaleUnit": "1per cache access", - "MetricGroup": "Miss_Ratio;L1I_Cache_Effectiveness" - }, - { - "MetricName": "l1i_cache_mpki", - "MetricExpr": "1000 * (L1I_CACHE_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L1 I-cache accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;L1I_Cache_Effectiveness" - }, - { - "MetricName": "l1i_cache_rd_miss_ratio", - "MetricExpr": "l1i_demand_misses / l1i_demand_accesses", - "BriefDescription": "This metric measures the ratio of L1 I-cache Read accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache for demand Instruction fetch traffic. Note that cache accesses in this cache is demand Instruction fetch.", - "ScaleUnit": "1per cache access", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_demand_accesses", - "MetricExpr": "L1I_CACHE_RD", - "BriefDescription": "This metric measures the count of L1 I-cache accesses caused by an Instruction fetch by the Instructions stream of the program", - "ScaleUnit": "1Count", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_demand_misses", - "MetricExpr": "L1I_CACHE_REFILL_RD", - "BriefDescription": "This metric measures the count of L1 I-cache misses caused by an Instruction fetch by the Instructions stream of the program", - "ScaleUnit": "1Count", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_prf_accuracy", - "MetricExpr": "100 * (l1i_useful_prf / l1i_refilled_prf)", - "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", - "ScaleUnit": "1percent of prefetch", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_prf_coverage", - "MetricExpr": "100 * (l1i_useful_prf / (l1i_demand_misses + l1i_refilled_prf))", - "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", - "ScaleUnit": "1percent of cache access", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_refilled_prf", - "MetricExpr": "L1I_CACHE_REFILL_HWPRF + L1I_CACHE_REFILL_PRFM", - "BriefDescription": "This metric measures the count of cache lines refilled by L1 Instruction prefetcher (hardware prefetches or software preload) into L1 I-cache.", - "ScaleUnit": "1count", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l1i_tlb_miss_ratio", - "MetricExpr": "L1I_TLB_REFILL / L1I_TLB", - "BriefDescription": "This metric measures the ratio of L1 Instruction TLB accesses missed to the total number of L1 Instruction TLB accesses. This gives an indication of the effectiveness of the L1 Instruction TLB.", - "ScaleUnit": "1per TLB access", - "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" - }, - { - "MetricName": "l1i_tlb_mpki", - "MetricExpr": "1000 * (L1I_TLB_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L1 Instruction TLB accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;ITLB_Effectiveness" - }, - { - "MetricName": "l1i_useful_prf", - "MetricExpr": "L1I_CACHE_HIT_RD_FPRF", - "BriefDescription": "This metric measures the count of cache lines refilled by L1 Instruction prefetcher (hardware prefetches or software preload) into L1 I-cache which are further used by Instruction stream of the program.", - "ScaleUnit": "1count", - "MetricGroup": "L1D_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2_cache_miss_ratio", - "MetricExpr": "L2D_CACHE_REFILL / L2D_CACHE", - "BriefDescription": "This metric measures the ratio of L2 cache accesses missed to the total number of L2 cache accesses. This gives an indication of the effectiveness of the L2 cache, which is a unified cache that stores both Data and Instruction.", - "ScaleUnit": "1per cache access", - "MetricGroup": "Miss_Ratio;L2_Cache_Effectiveness" - }, - { - "MetricName": "l2_cache_mpki", - "MetricExpr": "1000 * (L2D_CACHE_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L2 unified cache accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;L2_Cache_Effectiveness" - }, - { - "MetricName": "l2_tlb_miss_ratio", - "MetricExpr": "L2D_TLB_REFILL / L2D_TLB", - "BriefDescription": "This metric measures the ratio of L2 unified TLB accesses missed to the total number of L2 unified TLB accesses.", - "ScaleUnit": "1per TLB access", - "MetricGroup": "Miss_Ratio;ITLB_Effectiveness;DTLB_Effectiveness" - }, - { - "MetricName": "l2_tlb_mpki", - "MetricExpr": "1000 * (L2D_TLB_REFILL / INST_RETIRED)", - "BriefDescription": "This metric measures the number of L2 unified TLB accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;ITLB_Effectiveness;DTLB_Effectiveness" - }, - { - "MetricName": "l2d_cache_rwl1prf_miss_ratio", - "MetricExpr": "l2d_demand_misses / l2d_demand_accesses", - "BriefDescription": "This metric measures the ratio of L2 D-cache Read accesses missed to the total number of L2 D-cache accesses.", - "ScaleUnit": "1per cache access", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_demand_accesses", - "MetricExpr": "L2D_CACHE_RD + L2D_CACHE_WR + L2D_CACHE_L1PRF", - "BriefDescription": "This metric measures the count of L2 D-cache accesses incurred on an Instruction Fetch, Load, Store, or L1 prefetcher accesses by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_demand_misses", - "MetricExpr": "L2D_CACHE_REFILL_RD + L2D_CACHE_REFILL_WR + L2D_CACHE_REFILL_L1PRF", - "BriefDescription": "This metric measures the count of L2 D-cache misses incurred on an Instruction Fetch, Load, Store, or L1 prefetcher accesses by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_prf_accuracy", - "MetricExpr": "100 * (l2d_useful_prf / l2d_refilled_prf)", - "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", - "ScaleUnit": "1percent of prefetch", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_prf_coverage", - "MetricExpr": "100 * (l2d_useful_prf / (l2d_demand_misses + l2d_refilled_prf))", - "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", - "ScaleUnit": "1percent of cache access", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_refilled_prf", - "MetricExpr": "(L2D_CACHE_REFILL_PRF - L2D_CACHE_REFILL_L1PRF) + L2D_LFB_HIT_RWL1PRF_FHWPRF", - "BriefDescription": "This metric measures the count of cache lines refilled by L2 Data prefetcher (hardware prefetches or software preload) into L2 D-cache.", - "ScaleUnit": "1count", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l2d_useful_prf", - "MetricExpr": "L2D_CACHE_HIT_RWL1PRF_FPRF + L2D_LFB_HIT_RWL1PRF_FHWPRF", - "BriefDescription": "This metric measures the count of cache lines refilled by L2 Data prefetcher (hardware prefetches or software preload) into L2 D-cache which are further used by Instruction Fetch, Load, Store, or L1 prefetcher accesses from the Instruction stream of the program.", - "ScaleUnit": "1count", - "MetricGroup": "L2_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_cache_rwl1prfl2prf_miss_ratio", - "MetricExpr": "l3d_demand_misses / l3d_demand_accesses", - "BriefDescription": "This metric measures the ratio of L3 D-cache Read accesses missed to the total number of L3 D-cache accesses. This gives an indication of the effectiveness of the L2 D-cache for demand Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses traffic.", - "ScaleUnit": "1per cache access", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_demand_accesses", - "MetricExpr": "L3D_CACHE_RWL1PRFL2PRF", - "BriefDescription": "This metric measures the count of L3 D-cache accesses incurred on an Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_demand_misses", - "MetricExpr": "L3D_CACHE_REFILL_RWL1PRFL2PRF", - "BriefDescription": "This metric measures the count of L3 D-cache misses incurred on an Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the Instructions stream of the program", - "ScaleUnit": "1count", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_prf_accuracy", - "MetricExpr": "100 * (l3d_useful_prf / l3d_refilled_prf)", - "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the Instruction stream", - "ScaleUnit": "1percent of prefetch", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_prf_coverage", - "MetricExpr": "100 * (l3d_useful_prf / (l3d_demand_misses + l3d_refilled_prf))", - "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", - "ScaleUnit": "1percent of cache access", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_refilled_prf", - "MetricExpr": "L3D_CACHE_REFILL_HWPRF + L3D_CACHE_REFILL_PRFM - L3D_CACHE_REFILL_L1PRF - L3D_CACHE_REFILL_L2PRF", - "BriefDescription": "This metric measures the count of cache lines refilled by L3 Data prefetcher (hardware prefetches or software preload) into L3 D-cache.", - "ScaleUnit": "1count", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "l3d_useful_prf", - "MetricExpr": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", - "BriefDescription": "This metric measures the count of cache lines refilled by L3 Data prefetcher (hardware prefetches or software preload) into L3 D-cache which are further used by Instruction Fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses from the Instruction stream of the program.", - "ScaleUnit": "1count", - "MetricGroup": "L3_Prefetcher_Effectiveness" - }, - { - "MetricName": "ll_cache_read_hit_ratio", - "MetricExpr": "(LL_CACHE_RD - LL_CACHE_MISS_RD) / LL_CACHE_RD", - "BriefDescription": "This metric measures the ratio of last level cache Read accesses hit in the cache to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either Data memory access or Instruction fetch as this is a system level cache.", - "ScaleUnit": "1per cache access", - "MetricGroup": "LL_Cache_Effectiveness" - }, - { - "MetricName": "ll_cache_read_miss_ratio", - "MetricExpr": "LL_CACHE_MISS_RD / LL_CACHE_RD", - "BriefDescription": "This metric measures the ratio of last level cache Read accesses missed to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either Data memory access or Instruction fetch as this is a system level cache.", - "ScaleUnit": "1per cache access", - "MetricGroup": "Miss_Ratio;LL_Cache_Effectiveness" - }, - { - "MetricName": "ll_cache_read_mpki", - "MetricExpr": "1000 * (LL_CACHE_MISS_RD / INST_RETIRED)", - "BriefDescription": "This metric measures the number of last level cache Read accesses missed per thousand Instructions executed.", - "ScaleUnit": "1MPKI", - "MetricGroup": "MPKI;LL_Cache_Effectiveness" - }, - { - "MetricName": "load_average_latency", - "MetricExpr": "MEM_ACCESS_RD_PERCYC / MEM_ACCESS", - "BriefDescription": "This metric measures the average latency of Load operations in CPU cycles", - "ScaleUnit": "1CPU cycles", - "MetricGroup": "Average_Latency" - }, - { - "MetricName": "load_percentage", - "MetricExpr": "100 * (LD_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures Load operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "nonsve_fp_ops_per_cycle", - "MetricExpr": "FP_FIXED_OPS_SPEC / CPU_CYCLES", - "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by an Instruction that is not an SVE Instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", - "ScaleUnit": "1operations per cycle", - "MetricGroup": "FP_Arithmetic_Intensity" - }, - { - "MetricName": "retiring", - "MetricExpr": "100 * ((OP_RETIRED/OP_SPEC) * (1 - (STALL_SLOT/CPU_SLOT)))", - "BriefDescription": "This metric is the percentage of total slots that retired operations, which indicates cycles that were utilized efficiently.", - "ScaleUnit": "1percent of slots", - "MetricGroup": "TopdownL1" - }, - { - "MetricName": "scalar_fp_percentage", - "MetricExpr": "100 * (VFP_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures scalar floating point operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "simd_percentage", - "MetricExpr": "100 * (ASE_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures advanced SIMD operations as a percentage of total operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "store_percentage", - "MetricExpr": "100 * (ST_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures Store operations as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "sve_all_percentage", - "MetricExpr": "100 * (SVE_INST_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures scalable vector operations, including Loads and Stores, as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "Operation_Mix" - }, - { - "MetricName": "sve_fp_ops_per_cycle", - "MetricExpr": "FP_SCALE_OPS_SPEC / CPU_CYCLES", - "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by SVE Instructions. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", - "ScaleUnit": "1operations per cycle", - "MetricGroup": "FP_Arithmetic_Intensity" - }, - { - "MetricName": "sve_predicate_empty_percentage", - "MetricExpr": "100 * (SVE_PRED_EMPTY_SPEC / SVE_PRED_SPEC)", - "BriefDescription": "This metric measures scalable vector operations with no active predicates as a percentage of sve predicated operations speculatively executed.", - "ScaleUnit": "1percent of SVE predicated operations", - "MetricGroup": "SVE_Effectiveness" - }, - { - "MetricName": "sve_predicate_full_percentage", - "MetricExpr": "100 * (SVE_PRED_FULL_SPEC / SVE_PRED_SPEC)", - "BriefDescription": "This metric measures scalable vector operations with all active predicates as a percentage of sve predicated operations speculatively executed.", - "ScaleUnit": "1percent of SVE predicated operations", - "MetricGroup": "SVE_Effectiveness" - }, - { - "MetricName": "sve_predicate_partial_percentage", - "MetricExpr": "100 * (SVE_PRED_PARTIAL_SPEC / SVE_PRED_SPEC)", - "BriefDescription": "This metric measures scalable vector operations with at least one active predicates as a percentage of sve predicated operations speculatively executed.", - "ScaleUnit": "1percent of SVE predicated operations", - "MetricGroup": "SVE_Effectiveness" - }, - { - "MetricName": "sve_predicate_percentage", - "MetricExpr": "100 * (SVE_PRED_SPEC / INST_SPEC)", - "BriefDescription": "This metric measures scalable vector operations with predicates as a percentage of operations speculatively executed.", - "ScaleUnit": "1percent of operations", - "MetricGroup": "SVE_Effectiveness" - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json deleted file mode 100644 index 0d657789ee482..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json +++ /dev/null @@ -1,646 +0,0 @@ -[ - { - "ArchStdEvent": "SW_INCR", - "PublicDescription": "The Event counts software writes to the PMSWINC_EL0 (software PMU increment) register. The PMSWINC_EL0 register is a manually updated counter for use by application software. This Event could be used to measure any user program Event, such as accesses to a particular Data structure (by writing to the PMSWINC_EL0 register each time the Data structure is accessed). To use the PMSWINC_EL0 register and Event, developers must insert Instructions that write to the PMSWINC_EL0 register into the source code. Since the SW_INCR Event records Writes to the PMSWINC_EL0 register, there is no need to do a Read/Increment/Write sequence to the PMSWINC_EL0 register." - }, - { - "ArchStdEvent": "CHAIN", - "PublicDescription": "For odd-numbered counters, this Event increments the count by one for each overflow of the preceding even-numbered counter. For even-numbered counters, there is no increment. This Event is used when the even/odd pairs of registers are used as a single counter." - }, - { - "ArchStdEvent": "TRB_WRAP", - "PublicDescription": "The Event is generated each time the trace buffer current Write pointer is wrapped to the trace buffer base pointer." - }, - { - "ArchStdEvent": "TRCEXTOUT0", - "PublicDescription": "Trace unit external output 0." - }, - { - "ArchStdEvent": "TRCEXTOUT1", - "PublicDescription": "Trace unit external output 1." - }, - { - "ArchStdEvent": "TRCEXTOUT2", - "PublicDescription": "Trace unit external output 2." - }, - { - "ArchStdEvent": "TRCEXTOUT3", - "PublicDescription": "Trace unit external output 3." - }, - { - "ArchStdEvent": "CTI_TRIGOUT4", - "PublicDescription": "Cross-trigger Interface output trigger 4." - }, - { - "ArchStdEvent": "CTI_TRIGOUT5", - "PublicDescription": "Cross-trigger Interface output trigger 5." - }, - { - "ArchStdEvent": "CTI_TRIGOUT6", - "PublicDescription": "Cross-trigger Interface output trigger 6." - }, - { - "ArchStdEvent": "CTI_TRIGOUT7", - "PublicDescription": "Cross-trigger Interface output trigger 7." - }, - { - "EventCode": "0x00e1", - "EventName": "L1I_PRFM_REQ_DROP", - "PublicDescription": "L1 I-cache software prefetch dropped." - }, - { - "EventCode": "0x0100", - "EventName": "L1_PF_REFILL", - "PublicDescription": "L1 prefetch requests, refilled to L1 cache." - }, - { - "EventCode": "0x0120", - "EventName": "FLUSH", - "PublicDescription": "The Event counts both the CT flush and BX flush. The BR_MIS_PRED counts the BX flushes. So the FLUSH-BR_MIS_PRED gives the CT flushes." - }, - { - "EventCode": "0x0121", - "EventName": "FLUSH_MEM", - "PublicDescription": "Flushes due to memory hazards. This only includes CT flushes." - }, - { - "EventCode": "0x0122", - "EventName": "FLUSH_BAD_BRANCH", - "PublicDescription": "Flushes due to bad predicted Branch. This only includes CT flushes." - }, - { - "EventCode": "0x0124", - "EventName": "FLUSH_ISB", - "PublicDescription": "Flushes due to ISB or similar side-effects. This only includes CT flushes." - }, - { - "EventCode": "0x0125", - "EventName": "FLUSH_OTHER", - "PublicDescription": "Flushes due to other hazards. This only includes CT flushes." - }, - { - "EventCode": "0x0126", - "EventName": "STORE_STREAM", - "PublicDescription": "Stored lines in streaming no-Write-allocate mode." - }, - { - "EventCode": "0x0127", - "EventName": "NUKE_RAR", - "PublicDescription": "Load/Store nuke due to Read-after-Read ordering hazard." - }, - { - "EventCode": "0x0128", - "EventName": "NUKE_RAW", - "PublicDescription": "Load/Store nuke due to Read-after-Write ordering hazard." - }, - { - "EventCode": "0x0129", - "EventName": "L1_PF_GEN_PAGE", - "PublicDescription": "Load/Store prefetch to L1 generated, Page mode." - }, - { - "EventCode": "0x012a", - "EventName": "L1_PF_GEN_STRIDE", - "PublicDescription": "Load/Store prefetch to L1 generated, stride mode." - }, - { - "EventCode": "0x012b", - "EventName": "L2_PF_GEN_LD", - "PublicDescription": "Load prefetch to L2 generated." - }, - { - "EventCode": "0x012d", - "EventName": "LS_PF_TRAIN_TABLE_ALLOC", - "PublicDescription": "LS prefetch train table entry allocated." - }, - { - "EventCode": "0x0130", - "EventName": "LS_PF_GEN_TABLE_ALLOC", - "PublicDescription": "The Event counts the number of cycles with at least one table allocation, for L2 hardware prefetches (including the SW PRFM that are converted into hardware prefetches due to D-TLB miss). LS prefetch gen table allocation (for L2 prefetches)." - }, - { - "EventCode": "0x0131", - "EventName": "LS_PF_GEN_TABLE_ALLOC_PF_PEND", - "PublicDescription": "The Event counts the number of cycles in which at least one hardware prefetch is dropped due to the inability to identify a victim when the generation table is full. The hardware prefetch considered here includes the software PRFM that is converted into hardware prefetches due to D-TLB miss." - }, - { - "EventCode": "0x0132", - "EventName": "TBW", - "PublicDescription": "Tablewalks." - }, - { - "EventCode": "0x0134", - "EventName": "S1L2_HIT", - "PublicDescription": "Translation cache hit on S1L2 walk cache entry." - }, - { - "EventCode": "0x0135", - "EventName": "S1L1_HIT", - "PublicDescription": "Translation cache hit on S1L1 walk cache entry." - }, - { - "EventCode": "0x0136", - "EventName": "S1L0_HIT", - "PublicDescription": "Translation cache hit on S1L0 walk cache entry." - }, - { - "EventCode": "0x0137", - "EventName": "S2L2_HIT", - "PublicDescription": "Translation cache hit for S2L2 IPA walk cache entry." - }, - { - "EventCode": "0x0138", - "EventName": "IPA_REQ", - "PublicDescription": "Translation cache lookups for IPA to PA entries." - }, - { - "EventCode": "0x0139", - "EventName": "IPA_REFILL", - "PublicDescription": "Translation cache refills for IPA to PA entries." - }, - { - "EventCode": "0x013a", - "EventName": "S1_FLT", - "PublicDescription": "Stage1 tablewalk fault." - }, - { - "EventCode": "0x013b", - "EventName": "S2_FLT", - "PublicDescription": "Stage2 tablewalk fault." - }, - { - "EventCode": "0x013c", - "EventName": "COLT_REFILL", - "PublicDescription": "Aggregated page refill." - }, - { - "EventCode": "0x0145", - "EventName": "L1_PF_HIT", - "PublicDescription": "L1 prefetch requests, hitting in L1 cache." - }, - { - "EventCode": "0x0146", - "EventName": "L1_PF", - "PublicDescription": "L1 prefetch requests." - }, - { - "EventCode": "0x0147", - "EventName": "CACHE_LS_REFILL", - "PublicDescription": "L2 D-cache refill, Load/Store." - }, - { - "EventCode": "0x0148", - "EventName": "CACHE_PF", - "PublicDescription": "L2 prefetch requests." - }, - { - "EventCode": "0x0149", - "EventName": "CACHE_PF_HIT", - "PublicDescription": "L2 prefetch requests, hitting in L2 cache." - }, - { - "EventCode": "0x0150", - "EventName": "UNUSED_PF", - "PublicDescription": "L2 unused prefetch." - }, - { - "EventCode": "0x0151", - "EventName": "PFT_SENT", - "PublicDescription": "L2 prefetch TGT sent. Note that PFT_SENT != PFT_USEFUL + PFT_DROP. There may be PFT_SENT for which the accesses resulted in a SLC hit." - }, - { - "EventCode": "0x0152", - "EventName": "PFT_USEFUL", - "PublicDescription": "L2 prefetch TGT useful." - }, - { - "EventCode": "0x0153", - "EventName": "PFT_DROP", - "PublicDescription": "L2 prefetch TGT dropped." - }, - { - "EventCode": "0x0162", - "EventName": "LRQ_FULL", - "PublicDescription": "The Event counts the number of cycles the LRQ is full." - }, - { - "EventCode": "0x0163", - "EventName": "FETCH_FQ_EMPTY", - "PublicDescription": "Fetch Queue empty cycles." - }, - { - "EventCode": "0x0164", - "EventName": "FPG2", - "PublicDescription": "Forward progress guarantee. Medium range livelock triggered." - }, - { - "EventCode": "0x0165", - "EventName": "FPG", - "PublicDescription": "Forward progress guarantee. Tofu global livelock buster is triggered." - }, - { - "EventCode": "0x0172", - "EventName": "DEADBLOCK", - "PublicDescription": "Write-back evictions converted to Dataless EVICT. The victim line is deemed deadblock if the likeliness of a reuse is low. The Core uses Dataless evict to evict a deadblock; And it uses a evict with Data to evict an L2 line that is not a deadblock." - }, - { - "EventCode": "0x0173", - "EventName": "PF_PRQ_ALLOC_PF_PEND", - "PublicDescription": "L1 prefetch prq allocation (replacing pending)." - }, - { - "EventCode": "0x0178", - "EventName": "FETCH_ICACHE_INSTR", - "PublicDescription": "Instructions fetched from I-cache." - }, - { - "EventCode": "0x017b", - "EventName": "NEAR_CAS", - "PublicDescription": "Near atomics: compare and swap." - }, - { - "EventCode": "0x017c", - "EventName": "NEAR_CAS_PASS", - "PublicDescription": "Near atomics: compare and swap pass." - }, - { - "EventCode": "0x017d", - "EventName": "FAR_CAS", - "PublicDescription": "Far atomics: compare and swap." - }, - { - "EventCode": "0x0186", - "EventName": "L2_BTB_RELOAD_MAIN_BTB", - "PublicDescription": "Number of completed L1 BTB update initiated by L2 BTB hit which swap Branch information between L1 BTB and L2 BTB." - }, - { - "EventCode": "0x0190", - "EventName": "PF_MODE_0_CYCLES", - "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most aggressive mode." - }, - { - "EventCode": "0x0191", - "EventName": "PF_MODE_1_CYCLES", - "PublicDescription": "Number of cycles in which the hardware prefetcher is in the more aggressive mode." - }, - { - "EventCode": "0x0192", - "EventName": "PF_MODE_2_CYCLES", - "PublicDescription": "Number of cycles in which the hardware prefetcher is in the less aggressive mode." - }, - { - "EventCode": "0x0193", - "EventName": "PF_MODE_3_CYCLES", - "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most conservative mode." - }, - { - "EventCode": "0x0194", - "EventName": "TXREQ_LIMIT_MAX_CYCLES", - "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is the L2_TQ_SIZE." - }, - { - "EventCode": "0x0195", - "EventName": "TXREQ_LIMIT_3QUARTER_CYCLES", - "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 3/4 of the L2_TQ_SIZE and the L2_TQ_SIZE-1." - }, - { - "EventCode": "0x0196", - "EventName": "TXREQ_LIMIT_HALF_CYCLES", - "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/2 of the L2_TQ_SIZE and 3/4 of the L2_TQ_SIZE." - }, - { - "EventCode": "0x0197", - "EventName": "TXREQ_LIMIT_1QUARTER_CYCLES", - "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/4 of the L2_TQ_SIZE and 1/2 of the L2_TQ_SIZE." - }, - { - "EventCode": "0x019d", - "EventName": "PREFETCH_LATE_CMC", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by CMC prefetch request." - }, - { - "EventCode": "0x019e", - "EventName": "PREFETCH_LATE_BO", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by BO prefetch request." - }, - { - "EventCode": "0x019f", - "EventName": "PREFETCH_LATE_STRIDE", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by STRIDE prefetch request." - }, - { - "EventCode": "0x01a0", - "EventName": "PREFETCH_LATE_SPATIAL", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SPATIAL prefetch request." - }, - { - "EventCode": "0x01a2", - "EventName": "PREFETCH_LATE_TBW", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by TBW prefetch request." - }, - { - "EventCode": "0x01a3", - "EventName": "PREFETCH_LATE_PAGE", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by PAGE prefetch request." - }, - { - "EventCode": "0x01a4", - "EventName": "PREFETCH_LATE_GSMS", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by GSMS prefetch request." - }, - { - "EventCode": "0x01a5", - "EventName": "PREFETCH_LATE_SIP_CONS", - "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SIP_CONS prefetch request." - }, - { - "EventCode": "0x01a6", - "EventName": "PREFETCH_REFILL_CMC", - "PublicDescription": "PF/prefetch or PF/readclean request from CMC pf engine filled the L2 cache." - }, - { - "EventCode": "0x01a7", - "EventName": "PREFETCH_REFILL_BO", - "PublicDescription": "PF/prefetch or PF/readclean request from BO pf engine filled the L2 cache." - }, - { - "EventCode": "0x01a8", - "EventName": "PREFETCH_REFILL_STRIDE", - "PublicDescription": "PF/prefetch or PF/readclean request from STRIDE pf engine filled the L2 cache." - }, - { - "EventCode": "0x01a9", - "EventName": "PREFETCH_REFILL_SPATIAL", - "PublicDescription": "PF/prefetch or PF/readclean request from SPATIAL pf engine filled the L2 cache." - }, - { - "EventCode": "0x01ab", - "EventName": "PREFETCH_REFILL_TBW", - "PublicDescription": "PF/prefetch or PF/readclean request from TBW pf engine filled the L2 cache." - }, - { - "EventCode": "0x01ac", - "EventName": "PREFETCH_REFILL_PAGE", - "PublicDescription": "PF/prefetch or PF/readclean request from PAGE pf engine filled the L2 cache." - }, - { - "EventCode": "0x01ad", - "EventName": "PREFETCH_REFILL_GSMS", - "PublicDescription": "PF/prefetch or PF/readclean request from GSMS pf engine filled the L2 cache." - }, - { - "EventCode": "0x01ae", - "EventName": "PREFETCH_REFILL_SIP_CONS", - "PublicDescription": "PF/prefetch or PF/readclean request from SIP_CONS pf engine filled the L2 cache." - }, - { - "EventCode": "0x01af", - "EventName": "CACHE_HIT_LINE_PF_CMC", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by CMC prefetch request." - }, - { - "EventCode": "0x01b0", - "EventName": "CACHE_HIT_LINE_PF_BO", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by BO prefetch request." - }, - { - "EventCode": "0x01b1", - "EventName": "CACHE_HIT_LINE_PF_STRIDE", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by STRIDE prefetch request." - }, - { - "EventCode": "0x01b2", - "EventName": "CACHE_HIT_LINE_PF_SPATIAL", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SPATIAL prefetch request." - }, - { - "EventCode": "0x01b4", - "EventName": "CACHE_HIT_LINE_PF_TBW", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by TBW prefetch request." - }, - { - "EventCode": "0x01b5", - "EventName": "CACHE_HIT_LINE_PF_PAGE", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by PAGE prefetch request." - }, - { - "EventCode": "0x01b6", - "EventName": "CACHE_HIT_LINE_PF_GSMS", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by GSMS prefetch request." - }, - { - "EventCode": "0x01b7", - "EventName": "CACHE_HIT_LINE_PF_SIP_CONS", - "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SIP_CONS prefetch request." - }, - { - "EventCode": "0x01cb", - "EventName": "L2_TQ_OUTSTANDING", - "PublicDescription": "Outstanding tracker count, per cycle. The Event increments by the number of valid entries pertaining to this thread in the L2TQ, in each cycle. The Event can be used to calculate the occupancy of L2TQ by dividing this by the CPU_CYCLES Event. The L2TQ queue tracks the outstanding Read, Write ,and Snoop transactions. The Read transaction and the Write transaction entries are attributable to PE, whereas the Snoop transactions are not always attributable to PE." - }, - { - "EventCode": "0x01cc", - "EventName": "TXREQ_LIMIT_COUNT_CYCLES", - "PublicDescription": "The Event increments by the dynamic TXREQ value, in each cycle. This is a companion Event of TXREQ_LIMIT_MAX_CYCLES, TXREQ_LIMIT_3QUARTER_CYCLES, TXREQ_LIMIT_HALF_CYCLES, and TXREQ_LIMIT_1QUARTER_CYCLES." - }, - { - "EventCode": "0x01d2", - "EventName": "DVM_TLBI_RCVD", - "PublicDescription": "The Event counts the number of TLBI DVM message received over CHI interface, for *this* Core." - }, - { - "EventCode": "0x01d6", - "EventName": "DSB_COMMITING_LOCAL_TLBI", - "PublicDescription": "The Event counts the number of DSB that are retired and committed at least one local TLBI Instruction. This Event increments no more than once (in a cycle) even if the DSB commits multiple local TLBI Instruction." - }, - { - "EventCode": "0x01d7", - "EventName": "DSB_COMMITING_BROADCAST_TLBI", - "PublicDescription": "The Event counts the number of DSB that are retired and committed at least one broadcast TLBI Instruction. This Event increments no more than once (in a cycle) even if the DSB commits multiple broadcast TLBI Instruction." - }, - { - "EventCode": "0x01f0", - "EventName": "TMS_ST_TO_SMT_LATENCY", - "PublicDescription": "The Event counts the number of CPU cycles spent on TMS for ST-to-SMT switch. This Event is counted by both the threads - The Event in both threads increment during TMS for ST-to-SMT switch." - }, - { - "EventCode": "0x01f1", - "EventName": "TMS_SMT_TO_ST_LATENCY", - "PublicDescription": "The Event counts the number of CPU cycles spent on TMS for SMT-to-ST switch. The count also includes the CPU cycles spend due to an aborted SMT-to-ST TMS attempt. This Event is counted only by the thread that is not in WFI." - }, - { - "EventCode": "0x01f2", - "EventName": "TMS_ST_TO_SMT_COUNT", - "PublicDescription": "The Event counts the number of completed TMS from ST-to-SMT. This Event is counted only by the active thread (the one that is not in WFI). Note: When an active thread enters the Debug state in ST-Full resource mode, it is switched to SMT mode. This is because the inactive thread cannot wake up while the other thread remains in the Debug state. To prEvent this issue, threads operating in ST-Full resource mode are transitioned to SMT mode upon entering Debug state. The Event count will also reflect such switches from ST to SMT mode. (Also see the (NV_CPUACTLR14_EL1.chka_prEvent_st_tx_to_smt_when_tx_in_debug_state bit to disable this behavior.)" - }, - { - "EventCode": "0x01f3", - "EventName": "TMS_SMT_TO_ST_COUNT", - "PublicDescription": "The Event counts the number of completed TMS from SMT-to-ST. This Event is counted only by the thread that is not in WFI." - }, - { - "EventCode": "0x01f4", - "EventName": "TMS_SMT_TO_ST_COUNT_ABRT", - "PublicDescription": "The Event counts the number of aborted TMS from SMT-to-ST. This Event is counted only by the thread that is not in WFI." - }, - { - "EventCode": "0x021c", - "EventName": "CWT_ALLOC_ENTRY", - "PublicDescription": "Cache Way Tracker Allocate entry." - }, - { - "EventCode": "0x021d", - "EventName": "CWT_ALLOC_LINE", - "PublicDescription": "Cache Way Tracker Allocate line." - }, - { - "EventCode": "0x021e", - "EventName": "CWT_HIT", - "PublicDescription": "Cache Way Tracker hit." - }, - { - "EventCode": "0x021f", - "EventName": "CWT_HIT_TAG", - "PublicDescription": "Cache Way Tracker hit when ITAG lookup suppressed." - }, - { - "EventCode": "0x0220", - "EventName": "CWT_REPLAY_TAG", - "PublicDescription": "Cache Way Tracker causes ITAG replay due to miss when ITAG lookup suppressed." - }, - { - "EventCode": "0x0250", - "EventName": "GPT_REQ", - "PublicDescription": "GPT lookup." - }, - { - "EventCode": "0x0251", - "EventName": "GPT_WC_HIT", - "PublicDescription": "GPT lookup hit in Walk cache." - }, - { - "EventCode": "0x0252", - "EventName": "GPT_PG_HIT", - "PublicDescription": "GPT lookup hit in TLB." - }, - { - "EventCode": "0x01ba", - "EventName": "PREFETCH_LATE_STORE_ISSUE", - "PublicDescription": "The Event counts the number of demand requests that matches a Store-issue prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." - }, - { - "EventCode": "0x01bb", - "EventName": "PREFETCH_LATE_STORE_STRIDE", - "PublicDescription": "The Event counts the number of demand requests that matches a Store-stride prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." - }, - { - "EventCode": "0x01bc", - "EventName": "PREFETCH_LATE_PC_OFFSET", - "PublicDescription": "The Event counts the number of demand requests that matches a PC-offset prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." - }, - { - "EventCode": "0x01bd", - "EventName": "PREFETCH_LATE_IFUPF", - "PublicDescription": "The Event counts the number of demand requests that matches a IFU prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." - }, - { - "EventCode": "0x01be", - "EventName": "PREFETCH_REFILL_STORE_ISSUE", - "PublicDescription": "The Event counts the number of cache refills due to Store-Issue prefetcher." - }, - { - "EventCode": "0x01bf", - "EventName": "PREFETCH_REFILL_STORE_STRIDE", - "PublicDescription": "The Event counts the number of cache refills due to Store-stride prefetcher." - }, - { - "EventCode": "0x01c0", - "EventName": "PREFETCH_REFILL_PC_OFFSET", - "PublicDescription": "The Event counts the number of cache refills due to PC-offset prefetcher." - }, - { - "EventCode": "0x01c1", - "EventName": "PREFETCH_REFILL_IFUPF", - "PublicDescription": "The Event counts the number of cache refills due to IFU prefetcher." - }, - { - "EventCode": "0x01c2", - "EventName": "CACHE_HIT_LINE_PF_STORE_ISSUE", - "PublicDescription": "The Event counts the number of first hit to a cache line filled by Store-issue prefetcher." - }, - { - "EventCode": "0x01c3", - "EventName": "CACHE_HIT_LINE_PF_STORE_STRIDE", - "PublicDescription": "The Event counts the number of first hit to a cache line filled by Store-stride prefetcher." - }, - { - "EventCode": "0x01c4", - "EventName": "CACHE_HIT_LINE_PF_PC_OFFSET", - "PublicDescription": "The Event counts the number of first hit to a cache line filled by PC-offset prefetcher." - }, - { - "EventCode": "0x01c5", - "EventName": "CACHE_HIT_LINE_PF_IFUPF", - "PublicDescription": "The Event counts the number of first hit to a cache line filled by IFU prefetcher." - }, - { - "EventCode": "0x01c6", - "EventName": "L2_PF_GEN_ST_ISSUE", - "PublicDescription": "Store-issue prefetch to L2 generated." - }, - { - "EventCode": "0x01c7", - "EventName": "L2_PF_GEN_ST_STRIDE", - "PublicDescription": "Store-stride prefetch to L2 generated" - }, - { - "EventCode": "0x01ee", - "EventName": "CACHE_HIT_LINE_PF_CONVERTED_PRFM", - "PublicDescription": "The Event counts the number of first hit to a cache line filled by Converted-L1D-PRFM or Converted-L2D-PRFM. Note that L2D_CACHE_HIT_RWL1PRF_FPRFM is inclusive of CACHE_HIT_LINE_PF_CONVERTED_PRFM, where both the CACHE_HIT_LINE_PF_CONVERTED_PRFM and the L2D_CACHE_HIT_RWL1PRF_FPRFM increment on a first hit to L2 D-cache filled by Converted-L1D-PRFM or Converted-L2D-PRFM." - }, - { - "EventCode": "0x01ec", - "EventName": "PREFETCH_LATE_CONVERTED_PRFM", - "PublicDescription": "The Event counts the number of demand requests that matches a Converted-L1D-PRFM or Converted-L2D-PRFM pending refill request at L2 D-cache. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements. Note that this Event is not counted by the L2D_CACHE_HIT_RWL1PRF_LATE_HWPRF, though the Converted-L1D-PRFM or Converted-L2D-PRFM are replayed by the L2PRQ." - }, - { - "EventCode": "0x01ed", - "EventName": "PREFETCH_REFILL_CONVERTED_PRFM", - "PublicDescription": "The Event counts the number of L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM. Note : L2D_CACHE_REFILL_PRFM is inclusive of PREFETCH_REFILL_PRFM_CONVERTED, where both the PREFETCH_REFILL_PRFM_CONVERTED and the L2D_CACHE_REFILL_PRFM increment when L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM." - }, - { - "EventCode": "0x01eb", - "EventName": "L1DPRFM_L2DPRFM_TO_L2PRQ_CONVERTED", - "PublicDescription": "The Event counts the number of Converted-L1D-PRFMs and Converted-L2D-PRFM. Activities involving the Converted-L1D-PRFM are counted by the L1D_CACHE_PRFM. However they are *not* counted by the L1D_CACHE_REFILL_PRFM, and L1D_CACHE_REFILL, as these Converted-L1D-PRFM are treated as L2 D hardware prefetches. Activities around the Converted-L1D-PRFMs and Converted-L2D-PRFMs are counted by the L2D_CACHE_PRFM, L2D_CACHE_REFILL_PRFM and L2D_CACHE_REFILL Events." - }, - { - "EventCode": "0x01ce", - "EventName": "L3DPRFM_TO_L2PRQ_CONVERTED", - "PublicDescription": "The Event counts the number of Converted-L3D-PRFMs. These are indeed L3D PRFM and activities around these PRFM are counted by the L3D_CACHE_PRFM, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL Events." - }, - { - "EventCode": "0x0202", - "EventName": "L0I_CACHE_RD", - "PublicDescription": "The Event counts the number of predict blocks serviced out of L0 I-cache. Note: The L0 I-cache performs at most 4 L0 I look-up in a cycle. Two of which are to service PB from L0 I. And the other two to refill L0 I-cache from L1 I. This Event count only the L0 I-cache lookup pertaining to servicing the PB from L0 I." - }, - { - "EventCode": "0x0203", - "EventName": "L0I_CACHE_REFILL", - "PublicDescription": "The Event counts the number of L0I cache refill from L1 I-cache." - }, - { - "EventCode": "0x0207", - "EventName": "INTR_LATENCY", - "PublicDescription": "The Event counts the number of cycles elapsed between when an Interrupt is recognized (after masking) to when a uop associated with the first Instruction in the destination exception level is allocated. If there is some other flush condition that pre-empts the Interrupt, then the cycles counted terminates early at the first Instruction executed after that flush. In the Event of dropped Interrupts (when an Interrupt is deasserted before it is taken), this counter measures the number of cycles that elapse from the moment an Interrupt is recognized (post-masking) until the Interrupt is dropped or deasserted." - }, - { - "EventCode": "0x018f", - "EventName": "L1_PF_GEN_MCMC", - "PublicDescription": "Load/Store prefetch to L1 generated, MCMC." - }, - { - "EventCode": "0x0123", - "EventName": "FLUSH_STDBYPASS", - "PublicDescription": "Flushes due to bad predecode. This only includes CT flushes." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json deleted file mode 100644 index ab384239541b8..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json +++ /dev/null @@ -1,94 +0,0 @@ -[ - { - "ArchStdEvent": "INST_RETIRED", - "PublicDescription": "The Event counts Instructions that have been architecturally executed." - }, - { - "ArchStdEvent": "CID_WRITE_RETIRED", - "PublicDescription": "The Event counts architecturally executed Writes to the CONTEXTIDR_EL1 register, which usually contain the kernel PID and can be output with hardware trace." - }, - { - "ArchStdEvent": "BR_IMMED_RETIRED", - "PublicDescription": "The Event counts architecturally executed direct Branches." - }, - { - "ArchStdEvent": "BR_RETURN_RETIRED", - "PublicDescription": "The Event counts architecturally executed procedure returns." - }, - { - "ArchStdEvent": "TTBR_WRITE_RETIRED", - "PublicDescription": "The Event counts architectural Writes to TTBR0/1_EL1. If virtualization host extensions are enabled (by setting the HCR_EL2.E2H bit to 1), then accesses to TTBR0/1_EL1 that are redirected to TTBR0/1_EL2, or accesses to TTBR0/1_EL12, are counted. TTBRn registers are typically updated when the kernel is swapping user-space threads or applications." - }, - { - "ArchStdEvent": "BR_RETIRED", - "PublicDescription": "The Event counts architecturally executed Branches, whether the Branch is taken or not. Instructions that explicitly write to the PC are also counted. Note that exception generating Instructions, exception return Instructions and context synchronization Instructions are not counted." - }, - { - "ArchStdEvent": "BR_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts Branches counted by BR_RETIRED which were mispredicted and caused a pipeline flush." - }, - { - "ArchStdEvent": "OP_RETIRED", - "PublicDescription": "The Event counts micro-operations that are architecturally executed. This is a count of number of micro-operations retired from the commit queue in a single cycle." - }, - { - "ArchStdEvent": "BR_INDNR_TAKEN_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were taken." - }, - { - "ArchStdEvent": "BR_IMMED_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed direct Branches that were correctly predicted." - }, - { - "ArchStdEvent": "BR_IMMED_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed direct Branches that were mispredicted and caused a pipeline flush." - }, - { - "ArchStdEvent": "BR_IND_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns that were correctly predicted." - }, - { - "ArchStdEvent": "BR_IND_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns that were mispredicted and caused a pipeline flush." - }, - { - "ArchStdEvent": "BR_RETURN_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed procedure returns that were correctly predicted." - }, - { - "ArchStdEvent": "BR_RETURN_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed procedure returns that were mispredicted and caused a pipeline flush." - }, - { - "ArchStdEvent": "BR_INDNR_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were correctly predicted." - }, - { - "ArchStdEvent": "BR_INDNR_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches excluding procedure returns that were mispredicted and caused a pipeline flush." - }, - { - "ArchStdEvent": "BR_TAKEN_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed Branches that were taken and were correctly predicted." - }, - { - "ArchStdEvent": "BR_TAKEN_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed branches that were taken and were mispredicted causing a pipeline flush." - }, - { - "ArchStdEvent": "BR_SKIP_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed Branches that were not taken and were correctly predicted." - }, - { - "ArchStdEvent": "BR_SKIP_MIS_PRED_RETIRED", - "PublicDescription": "The Event counts architecturally executed Branches that were not taken and were mispredicted causing a pipeline flush." - }, - { - "ArchStdEvent": "BR_PRED_RETIRED", - "PublicDescription": "The Event counts Branch Instructions counted by BR_RETIRED which were correctly predicted." - }, - { - "ArchStdEvent": "BR_IND_RETIRED", - "PublicDescription": "The Event counts architecturally executed indirect Branches including procedure returns." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json deleted file mode 100644 index 8d1fe2a8b161f..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json +++ /dev/null @@ -1,42 +0,0 @@ -[ - { - "ArchStdEvent": "SAMPLE_POP", - "PublicDescription": "The Event counts statistical profiling sample population, the count of all operations that could be sampled but may or may not be chosen for sampling." - }, - { - "ArchStdEvent": "SAMPLE_FEED", - "PublicDescription": "The Event counts statistical profiling samples taken for sampling." - }, - { - "ArchStdEvent": "SAMPLE_FILTRATE", - "PublicDescription": "The Event counts statistical profiling samples taken which are not removed by filtering." - }, - { - "ArchStdEvent": "SAMPLE_COLLISION", - "PublicDescription": "The Event counts statistical profiling samples that have collided with a previous sample and so therefore not taken." - }, - { - "ArchStdEvent": "SAMPLE_FEED_BR", - "PublicDescription": "The Event counts statistical profiling samples taken which are Branches." - }, - { - "ArchStdEvent": "SAMPLE_FEED_LD", - "PublicDescription": "The Event counts statistical profiling samples taken which are Loads or Load atomic operations." - }, - { - "ArchStdEvent": "SAMPLE_FEED_ST", - "PublicDescription": "The Event counts statistical profiling samples taken which are Stores or Store atomic operations." - }, - { - "ArchStdEvent": "SAMPLE_FEED_OP", - "PublicDescription": "The Event counts statistical profiling samples taken which are matching any operation type filters supported." - }, - { - "ArchStdEvent": "SAMPLE_FEED_EVENT", - "PublicDescription": "The Event counts statistical profiling samples taken which are matching Event packet filter constraints." - }, - { - "ArchStdEvent": "SAMPLE_FEED_LAT", - "PublicDescription": "The Event counts statistical profiling samples taken which are exceeding minimum latency set by operation latency filter constraints." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json deleted file mode 100644 index 18df3f7011150..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json +++ /dev/null @@ -1,230 +0,0 @@ -[ - { - "ArchStdEvent": "INST_SPEC", - "PublicDescription": "The Event counts operations that have been speculatively executed." - }, - { - "ArchStdEvent": "OP_SPEC", - "PublicDescription": "The Event counts micro-operations speculatively executed. This is the count of the number of micro-operations dispatched in a cycle." - }, - { - "ArchStdEvent": "UNALIGNED_LD_SPEC", - "PublicDescription": "The Event counts unaligned memory Read operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. The Event does not count preload operations (PLD, PLI). This Event is a subset of the UNALIGNED_LDST_SPEC Event." - }, - { - "ArchStdEvent": "UNALIGNED_ST_SPEC", - "PublicDescription": "The Event counts unaligned memory Write operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. This Event is a subset of the UNALIGNED_LDST_SPEC Event." - }, - { - "ArchStdEvent": "UNALIGNED_LDST_SPEC", - "PublicDescription": "The Event counts unaligned memory operations issued by the CPU. This Event counts unaligned accesses (as defined by the actual Instruction), even if they are subsequently issued as multiple aligned accesses. This Event is the sum of the UNALIGNED_ST_SPEC and UNALIGNED_LD_SPEC Events." - }, - { - "ArchStdEvent": "LDREX_SPEC", - "PublicDescription": "The Event counts Load-Exclusive operations that have been speculatively executed. For example: LDREX, LDX" - }, - { - "ArchStdEvent": "STREX_PASS_SPEC", - "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed and have successfully completed the Store operation." - }, - { - "ArchStdEvent": "STREX_FAIL_SPEC", - "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed and have not successfully completed the Store operation." - }, - { - "ArchStdEvent": "STREX_SPEC", - "PublicDescription": "The Event counts Store-exclusive operations that have been speculatively executed. This Event is the sum of STREX_PASS_SPEC and STREX_FAIL_SPEC Events." - }, - { - "ArchStdEvent": "LD_SPEC", - "PublicDescription": "The Event counts speculatively executed Load operations including Single Instruction Multiple Data (SIMD) Load operations." - }, - { - "ArchStdEvent": "ST_SPEC", - "PublicDescription": "The Event counts speculatively executed Store operations including Single Instruction Multiple Data (SIMD) Store operations." - }, - { - "ArchStdEvent": "LDST_SPEC", - "PublicDescription": "The Event counts Load and Store operations that have been speculatively executed." - }, - { - "ArchStdEvent": "DP_SPEC", - "PublicDescription": "The Event counts speculatively executed logical or arithmetic Instructions such as MOV/MVN operations." - }, - { - "ArchStdEvent": "ASE_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD operations excluding Load, Store, and Move micro-operations that move Data to or from SIMD (vector) registers." - }, - { - "ArchStdEvent": "VFP_SPEC", - "PublicDescription": "The Event counts speculatively executed floating point operations. This Event does not count operations that move Data to or from floating point (vector) registers." - }, - { - "ArchStdEvent": "PC_WRITE_SPEC", - "PublicDescription": "The Event counts speculatively executed operations which cause software changes of the PC. Those operations include all taken Branch operations." - }, - { - "ArchStdEvent": "CRYPTO_SPEC", - "PublicDescription": "The Event counts speculatively executed cryptographic operations except for PMULL and VMULL operations." - }, - { - "ArchStdEvent": "BR_IMMED_SPEC", - "PublicDescription": "The Event counts direct Branch operations which are speculatively executed." - }, - { - "ArchStdEvent": "BR_RETURN_SPEC", - "PublicDescription": "The Event counts procedure return operations (RET, RETAA and RETAB) which are speculatively executed." - }, - { - "ArchStdEvent": "BR_INDIRECT_SPEC", - "PublicDescription": "The Event counts indirect branch operations including procedure returns, which are speculatively executed. This includes operations that force a software change of the PC, other than exception-generating operations and direct Branch Instructions. Some examples of the Instructions counted by this Event include BR Xn, RET, etc." - }, - { - "ArchStdEvent": "ISB_SPEC", - "PublicDescription": "The Event counts ISB operations that are executed." - }, - { - "ArchStdEvent": "DSB_SPEC", - "PublicDescription": "The Event counts DSB operations that are speculatively issued to Load/Store unit in the CPU." - }, - { - "ArchStdEvent": "DMB_SPEC", - "PublicDescription": "The Event counts DMB operations that are speculatively issued to the Load/Store unit in the CPU. This Event does not count implied barriers from Load-acquire/Store-release operations." - }, - { - "ArchStdEvent": "CSDB_SPEC", - "PublicDescription": "The Event counts CSDB operations that are speculatively issued to the Load/Store unit in the CPU. This Event does not count implied barriers from Load-acquire/Store-release operations." - }, - { - "ArchStdEvent": "RC_LD_SPEC", - "PublicDescription": "The Event counts any Load acquire operations that are speculatively executed. For example: LDAR, LDARH, LDARB" - }, - { - "ArchStdEvent": "RC_ST_SPEC", - "PublicDescription": "The Event counts any Store release operations that are speculatively executed. For example: STLR, STLRH, STLRB" - }, - { - "ArchStdEvent": "SIMD_INST_SPEC", - "PublicDescription": "The Event counts speculatively executed operations that are SIMD or SVE vector operations or Advanced SIMD non-scalar operations." - }, - { - "ArchStdEvent": "ASE_INST_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD operations." - }, - { - "ArchStdEvent": "SVE_INST_SPEC", - "PublicDescription": "The Event counts speculatively executed operations that are SVE operations." - }, - { - "ArchStdEvent": "INT_SPEC", - "PublicDescription": "The Event counts speculatively executed integer arithmetic operations." - }, - { - "ArchStdEvent": "SVE_PRED_SPEC", - "PublicDescription": "The Event counts speculatively executed predicated SVE operations. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." - }, - { - "ArchStdEvent": "SVE_PRED_EMPTY_SPEC", - "PublicDescription": "The Event counts speculatively executed predicated SVE operations with no active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." - }, - { - "ArchStdEvent": "SVE_PRED_FULL_SPEC", - "PublicDescription": "The Event counts speculatively executed predicated SVE operations with all predicate elements active. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." - }, - { - "ArchStdEvent": "SVE_PRED_PARTIAL_SPEC", - "PublicDescription": "The Event counts speculatively executed predicated SVE operations with at least one but not all active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." - }, - { - "ArchStdEvent": "SVE_PRED_NOT_FULL_SPEC", - "PublicDescription": "The Event counts speculatively executed predicated SVE operations with at least one non active predicate elements. This counter also counts SVE operation due to Instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to Instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* Events." - }, - { - "ArchStdEvent": "PRF_SPEC", - "PublicDescription": "The Event counts speculatively executed operations that prefetch memory. For example, Scalar: PRFM, SVE: PRFB, PRFD, PRFH, or PRFW." - }, - { - "ArchStdEvent": "SVE_LDFF_SPEC", - "PublicDescription": "The Event counts speculatively executed SVE first fault or non-fault Load operations." - }, - { - "ArchStdEvent": "SVE_LDFF_FAULT_SPEC", - "PublicDescription": "The Event counts speculatively executed SVE first fault or non-fault Load operations that clear at least one bit in the FFR." - }, - { - "ArchStdEvent": "ASE_SVE_INT8_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type an 8-bit integer." - }, - { - "ArchStdEvent": "ASE_SVE_INT16_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 16-bit integer." - }, - { - "ArchStdEvent": "ASE_SVE_INT32_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 32-bit integer." - }, - { - "ArchStdEvent": "ASE_SVE_INT64_SPEC", - "PublicDescription": "The Event counts speculatively executed Advanced SIMD or SVE integer operations with the largest Data type a 64-bit integer." - }, - { - "EventCode": "0x011d", - "EventName": "SPEC_RET_STACK_FULL", - "PublicDescription": "The Event counts predict pipe stalls due to speculative return address predictor full." - }, - { - "EventCode": "0x011f", - "EventName": "MOPS_SPEC", - "PublicDescription": "Macro-ops speculatively decoded." - }, - { - "EventCode": "0x0180", - "EventName": "BR_SPEC_PRED_TAKEN", - "PublicDescription": "Number of Predicted Taken from Branch Predictor." - }, - { - "EventCode": "0x0181", - "EventName": "BR_SPEC_PRED_TAKEN_FROM_L2BTB", - "PublicDescription": "Number of Predicted Taken Branch from L2 BTB." - }, - { - "EventCode": "0x0182", - "EventName": "BR_SPEC_PRED_TAKEN_MULTI", - "PublicDescription": "Number of Predicted Taken for Polymorphic Branch." - }, - { - "EventCode": "0x0185", - "EventName": "BR_SPEC_PRED_STATIC", - "PublicDescription": "Number of post fetch prediction." - }, - { - "EventCode": "0x01d0", - "EventName": "TLBI_LOCAL_SPEC", - "PublicDescription": "A non-broadcast TLBI Instruction executed (Speculatively or otherwise) on *this* PE." - }, - { - "EventCode": "0x01d1", - "EventName": "TLBI_BROADCAST_SPEC", - "PublicDescription": "A broadcast TLBI Instruction executed (Speculatively or otherwise) on *this* PE." - }, - { - "EventCode": "0x0200", - "EventName": "SIMD_CRYPTO_INST_SPEC", - "PublicDescription": "SIMD, SVE, and CRYPTO Instructions speculatively decoded." - }, - { - "EventCode": "0x01e7", - "EventName": "BR_SPEC_PRED_ALN_REDIR", - "PublicDescription": "BPU predict pipe align redirect (either AL-APQ hit/miss)." - }, - { - "EventCode": "0x022e", - "EventName": "VPRED_LD_SPEC", - "PublicDescription": "The Event counts the number of Speculatively-executed-Load operations with addresses produced by the value-prediction mechanism. The loaded Data might be discarded if the predicted address differs from the actual address." - }, - { - "EventCode": "0x022f", - "EventName": "VPRED_LD_SPEC_MISMATCH", - "PublicDescription": "The Event counts a subset of VPRED_LD_SPEC where the predicted Load address and the actual address mismatched." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json deleted file mode 100644 index 652c1e3305d08..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json +++ /dev/null @@ -1,145 +0,0 @@ -[ - { - "ArchStdEvent": "STALL_FRONTEND", - "PublicDescription": "The Event counts cycles when frontend could not send any micro-operations to the rename stage because of frontend resource stalls caused by fetch memory latency or Branch prediction flow stalls. STALL_FRONTEND_SLOTS counts SLOTS during the cycle when this Event counts. STALL_SLOT_FRONTEND will count SLOTS when this Event is counted on this CPU." - }, - { - "ArchStdEvent": "STALL_BACKEND", - "PublicDescription": "The Event counts cycles whenever the rename unit is unable to send any micro-operations to the backend of the pipeline because of backend resource constraints. Backend resource constraints can include issue stage fullness, execution stage fullness, or other internal pipeline resource fullness. All the backend slots were empty during the cycle when this Event counts." - }, - { - "ArchStdEvent": "STALL", - "PublicDescription": "The Event counts cycles when no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall). This Event is the sum of STALL_FRONTEND and STALL_BACKEND." - }, - { - "ArchStdEvent": "STALL_SLOT_BACKEND", - "PublicDescription": "The Event counts slots per cycle in which no operations are sent from the rename unit to the backend due to backend resource constraints. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND counts at least 1. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND is SLOTS." - }, - { - "ArchStdEvent": "STALL_SLOT_FRONTEND", - "PublicDescription": "The Event counts slots per cycle in which no operations are sent to the rename unit from the frontend due to frontend resource constraints. STALL_FRONTEND counts during the cycle when STALL_SLOT_FRONTEND is SLOTS." - }, - { - "ArchStdEvent": "STALL_SLOT", - "PublicDescription": "The Event counts slots per cycle in which no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall). STALL_SLOT is the sum of STALL_SLOT_FRONTEND and STALL_SLOT_BACKEND." - }, - { - "ArchStdEvent": "STALL_BACKEND_MEM", - "PublicDescription": "The Event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the last level Core cache. Last level cache in this CPU is Level 2, hence this Event counts same as STALL_BACKEND_L2D." - }, - { - "ArchStdEvent": "STALL_FRONTEND_MEMBOUND", - "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the memory resources." - }, - { - "ArchStdEvent": "STALL_FRONTEND_L1I", - "PublicDescription": "The Event counts cycles when the frontend is stalled because there is an Instruction fetch request pending in the L1 I-cache." - }, - { - "ArchStdEvent": "STALL_FRONTEND_MEM", - "PublicDescription": "The Event counts cycles when the frontend is stalled because there is an Instruction fetch request pending in the last level Core cache. Last level cache in this CPU is Level 2, hence this Event counts rather than STALL_FRONTEND_L2I." - }, - { - "ArchStdEvent": "STALL_FRONTEND_TLB", - "PublicDescription": "The Event counts when the frontend is stalled on any TLB misses being handled. This Event also counts the TLB accesses made by hardware prefetches." - }, - { - "ArchStdEvent": "STALL_FRONTEND_CPUBOUND", - "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the CPU resources excluding memory resources." - }, - { - "ArchStdEvent": "STALL_FRONTEND_FLOW", - "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the Branch prediction unit." - }, - { - "ArchStdEvent": "STALL_FRONTEND_FLUSH", - "PublicDescription": "The Event counts cycles when the frontend could not send any micro-operations to the rename stage as the frontend is recovering from a machine flush or resteer. Example scenarios that cause a flush include Branch mispredictions, taken exceptions, microarchitectural flush etc." - }, - { - "ArchStdEvent": "STALL_BACKEND_MEMBOUND", - "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to resource constraints in the memory resources." - }, - { - "ArchStdEvent": "STALL_BACKEND_L1D", - "PublicDescription": "The Event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the L1 D-cache." - }, - { - "ArchStdEvent": "STALL_BACKEND_TLB", - "PublicDescription": "The Event counts cycles when the backend is stalled on any demand TLB misses being handled." - }, - { - "ArchStdEvent": "STALL_BACKEND_ST", - "PublicDescription": "The Event counts cycles when the backend is stalled and there is a Store that has not reached the pre-commit stage." - }, - { - "ArchStdEvent": "STALL_BACKEND_CPUBOUND", - "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to any resource constraints in the CPU excluding memory resources." - }, - { - "ArchStdEvent": "STALL_BACKEND_BUSY", - "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations because the issue queues are full to take any operations for execution." - }, - { - "ArchStdEvent": "STALL_BACKEND_ILOCK", - "PublicDescription": "The Event counts cycles when the backend could not accept any micro-operations due to resource constraints imposed by input dependency." - }, - { - "ArchStdEvent": "STALL_BACKEND_RENAME", - "PublicDescription": "The Event counts cycles when backend is stalled even when operations are available from the frontend but at least one is not ready to be sent to the backend because no rename register is available." - }, - { - "EventCode": "0x0158", - "EventName": "FLAG_DISP_STALL", - "PublicDescription": "Rename stalled due to FRF(Flag register file) full." - }, - { - "EventCode": "0x0159", - "EventName": "GEN_DISP_STALL", - "PublicDescription": "Rename stalled due to GRF (General-purpose register file) full." - }, - { - "EventCode": "0x015a", - "EventName": "VEC_DISP_STALL", - "PublicDescription": "Rename stalled due to VRF (Vector register file) full." - }, - { - "EventCode": "0x015c", - "EventName": "SX_IQ_STALL", - "PublicDescription": "Dispatch stalled due to IQ full, SX." - }, - { - "EventCode": "0x015d", - "EventName": "MX_IQ_STALL", - "PublicDescription": "Dispatch stalled due to IQ full, MX." - }, - { - "EventCode": "0x015e", - "EventName": "LS_IQ_STALL", - "PublicDescription": "Dispatch stalled due to IQ full, LS." - }, - { - "EventCode": "0x015f", - "EventName": "VX_IQ_STALL", - "PublicDescription": "Dispatch stalled due to IQ full, VX." - }, - { - "EventCode": "0x0160", - "EventName": "MCQ_FULL_STALL", - "PublicDescription": "Dispatch stalled due to MCQ full." - }, - { - "EventCode": "0x01cf", - "EventName": "PRD_DISP_STALL", - "PublicDescription": "Rename stalled due to predicate registers (physical) are full." - }, - { - "EventCode": "0x01e0", - "EventName": "CSDB_STALL", - "PublicDescription": "Rename stalled due to CSDB." - }, - { - "EventCode": "0x01e2", - "EventName": "STALL_SLOT_FRONTEND_WITHOUT_MISPRED", - "PublicDescription": "Stall slot frontend during non-mispredicted branch. The Event counts the STALL_STOT_FRONTEND Events, except for the 4 cycles following a mispredicted branch Event or 4 cycles following a commit flush&restart Event." - } -] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json deleted file mode 100644 index 9a81a62a26462..0000000000000 --- a/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json +++ /dev/null @@ -1,158 +0,0 @@ -[ - { - "ArchStdEvent": "L1I_TLB_REFILL", - "PublicDescription": "The Event counts L1 Instruction TLB refills from any Instruction fetch (Demand, hardware prefetch and software preload accesses). If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB." - }, - { - "ArchStdEvent": "L1D_TLB_REFILL", - "PublicDescription": "The Event counts L1 Data TLB accesses that resulted in TLB refills. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count on an access from an AT(address translation) Instruction. This Event is the sum of the L1D_TLB_REFILL_RD and L1D_TLB_REFILL_WR Events." - }, - { - "ArchStdEvent": "L1D_TLB", - "PublicDescription": "The Event counts L1 Data TLB accesses caused by any memory Load or Store operation. Note that Load or Store Instructions can be broken up into multiple memory operations. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "L1I_TLB", - "PublicDescription": "The Event counts L1 Instruction TLB accesses (caused by Demand or hardware prefetch or software preload accesses), whether the access hits or misses in the TLB. This Event counts both demand accesses and prefetch or preload generated accesses. This Event is a superset of the L1I_TLB_REFILL Event." - }, - { - "ArchStdEvent": "L2D_TLB_REFILL", - "PublicDescription": "The Event counts L2 TLB refills caused by memory operations from both Data and Instruction fetch, except for those caused by TLB maintenance operations and hardware prefetches. This Event is the sum of the L2D_TLB_REFILL_RD and L2D_TLB_REFILL_WR Events." - }, - { - "ArchStdEvent": "L2D_TLB", - "PublicDescription": "The Event counts L2 TLB accesses except those caused by TLB maintenance operations. This Event is the sum of the L2D_TLB_RD and L2D_TLB_WR Events." - }, - { - "ArchStdEvent": "DTLB_WALK", - "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations. This Event does not include prefetches." - }, - { - "ArchStdEvent": "ITLB_WALK", - "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations. This Event does not include prefetches." - }, - { - "ArchStdEvent": "L1D_TLB_REFILL_RD", - "PublicDescription": "The Event counts L1 Data TLB refills caused by memory Read operations. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count on an access from an Address Translation (AT) Instruction. This Event is a subset of the L1D_TLB_REFILL Event." - }, - { - "ArchStdEvent": "L1D_TLB_REFILL_WR", - "PublicDescription": "The Event counts L1 Data TLB refills caused by Data side memory Write operations. If there are multiple misses in the TLB that are resolved by the refill, then this Event only counts once. This Event counts for refills caused by preload Instructions or hardware prefetch accesses. This Event counts regardless of whether the miss hits in L2 or results in a translation table walk. This Event will not count if the table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This Event will not count with an access from an Address Translation (AT) Instruction. This Event is a subset of the L1D_TLB_REFILL Event." - }, - { - "ArchStdEvent": "L1D_TLB_RD", - "PublicDescription": "The Event counts L1 Data TLB accesses caused by memory Read operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "L1D_TLB_WR", - "PublicDescription": "The Event counts any L1 Data side TLB accesses caused by memory Write operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "L2D_TLB_REFILL_RD", - "PublicDescription": "The Event counts L2 TLB refills caused by memory Read operations from both Data and Instruction fetch except for those caused by TLB maintenance operations or hardware prefetches. This Event is a subset of the L2D_TLB_REFILL Event." - }, - { - "ArchStdEvent": "L2D_TLB_REFILL_WR", - "PublicDescription": "The Event counts L2 TLB refills caused by memory Write operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB_REFILL Event." - }, - { - "ArchStdEvent": "L2D_TLB_RD", - "PublicDescription": "The Event counts L2 TLB accesses caused by memory Read operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB Event." - }, - { - "ArchStdEvent": "L2D_TLB_WR", - "PublicDescription": "The Event counts L2 TLB accesses caused by memory Write operations from both Data and Instruction fetch except for those caused by TLB maintenance operations. This Event is a subset of the L2D_TLB Event." - }, - { - "ArchStdEvent": "DTLB_WALK_PERCYC", - "PublicDescription": "The Event counts the number of Data translation table walks in progress per cycle." - }, - { - "ArchStdEvent": "ITLB_WALK_PERCYC", - "PublicDescription": "The Event counts the number of Instruction translation table walks in progress per cycle." - }, - { - "ArchStdEvent": "L1D_TLB_RW", - "PublicDescription": "The Event counts L1 Data TLB demand accesses caused by memory Read or Write operations. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "L1I_TLB_RD", - "PublicDescription": "The Event counts L1 Instruction TLB demand accesses whether the access hits or misses in the TLB." - }, - { - "ArchStdEvent": "L1D_TLB_PRFM", - "PublicDescription": "The Event counts L1 Data TLB accesses generated by software prefetch or preload memory accesses. Load or Store Instructions can be broken into multiple memory operations. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "L1I_TLB_PRFM", - "PublicDescription": "The Event counts L1 Instruction TLB accesses generated by software preload or prefetch Instructions. This Event counts whether the access hits or misses in the TLB. This Event does not count TLB maintenance operations." - }, - { - "ArchStdEvent": "DTLB_HWUPD", - "PublicDescription": "The Event counts number of memory accesses triggered by a Data translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that this Event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." - }, - { - "ArchStdEvent": "ITLB_HWUPD", - "PublicDescription": "The Event counts number of memory accesses triggered by an Instruction translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." - }, - { - "ArchStdEvent": "DTLB_STEP", - "PublicDescription": "The Event counts number of memory accesses triggered by a demand Data translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that this Event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." - }, - { - "ArchStdEvent": "ITLB_STEP", - "PublicDescription": "The Event counts number of memory accesses triggered by an Instruction translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." - }, - { - "ArchStdEvent": "DTLB_WALK_LARGE", - "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_BLOCK is implemented, then it is an alias for this Event in this family. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "ITLB_WALK_LARGE", - "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_BLOCK Event. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "DTLB_WALK_SMALL", - "PublicDescription": "The Event counts number of Data translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_PAGE Event is implemented, then it is an alias for this Event in this family. Note that partial translations that cause a translation table walk are also counted. Also note that this Event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "ITLB_WALK_SMALL", - "PublicDescription": "The Event counts number of Instruction translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_PAGE Event. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "DTLB_WALK_RW", - "PublicDescription": "The Event counts number of demand Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "ITLB_WALK_RD", - "PublicDescription": "The Event counts number of demand Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "DTLB_WALK_PRFM", - "PublicDescription": "The Event counts number of software prefetches or preloads generated Data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "ArchStdEvent": "ITLB_WALK_PRFM", - "PublicDescription": "The Event counts number of software prefetches or preloads generated Instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this Event does not count walks triggered by TLB maintenance operations." - }, - { - "EventCode": "0x010e", - "EventName": "L1D_TLB_REFILL_RD_PF", - "PublicDescription": "L1 Data TLB refill, Read, prefetch." - }, - { - "EventCode": "0x010f", - "EventName": "L2TLB_PF_REFILL", - "PublicDescription": "L2 Data TLB refill, Read, prefetch. The Event counts MMU refills due to internal PFStream requests." - }, - { - "EventCode": "0x0223", - "EventName": "L1I_TLB_REFILL_RD", - "PublicDescription": "L1 Instruction TLB refills due to Demand miss." - }, - { - "EventCode": "0x0224", - "EventName": "L1I_TLB_REFILL_PRFM", - "PublicDescription": "L1 Instruction TLB refills due to Software prefetch miss." - } -] From 7fe61d0f7d09cc301867b545d3ae58b2f0b95150 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:26 -0700 Subject: [PATCH 189/464] Revert "NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 C2C PMU" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit 4defdaeacc4b11f5de335a9fc2507dc2df77eb4b. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 151 --- drivers/perf/Kconfig | 7 - drivers/perf/Makefile | 1 - drivers/perf/nvidia_t410_c2c_pmu.c | 1061 ----------------- 4 files changed, 1220 deletions(-) delete mode 100644 drivers/perf/nvidia_t410_c2c_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index f81f356debe1f..11fc1c88346a0 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -9,9 +9,6 @@ metrics like memory bandwidth, latency, and utilization: * PCIE * PCIE-TGT * CPU Memory (CMEM) Latency -* NVLink-C2C -* NV-CLink -* NV-DLink PMU Driver ---------- @@ -370,151 +367,3 @@ see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. Example usage:: perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' - -NVLink-C2C PMU --------------- - -This PMU monitors latency events of memory read/write requests that pass through -the NVIDIA Chip-to-Chip (C2C) interface. Bandwidth events are not available -in this PMU, unlike the C2C PMU in Grace (Tegra241 SoC). - -The events and configuration options of this PMU device are available in sysfs, -see /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_. - -The list of events: - - * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. - * IN_RD_REQ: the number of incoming read requests. - * IN_WR_CUM_OUTS: accumulated outstanding request (in cycles) of incoming write requests. - * IN_WR_REQ: the number of incoming write requests. - * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. - * OUT_RD_REQ: the number of outgoing read requests. - * OUT_WR_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing write requests. - * OUT_WR_REQ: the number of outgoing write requests. - * CYCLES: NVLink-C2C interface cycle counts. - -The incoming events count the reads/writes from remote device to the SoC. -The outgoing events count the reads/writes from the SoC to remote device. - -The sysfs /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_/peer -contains the information about the connected device. - -When the C2C interface is connected to GPU(s), the user can use the -"gpu_mask" parameter to filter traffic to/from specific GPU(s). Each bit represents the GPU -index, e.g. "gpu_mask=0x1" corresponds to GPU 0 and "gpu_mask=0x3" is for GPU 0 and 1. -The PMU will monitor all GPUs by default if not specified. - -When connected to another SoC, only the read events are available. - -The events can be used to calculate the average latency of the read/write requests:: - - C2C_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS - - IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ - IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ - - IN_WR_AVG_LATENCY_IN_CYCLES = IN_WR_CUM_OUTS / IN_WR_REQ - IN_WR_AVG_LATENCY_IN_NS = IN_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ - - OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ - OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ - - OUT_WR_AVG_LATENCY_IN_CYCLES = OUT_WR_CUM_OUTS / OUT_WR_REQ - OUT_WR_AVG_LATENCY_IN_NS = OUT_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ - -Example usage: - - * Count incoming traffic from all GPUs connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_req/ - - * Count incoming traffic from GPU 0 connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x1/ - - * Count incoming traffic from GPU 1 connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x2/ - - * Count outgoing traffic to all GPUs connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_req/ - - * Count outgoing traffic to GPU 0 connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x1/ - - * Count outgoing traffic to GPU 1 connected via NVLink-C2C:: - - perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x2/ - -NV-CLink PMU ------------- - -This PMU monitors latency events of memory read requests that pass through -the NV-CLINK interface. Bandwidth events are not available in this PMU. -In Tegra410 SoC, the NV-CLink interface is used to connect to another Tegra410 -SoC and this PMU only counts read traffic. - -The events and configuration options of this PMU device are available in sysfs, -see /sys/bus/event_source/devices/nvidia_nvclink_pmu_. - -The list of events: - - * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. - * IN_RD_REQ: the number of incoming read requests. - * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. - * OUT_RD_REQ: the number of outgoing read requests. - * CYCLES: NV-CLINK interface cycle counts. - -The incoming events count the reads from remote device to the SoC. -The outgoing events count the reads from the SoC to remote device. - -The events can be used to calculate the average latency of the read requests:: - - CLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS - - IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ - IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ - - OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ - OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ - -Example usage: - - * Count incoming read traffic from remote SoC connected via NV-CLINK:: - - perf stat -a -e nvidia_nvclink_pmu_0/in_rd_req/ - - * Count outgoing read traffic to remote SoC connected via NV-CLINK:: - - perf stat -a -e nvidia_nvclink_pmu_0/out_rd_req/ - -NV-DLink PMU ------------- - -This PMU monitors latency events of memory read requests that pass through -the NV-DLINK interface. Bandwidth events are not available in this PMU. -In Tegra410 SoC, this PMU only counts CXL memory read traffic. - -The events and configuration options of this PMU device are available in sysfs, -see /sys/bus/event_source/devices/nvidia_nvdlink_pmu_. - -The list of events: - - * IN_RD_CUM_OUTS: accumulated outstanding read requests (in cycles) to CXL memory. - * IN_RD_REQ: the number of read requests to CXL memory. - * CYCLES: NV-DLINK interface cycle counts. - -The events can be used to calculate the average latency of the read requests:: - - DLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS - - IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ - IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / DLINK_FREQ_IN_GHZ - -Example usage: - - * Count read events to CXL memory:: - - perf stat -a -e '{nvidia_nvdlink_pmu_0/in_rd_req/,nvidia_nvdlink_pmu_0/in_rd_cum_outs/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 7ee36efe6bc0f..9fed3c41d5ea0 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -318,11 +318,4 @@ config NVIDIA_TEGRA410_CMEM_LATENCY_PMU Enable perf support for CPU memory latency counters monitoring on NVIDIA Tegra410 SoC. -config NVIDIA_TEGRA410_C2C_PMU - tristate "NVIDIA Tegra410 C2C PMU" - depends on ARM64 && ACPI - help - Enable perf support for counters in NVIDIA C2C interface of NVIDIA - Tegra410 SoC. - endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index eb8a022dad9a7..4aa6aad393c2d 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -36,4 +36,3 @@ obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o -obj-$(CONFIG_NVIDIA_TEGRA410_C2C_PMU) += nvidia_t410_c2c_pmu.o diff --git a/drivers/perf/nvidia_t410_c2c_pmu.c b/drivers/perf/nvidia_t410_c2c_pmu.c deleted file mode 100644 index 362e0e5f8b24c..0000000000000 --- a/drivers/perf/nvidia_t410_c2c_pmu.c +++ /dev/null @@ -1,1061 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * NVIDIA Tegra410 C2C PMU driver. - * - * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* The C2C interface types in Tegra410. */ -#define C2C_TYPE_NVLINK 0x0 -#define C2C_TYPE_NVCLINK 0x1 -#define C2C_TYPE_NVDLINK 0x2 -#define C2C_TYPE_COUNT 0x3 - -/* The type of the peer device connected to the C2C interface. */ -#define C2C_PEER_TYPE_CPU 0x0 -#define C2C_PEER_TYPE_GPU 0x1 -#define C2C_PEER_TYPE_CXLMEM 0x2 -#define C2C_PEER_TYPE_COUNT 0x3 - -/* The number of peer devices can be connected to the C2C interface. */ -#define C2C_NR_PEER_CPU 0x1 -#define C2C_NR_PEER_GPU 0x2 -#define C2C_NR_PEER_CXLMEM 0x1 -#define C2C_NR_PEER_MAX 0x2 - -/* Number of instances on each interface. */ -#define C2C_NR_INST_NVLINK 14 -#define C2C_NR_INST_NVCLINK 12 -#define C2C_NR_INST_NVDLINK 16 -#define C2C_NR_INST_MAX 16 - -/* Register offsets. */ -#define C2C_CTRL 0x864 -#define C2C_IN_STATUS 0x868 -#define C2C_CYCLE_CNTR 0x86c -#define C2C_IN_RD_CUM_OUTS_CNTR 0x874 -#define C2C_IN_RD_REQ_CNTR 0x87c -#define C2C_IN_WR_CUM_OUTS_CNTR 0x884 -#define C2C_IN_WR_REQ_CNTR 0x88c -#define C2C_OUT_STATUS 0x890 -#define C2C_OUT_RD_CUM_OUTS_CNTR 0x898 -#define C2C_OUT_RD_REQ_CNTR 0x8a0 -#define C2C_OUT_WR_CUM_OUTS_CNTR 0x8a8 -#define C2C_OUT_WR_REQ_CNTR 0x8b0 - -/* C2C_IN_STATUS register field. */ -#define C2C_IN_STATUS_CYCLE_OVF BIT(0) -#define C2C_IN_STATUS_IN_RD_CUM_OUTS_OVF BIT(1) -#define C2C_IN_STATUS_IN_RD_REQ_OVF BIT(2) -#define C2C_IN_STATUS_IN_WR_CUM_OUTS_OVF BIT(3) -#define C2C_IN_STATUS_IN_WR_REQ_OVF BIT(4) - -/* C2C_OUT_STATUS register field. */ -#define C2C_OUT_STATUS_OUT_RD_CUM_OUTS_OVF BIT(0) -#define C2C_OUT_STATUS_OUT_RD_REQ_OVF BIT(1) -#define C2C_OUT_STATUS_OUT_WR_CUM_OUTS_OVF BIT(2) -#define C2C_OUT_STATUS_OUT_WR_REQ_OVF BIT(3) - -/* Events. */ -#define C2C_EVENT_CYCLES 0x0 -#define C2C_EVENT_IN_RD_CUM_OUTS 0x1 -#define C2C_EVENT_IN_RD_REQ 0x2 -#define C2C_EVENT_IN_WR_CUM_OUTS 0x3 -#define C2C_EVENT_IN_WR_REQ 0x4 -#define C2C_EVENT_OUT_RD_CUM_OUTS 0x5 -#define C2C_EVENT_OUT_RD_REQ 0x6 -#define C2C_EVENT_OUT_WR_CUM_OUTS 0x7 -#define C2C_EVENT_OUT_WR_REQ 0x8 - -#define C2C_NUM_EVENTS 0x9 -#define C2C_MASK_EVENT 0xFF -#define C2C_MAX_ACTIVE_EVENTS 32 - -#define C2C_ACTIVE_CPU_MASK 0x0 -#define C2C_ASSOCIATED_CPU_MASK 0x1 - -/* - * Maximum poll count for reading counter value using high-low-high sequence. - */ -#define HILOHI_MAX_POLL 1000 - -static unsigned long nv_c2c_pmu_cpuhp_state; - -/* PMU descriptor. */ - -/* Tracks the events assigned to the PMU for a given logical index. */ -struct nv_c2c_pmu_hw_events { - /* The events that are active. */ - struct perf_event *events[C2C_MAX_ACTIVE_EVENTS]; - - /* - * Each bit indicates a logical counter is being used (or not) for an - * event. - */ - DECLARE_BITMAP(used_ctrs, C2C_MAX_ACTIVE_EVENTS); -}; - -struct nv_c2c_pmu { - struct pmu pmu; - struct device *dev; - struct acpi_device *acpi_dev; - - const char *name; - const char *identifier; - - unsigned int c2c_type; - unsigned int peer_type; - unsigned int socket; - unsigned int nr_inst; - unsigned int nr_peer; - unsigned long peer_insts[C2C_NR_PEER_MAX][BITS_TO_LONGS(C2C_NR_INST_MAX)]; - u32 filter_default; - - struct nv_c2c_pmu_hw_events hw_events; - - cpumask_t associated_cpus; - cpumask_t active_cpu; - - struct hlist_node cpuhp_node; - - struct attribute **formats; - const struct attribute_group *attr_groups[6]; - - void __iomem *base_broadcast; - void __iomem *base[C2C_NR_INST_MAX]; -}; - -#define to_c2c_pmu(p) (container_of(p, struct nv_c2c_pmu, pmu)) - -/* Get event type from perf_event. */ -static inline u32 get_event_type(struct perf_event *event) -{ - return (event->attr.config) & C2C_MASK_EVENT; -} - -static inline u32 get_filter_mask(struct perf_event *event) -{ - u32 filter; - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); - - filter = ((u32)event->attr.config1) & c2c_pmu->filter_default; - if (filter == 0) - filter = c2c_pmu->filter_default; - - return filter; -} - -/* PMU operations. */ - -static int nv_c2c_pmu_get_event_idx(struct nv_c2c_pmu_hw_events *hw_events, - struct perf_event *event) -{ - u32 idx; - - idx = find_first_zero_bit(hw_events->used_ctrs, C2C_MAX_ACTIVE_EVENTS); - if (idx >= C2C_MAX_ACTIVE_EVENTS) - return -EAGAIN; - - set_bit(idx, hw_events->used_ctrs); - - return idx; -} - -static bool -nv_c2c_pmu_validate_event(struct pmu *pmu, - struct nv_c2c_pmu_hw_events *hw_events, - struct perf_event *event) -{ - if (is_software_event(event)) - return true; - - /* Reject groups spanning multiple HW PMUs. */ - if (event->pmu != pmu) - return false; - - return nv_c2c_pmu_get_event_idx(hw_events, event) >= 0; -} - -/* - * Make sure the group of events can be scheduled at once - * on the PMU. - */ -static bool nv_c2c_pmu_validate_group(struct perf_event *event) -{ - struct perf_event *sibling, *leader = event->group_leader; - struct nv_c2c_pmu_hw_events fake_hw_events; - - if (event->group_leader == event) - return true; - - memset(&fake_hw_events, 0, sizeof(fake_hw_events)); - - if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, leader)) - return false; - - for_each_sibling_event(sibling, leader) { - if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, - sibling)) - return false; - } - - return nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, event); -} - -static int nv_c2c_pmu_event_init(struct perf_event *event) -{ - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); - struct hw_perf_event *hwc = &event->hw; - u32 event_type = get_event_type(event); - - if (event->attr.type != event->pmu->type || - event_type >= C2C_NUM_EVENTS) - return -ENOENT; - - /* - * Following other "uncore" PMUs, we do not support sampling mode or - * attach to a task (per-process mode). - */ - if (is_sampling_event(event)) { - dev_dbg(c2c_pmu->pmu.dev, "Can't support sampling events\n"); - return -EOPNOTSUPP; - } - - if (event->cpu < 0 || event->attach_state & PERF_ATTACH_TASK) { - dev_dbg(c2c_pmu->pmu.dev, "Can't support per-task counters\n"); - return -EINVAL; - } - - /* - * Make sure the CPU assignment is on one of the CPUs associated with - * this PMU. - */ - if (!cpumask_test_cpu(event->cpu, &c2c_pmu->associated_cpus)) { - dev_dbg(c2c_pmu->pmu.dev, - "Requested cpu is not associated with the PMU\n"); - return -EINVAL; - } - - /* Enforce the current active CPU to handle the events in this PMU. */ - event->cpu = cpumask_first(&c2c_pmu->active_cpu); - if (event->cpu >= nr_cpu_ids) - return -EINVAL; - - if (!nv_c2c_pmu_validate_group(event)) - return -EINVAL; - - hwc->idx = -1; - hwc->config = event_type; - - return 0; -} - -/* - * Read 64-bit register as a pair of 32-bit registers using hi-lo-hi sequence. - */ -static u64 read_reg64_hilohi(const void __iomem *addr, u32 max_poll_count) -{ - u32 val_lo, val_hi; - u64 val; - - /* Use high-low-high sequence to avoid tearing */ - do { - if (max_poll_count-- == 0) { - pr_err("NV C2C PMU: timeout hi-low-high sequence\n"); - return 0; - } - - val_hi = readl(addr + 4); - val_lo = readl(addr); - } while (val_hi != readl(addr + 4)); - - val = (((u64)val_hi << 32) | val_lo); - - return val; -} - -static void nv_c2c_pmu_check_status(struct nv_c2c_pmu *c2c_pmu, u32 instance) -{ - u32 in_status, out_status; - - in_status = readl(c2c_pmu->base[instance] + C2C_IN_STATUS); - out_status = readl(c2c_pmu->base[instance] + C2C_OUT_STATUS); - - if (in_status || out_status) - dev_warn(c2c_pmu->dev, - "C2C PMU overflow in: 0x%x, out: 0x%x\n", - in_status, out_status); -} - -static u32 nv_c2c_ctr_offset[C2C_NUM_EVENTS] = { - [C2C_EVENT_CYCLES] = C2C_CYCLE_CNTR, - [C2C_EVENT_IN_RD_CUM_OUTS] = C2C_IN_RD_CUM_OUTS_CNTR, - [C2C_EVENT_IN_RD_REQ] = C2C_IN_RD_REQ_CNTR, - [C2C_EVENT_IN_WR_CUM_OUTS] = C2C_IN_WR_CUM_OUTS_CNTR, - [C2C_EVENT_IN_WR_REQ] = C2C_IN_WR_REQ_CNTR, - [C2C_EVENT_OUT_RD_CUM_OUTS] = C2C_OUT_RD_CUM_OUTS_CNTR, - [C2C_EVENT_OUT_RD_REQ] = C2C_OUT_RD_REQ_CNTR, - [C2C_EVENT_OUT_WR_CUM_OUTS] = C2C_OUT_WR_CUM_OUTS_CNTR, - [C2C_EVENT_OUT_WR_REQ] = C2C_OUT_WR_REQ_CNTR, -}; - -static u64 nv_c2c_pmu_read_counter(struct perf_event *event) -{ - u32 ctr_id, ctr_offset, filter_mask, filter_idx, inst_idx; - unsigned long *inst_mask; - DECLARE_BITMAP(filter_bitmap, C2C_NR_PEER_MAX); - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); - u64 val = 0; - - filter_mask = get_filter_mask(event); - bitmap_from_arr32(filter_bitmap, &filter_mask, c2c_pmu->nr_peer); - - ctr_id = event->hw.config; - ctr_offset = nv_c2c_ctr_offset[ctr_id]; - - for_each_set_bit(filter_idx, filter_bitmap, c2c_pmu->nr_peer) { - inst_mask = c2c_pmu->peer_insts[filter_idx]; - for_each_set_bit(inst_idx, inst_mask, c2c_pmu->nr_inst) { - nv_c2c_pmu_check_status(c2c_pmu, inst_idx); - - /* - * Each instance share same clock and the driver always - * enables all instances. So we can use the counts from - * one instance for cycle counter. - */ - if (ctr_id == C2C_EVENT_CYCLES) - return read_reg64_hilohi( - c2c_pmu->base[inst_idx] + ctr_offset, - HILOHI_MAX_POLL); - - /* - * For other events, sum up the counts from all instances. - */ - val += read_reg64_hilohi( - c2c_pmu->base[inst_idx] + ctr_offset, - HILOHI_MAX_POLL); - } - } - - return val; -} - -static void nv_c2c_pmu_event_update(struct perf_event *event) -{ - struct hw_perf_event *hwc = &event->hw; - u64 prev, now; - - do { - prev = local64_read(&hwc->prev_count); - now = nv_c2c_pmu_read_counter(event); - } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); - - local64_add(now - prev, &event->count); -} - -static void nv_c2c_pmu_start(struct perf_event *event, int pmu_flags) -{ - event->hw.state = 0; -} - -static void nv_c2c_pmu_stop(struct perf_event *event, int pmu_flags) -{ - event->hw.state |= PERF_HES_STOPPED; -} - -static int nv_c2c_pmu_add(struct perf_event *event, int flags) -{ - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); - struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; - struct hw_perf_event *hwc = &event->hw; - int idx; - - if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), - &c2c_pmu->associated_cpus))) - return -ENOENT; - - idx = nv_c2c_pmu_get_event_idx(hw_events, event); - if (idx < 0) - return idx; - - hw_events->events[idx] = event; - hwc->idx = idx; - hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; - - if (flags & PERF_EF_START) - nv_c2c_pmu_start(event, PERF_EF_RELOAD); - - /* Propagate changes to the userspace mapping. */ - perf_event_update_userpage(event); - - return 0; -} - -static void nv_c2c_pmu_del(struct perf_event *event, int flags) -{ - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); - struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; - struct hw_perf_event *hwc = &event->hw; - int idx = hwc->idx; - - nv_c2c_pmu_stop(event, PERF_EF_UPDATE); - - hw_events->events[idx] = NULL; - - clear_bit(idx, hw_events->used_ctrs); - - perf_event_update_userpage(event); -} - -static void nv_c2c_pmu_read(struct perf_event *event) -{ - nv_c2c_pmu_event_update(event); -} - -static void nv_c2c_pmu_enable(struct pmu *pmu) -{ - void __iomem *bcast; - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); - - /* Check if any filter is enabled. */ - if (bitmap_empty(c2c_pmu->hw_events.used_ctrs, C2C_MAX_ACTIVE_EVENTS)) - return; - - /* Enable all the counters. */ - bcast = c2c_pmu->base_broadcast; - writel(0x1UL, bcast + C2C_CTRL); -} - -static void nv_c2c_pmu_disable(struct pmu *pmu) -{ - unsigned int idx; - void __iomem *bcast; - struct perf_event *event; - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); - - /* Disable all the counters. */ - bcast = c2c_pmu->base_broadcast; - writel(0x0UL, bcast + C2C_CTRL); - - /* - * The counters will start from 0 again on restart. - * Update the events immediately to avoid losing the counts. - */ - for_each_set_bit(idx, c2c_pmu->hw_events.used_ctrs, - C2C_MAX_ACTIVE_EVENTS) { - event = c2c_pmu->hw_events.events[idx]; - - if (!event) - continue; - - nv_c2c_pmu_event_update(event); - - local64_set(&event->hw.prev_count, 0ULL); - } -} - -/* PMU identifier attribute. */ - -static ssize_t nv_c2c_pmu_identifier_show(struct device *dev, - struct device_attribute *attr, - char *page) -{ - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); - - return sysfs_emit(page, "%s\n", c2c_pmu->identifier); -} - -static struct device_attribute nv_c2c_pmu_identifier_attr = - __ATTR(identifier, 0444, nv_c2c_pmu_identifier_show, NULL); - -static struct attribute *nv_c2c_pmu_identifier_attrs[] = { - &nv_c2c_pmu_identifier_attr.attr, - NULL, -}; - -static struct attribute_group nv_c2c_pmu_identifier_attr_group = { - .attrs = nv_c2c_pmu_identifier_attrs, -}; - -/* Peer attribute. */ - -static ssize_t nv_c2c_pmu_peer_show(struct device *dev, - struct device_attribute *attr, - char *page) -{ - const char *peer_type[C2C_PEER_TYPE_COUNT] = { - [C2C_PEER_TYPE_CPU] = "cpu", - [C2C_PEER_TYPE_GPU] = "gpu", - [C2C_PEER_TYPE_CXLMEM] = "cxlmem", - }; - - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); - return sysfs_emit(page, "nr_%s=%u\n", peer_type[c2c_pmu->peer_type], - c2c_pmu->nr_peer); -} - -static struct device_attribute nv_c2c_pmu_peer_attr = - __ATTR(peer, 0444, nv_c2c_pmu_peer_show, NULL); - -static struct attribute *nv_c2c_pmu_peer_attrs[] = { - &nv_c2c_pmu_peer_attr.attr, - NULL, -}; - -static struct attribute_group nv_c2c_pmu_peer_attr_group = { - .attrs = nv_c2c_pmu_peer_attrs, -}; - -/* Format attributes. */ - -#define NV_C2C_PMU_EXT_ATTR(_name, _func, _config) \ - (&((struct dev_ext_attribute[]){ \ - { \ - .attr = __ATTR(_name, 0444, _func, NULL), \ - .var = (void *)_config \ - } \ - })[0].attr.attr) - -#define NV_C2C_PMU_FORMAT_ATTR(_name, _config) \ - NV_C2C_PMU_EXT_ATTR(_name, device_show_string, _config) - -#define NV_C2C_PMU_FORMAT_EVENT_ATTR \ - NV_C2C_PMU_FORMAT_ATTR(event, "config:0-3") - -static struct attribute *nv_c2c_nvlink_pmu_formats[] = { - NV_C2C_PMU_FORMAT_EVENT_ATTR, - NV_C2C_PMU_FORMAT_ATTR(gpu_mask, "config1:0-1"), - NULL, -}; - -static struct attribute *nv_c2c_pmu_formats[] = { - NV_C2C_PMU_FORMAT_EVENT_ATTR, - NULL, -}; - -static struct attribute_group * -nv_c2c_pmu_alloc_format_attr_group(struct nv_c2c_pmu *c2c_pmu) -{ - struct attribute_group *format_group; - struct device *dev = c2c_pmu->dev; - - format_group = - devm_kzalloc(dev, sizeof(struct attribute_group), GFP_KERNEL); - if (!format_group) - return NULL; - - format_group->name = "format"; - format_group->attrs = c2c_pmu->formats; - - return format_group; -} - -/* Event attributes. */ - -static ssize_t nv_c2c_pmu_sysfs_event_show(struct device *dev, - struct device_attribute *attr, - char *buf) -{ - struct perf_pmu_events_attr *pmu_attr; - - pmu_attr = container_of(attr, typeof(*pmu_attr), attr); - return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); -} - -#define NV_C2C_PMU_EVENT_ATTR(_name, _config) \ - PMU_EVENT_ATTR_ID(_name, nv_c2c_pmu_sysfs_event_show, _config) - -static struct attribute *nv_c2c_pmu_events[] = { - NV_C2C_PMU_EVENT_ATTR(cycles, C2C_EVENT_CYCLES), - NV_C2C_PMU_EVENT_ATTR(in_rd_cum_outs, C2C_EVENT_IN_RD_CUM_OUTS), - NV_C2C_PMU_EVENT_ATTR(in_rd_req, C2C_EVENT_IN_RD_REQ), - NV_C2C_PMU_EVENT_ATTR(in_wr_cum_outs, C2C_EVENT_IN_WR_CUM_OUTS), - NV_C2C_PMU_EVENT_ATTR(in_wr_req, C2C_EVENT_IN_WR_REQ), - NV_C2C_PMU_EVENT_ATTR(out_rd_cum_outs, C2C_EVENT_OUT_RD_CUM_OUTS), - NV_C2C_PMU_EVENT_ATTR(out_rd_req, C2C_EVENT_OUT_RD_REQ), - NV_C2C_PMU_EVENT_ATTR(out_wr_cum_outs, C2C_EVENT_OUT_WR_CUM_OUTS), - NV_C2C_PMU_EVENT_ATTR(out_wr_req, C2C_EVENT_OUT_WR_REQ), - NULL -}; - -static umode_t -nv_c2c_pmu_event_attr_is_visible(struct kobject *kobj, struct attribute *attr, - int unused) -{ - struct device *dev = kobj_to_dev(kobj); - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); - struct perf_pmu_events_attr *eattr; - - eattr = container_of(attr, typeof(*eattr), attr.attr); - - if (c2c_pmu->c2c_type == C2C_TYPE_NVDLINK) { - /* Only incoming reads are available. */ - switch (eattr->id) { - case C2C_EVENT_IN_WR_CUM_OUTS: - case C2C_EVENT_IN_WR_REQ: - case C2C_EVENT_OUT_RD_CUM_OUTS: - case C2C_EVENT_OUT_RD_REQ: - case C2C_EVENT_OUT_WR_CUM_OUTS: - case C2C_EVENT_OUT_WR_REQ: - return 0; - default: - return attr->mode; - } - } else { - /* Hide the write events if C2C connected to another SoC. */ - if (c2c_pmu->peer_type == C2C_PEER_TYPE_CPU) { - switch (eattr->id) { - case C2C_EVENT_IN_WR_CUM_OUTS: - case C2C_EVENT_IN_WR_REQ: - case C2C_EVENT_OUT_WR_CUM_OUTS: - case C2C_EVENT_OUT_WR_REQ: - return 0; - default: - return attr->mode; - } - } - } - - return attr->mode; -} - -static const struct attribute_group nv_c2c_pmu_events_group = { - .name = "events", - .attrs = nv_c2c_pmu_events, - .is_visible = nv_c2c_pmu_event_attr_is_visible, -}; - -/* Cpumask attributes. */ - -static ssize_t nv_c2c_pmu_cpumask_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct pmu *pmu = dev_get_drvdata(dev); - struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); - struct dev_ext_attribute *eattr = - container_of(attr, struct dev_ext_attribute, attr); - unsigned long mask_id = (unsigned long)eattr->var; - const cpumask_t *cpumask; - - switch (mask_id) { - case C2C_ACTIVE_CPU_MASK: - cpumask = &c2c_pmu->active_cpu; - break; - case C2C_ASSOCIATED_CPU_MASK: - cpumask = &c2c_pmu->associated_cpus; - break; - default: - return 0; - } - return cpumap_print_to_pagebuf(true, buf, cpumask); -} - -#define NV_C2C_PMU_CPUMASK_ATTR(_name, _config) \ - NV_C2C_PMU_EXT_ATTR(_name, nv_c2c_pmu_cpumask_show, \ - (unsigned long)_config) - -static struct attribute *nv_c2c_pmu_cpumask_attrs[] = { - NV_C2C_PMU_CPUMASK_ATTR(cpumask, C2C_ACTIVE_CPU_MASK), - NV_C2C_PMU_CPUMASK_ATTR(associated_cpus, C2C_ASSOCIATED_CPU_MASK), - NULL, -}; - -static const struct attribute_group nv_c2c_pmu_cpumask_attr_group = { - .attrs = nv_c2c_pmu_cpumask_attrs, -}; - -/* Per PMU device attribute groups. */ - -static int nv_c2c_pmu_alloc_attr_groups(struct nv_c2c_pmu *c2c_pmu) -{ - const struct attribute_group **attr_groups = c2c_pmu->attr_groups; - - attr_groups[0] = nv_c2c_pmu_alloc_format_attr_group(c2c_pmu); - attr_groups[1] = &nv_c2c_pmu_events_group; - attr_groups[2] = &nv_c2c_pmu_cpumask_attr_group; - attr_groups[3] = &nv_c2c_pmu_identifier_attr_group; - attr_groups[4] = &nv_c2c_pmu_peer_attr_group; - - if (!attr_groups[0]) - return -ENOMEM; - - return 0; -} - -static int nv_c2c_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) -{ - struct nv_c2c_pmu *c2c_pmu = - hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); - - if (!cpumask_test_cpu(cpu, &c2c_pmu->associated_cpus)) - return 0; - - /* If the PMU is already managed, there is nothing to do */ - if (!cpumask_empty(&c2c_pmu->active_cpu)) - return 0; - - /* Use this CPU for event counting */ - cpumask_set_cpu(cpu, &c2c_pmu->active_cpu); - - return 0; -} - -static int nv_c2c_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) -{ - unsigned int dst; - - struct nv_c2c_pmu *c2c_pmu = - hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); - - /* Nothing to do if this CPU doesn't own the PMU */ - if (!cpumask_test_and_clear_cpu(cpu, &c2c_pmu->active_cpu)) - return 0; - - /* Choose a new CPU to migrate ownership of the PMU to */ - dst = cpumask_any_and_but(&c2c_pmu->associated_cpus, - cpu_online_mask, cpu); - if (dst >= nr_cpu_ids) - return 0; - - /* Use this CPU for event counting */ - perf_pmu_migrate_context(&c2c_pmu->pmu, cpu, dst); - cpumask_set_cpu(dst, &c2c_pmu->active_cpu); - - return 0; -} - -static int nv_c2c_pmu_get_cpus(struct nv_c2c_pmu *c2c_pmu) -{ - int ret = 0, socket = c2c_pmu->socket, cpu; - - for_each_possible_cpu(cpu) { - if (cpu_to_node(cpu) == socket) - cpumask_set_cpu(cpu, &c2c_pmu->associated_cpus); - } - - if (cpumask_empty(&c2c_pmu->associated_cpus)) { - dev_dbg(c2c_pmu->dev, - "No cpu associated with C2C PMU socket-%u\n", socket); - ret = -ENODEV; - } - - return ret; -} - -static int nv_c2c_pmu_init_socket(struct nv_c2c_pmu *c2c_pmu) -{ - const char *uid_str; - int ret, socket; - - uid_str = acpi_device_uid(c2c_pmu->acpi_dev); - if (!uid_str) { - ret = -ENODEV; - goto fail; - } - - ret = kstrtou32(uid_str, 0, &socket); - if (ret) - goto fail; - - c2c_pmu->socket = socket; - return 0; - -fail: - dev_err(c2c_pmu->dev, "Failed to initialize socket\n"); - return ret; -} - -static int nv_c2c_pmu_init_id(struct nv_c2c_pmu *c2c_pmu) -{ - const char *name_fmt[C2C_TYPE_COUNT] = { - [C2C_TYPE_NVLINK] = "nvidia_nvlink_c2c_pmu_%u", - [C2C_TYPE_NVCLINK] = "nvidia_nvclink_pmu_%u", - [C2C_TYPE_NVDLINK] = "nvidia_nvdlink_pmu_%u", - }; - - char *name; - int ret; - - name = devm_kasprintf(c2c_pmu->dev, GFP_KERNEL, - name_fmt[c2c_pmu->c2c_type], c2c_pmu->socket); - if (!name) { - ret = -ENOMEM; - goto fail; - } - - c2c_pmu->name = name; - - c2c_pmu->identifier = acpi_device_hid(c2c_pmu->acpi_dev); - - return 0; - -fail: - dev_err(c2c_pmu->dev, "Failed to initialize name\n"); - return ret; -} - -static int nv_c2c_pmu_init_filter(struct nv_c2c_pmu *c2c_pmu) -{ - u32 cpu_en = 0; - struct device *dev = c2c_pmu->dev; - - if (c2c_pmu->c2c_type == C2C_TYPE_NVDLINK) { - c2c_pmu->peer_type = C2C_PEER_TYPE_CXLMEM; - - c2c_pmu->nr_inst = C2C_NR_INST_NVDLINK; - c2c_pmu->peer_insts[0][0] = (1UL << c2c_pmu->nr_inst) - 1; - - c2c_pmu->nr_peer = C2C_NR_PEER_CXLMEM; - c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; - - c2c_pmu->formats = nv_c2c_pmu_formats; - - return 0; - } - - c2c_pmu->nr_inst = (c2c_pmu->c2c_type == C2C_TYPE_NVLINK) ? - C2C_NR_INST_NVLINK : C2C_NR_INST_NVCLINK; - - if (device_property_read_u32(dev, "cpu_en_mask", &cpu_en)) - dev_dbg(dev, "no cpu_en_mask property\n"); - - if (cpu_en) { - c2c_pmu->peer_type = C2C_PEER_TYPE_CPU; - - /* Fill peer_insts bitmap with instances connected to peer CPU. */ - bitmap_from_arr32(c2c_pmu->peer_insts[0], &cpu_en, - c2c_pmu->nr_inst); - - c2c_pmu->nr_peer = 1; - c2c_pmu->formats = nv_c2c_pmu_formats; - } else { - u32 i; - u32 gpu_en = 0; - const char *props[C2C_NR_PEER_MAX] = { - "gpu0_en_mask", "gpu1_en_mask" - }; - - for (i = 0; i < C2C_NR_PEER_MAX; i++) { - if (device_property_read_u32(dev, props[i], &gpu_en)) - dev_dbg(dev, "no %s property\n", props[i]); - - if (gpu_en) { - /* Fill peer_insts bitmap with instances connected to peer GPU. */ - bitmap_from_arr32(c2c_pmu->peer_insts[i], &gpu_en, - c2c_pmu->nr_inst); - - c2c_pmu->nr_peer++; - } - } - - if (c2c_pmu->nr_peer == 0) { - dev_err(dev, "No GPU is enabled\n"); - return -EINVAL; - } - - c2c_pmu->peer_type = C2C_PEER_TYPE_GPU; - c2c_pmu->formats = nv_c2c_nvlink_pmu_formats; - } - - c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; - - return 0; -} - -static void *nv_c2c_pmu_init_pmu(struct platform_device *pdev) -{ - int ret; - struct nv_c2c_pmu *c2c_pmu; - struct acpi_device *acpi_dev; - struct device *dev = &pdev->dev; - - acpi_dev = ACPI_COMPANION(dev); - if (!acpi_dev) - return ERR_PTR(-ENODEV); - - c2c_pmu = devm_kzalloc(dev, sizeof(*c2c_pmu), GFP_KERNEL); - if (!c2c_pmu) - return ERR_PTR(-ENOMEM); - - c2c_pmu->dev = dev; - c2c_pmu->acpi_dev = acpi_dev; - c2c_pmu->c2c_type = (unsigned int)(unsigned long)device_get_match_data(dev); - platform_set_drvdata(pdev, c2c_pmu); - - ret = nv_c2c_pmu_init_socket(c2c_pmu); - if (ret) - goto done; - - ret = nv_c2c_pmu_init_id(c2c_pmu); - if (ret) - goto done; - - ret = nv_c2c_pmu_init_filter(c2c_pmu); - if (ret) - goto done; - -done: - if (ret) - return ERR_PTR(ret); - - return c2c_pmu; -} - -static int nv_c2c_pmu_init_mmio(struct nv_c2c_pmu *c2c_pmu) -{ - int i; - struct device *dev = c2c_pmu->dev; - struct platform_device *pdev = to_platform_device(dev); - - /* Map the address of all the instances. */ - for (i = 0; i < c2c_pmu->nr_inst; i++) { - c2c_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); - if (IS_ERR(c2c_pmu->base[i])) { - dev_err(dev, "Failed map address for instance %d\n", i); - return PTR_ERR(c2c_pmu->base[i]); - } - } - - /* Map broadcast address. */ - c2c_pmu->base_broadcast = devm_platform_ioremap_resource(pdev, - c2c_pmu->nr_inst); - if (IS_ERR(c2c_pmu->base_broadcast)) { - dev_err(dev, "Failed map broadcast address\n"); - return PTR_ERR(c2c_pmu->base_broadcast); - } - - return 0; -} - -static int nv_c2c_pmu_register_pmu(struct nv_c2c_pmu *c2c_pmu) -{ - int ret; - - ret = cpuhp_state_add_instance(nv_c2c_pmu_cpuhp_state, - &c2c_pmu->cpuhp_node); - if (ret) { - dev_err(c2c_pmu->dev, "Error %d registering hotplug\n", ret); - return ret; - } - - c2c_pmu->pmu = (struct pmu) { - .parent = c2c_pmu->dev, - .task_ctx_nr = perf_invalid_context, - .pmu_enable = nv_c2c_pmu_enable, - .pmu_disable = nv_c2c_pmu_disable, - .event_init = nv_c2c_pmu_event_init, - .add = nv_c2c_pmu_add, - .del = nv_c2c_pmu_del, - .start = nv_c2c_pmu_start, - .stop = nv_c2c_pmu_stop, - .read = nv_c2c_pmu_read, - .attr_groups = c2c_pmu->attr_groups, - .capabilities = PERF_PMU_CAP_NO_EXCLUDE | - PERF_PMU_CAP_NO_INTERRUPT, - }; - - ret = perf_pmu_register(&c2c_pmu->pmu, c2c_pmu->name, -1); - if (ret) { - dev_err(c2c_pmu->dev, "Failed to register C2C PMU: %d\n", ret); - cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, - &c2c_pmu->cpuhp_node); - return ret; - } - - return 0; -} - -static int nv_c2c_pmu_probe(struct platform_device *pdev) -{ - int ret; - struct nv_c2c_pmu *c2c_pmu; - - c2c_pmu = nv_c2c_pmu_init_pmu(pdev); - if (IS_ERR(c2c_pmu)) - return PTR_ERR(c2c_pmu); - - ret = nv_c2c_pmu_init_mmio(c2c_pmu); - if (ret) - return ret; - - ret = nv_c2c_pmu_get_cpus(c2c_pmu); - if (ret) - return ret; - - ret = nv_c2c_pmu_alloc_attr_groups(c2c_pmu); - if (ret) - return ret; - - ret = nv_c2c_pmu_register_pmu(c2c_pmu); - if (ret) - return ret; - - dev_dbg(c2c_pmu->dev, "Registered %s PMU\n", c2c_pmu->name); - - return 0; -} - -static void nv_c2c_pmu_device_remove(struct platform_device *pdev) -{ - struct nv_c2c_pmu *c2c_pmu = platform_get_drvdata(pdev); - - perf_pmu_unregister(&c2c_pmu->pmu); - cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, &c2c_pmu->cpuhp_node); -} - -static const struct acpi_device_id nv_c2c_pmu_acpi_match[] = { - { "NVDA2023", (kernel_ulong_t)C2C_TYPE_NVLINK }, - { "NVDA2022", (kernel_ulong_t)C2C_TYPE_NVCLINK }, - { "NVDA2020", (kernel_ulong_t)C2C_TYPE_NVDLINK }, - { } -}; -MODULE_DEVICE_TABLE(acpi, nv_c2c_pmu_acpi_match); - -static struct platform_driver nv_c2c_pmu_driver = { - .driver = { - .name = "nvidia-t410-c2c-pmu", - .acpi_match_table = ACPI_PTR(nv_c2c_pmu_acpi_match), - .suppress_bind_attrs = true, - }, - .probe = nv_c2c_pmu_probe, - .remove = nv_c2c_pmu_device_remove, -}; - -static int __init nv_c2c_pmu_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, - "perf/nvidia/c2c:online", - nv_c2c_pmu_online_cpu, - nv_c2c_pmu_cpu_teardown); - if (ret < 0) - return ret; - - nv_c2c_pmu_cpuhp_state = ret; - return platform_driver_register(&nv_c2c_pmu_driver); -} - -static void __exit nv_c2c_pmu_exit(void) -{ - platform_driver_unregister(&nv_c2c_pmu_driver); - cpuhp_remove_multi_state(nv_c2c_pmu_cpuhp_state); -} - -module_init(nv_c2c_pmu_init); -module_exit(nv_c2c_pmu_exit); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("NVIDIA Tegra410 C2C PMU driver"); -MODULE_AUTHOR("Besar Wicaksono "); From 3ee027167634174e581fc2b51069917cf34eafc3 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:38 -0700 Subject: [PATCH 190/464] Revert "NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 CPU Memory Latency PMU" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit eff2e93346d1a67c66ef96abdf6263c5bba111f5. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 25 - drivers/perf/Kconfig | 7 - drivers/perf/Makefile | 1 - drivers/perf/nvidia_t410_cmem_latency_pmu.c | 727 ------------------ 4 files changed, 760 deletions(-) delete mode 100644 drivers/perf/nvidia_t410_cmem_latency_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 11fc1c88346a0..07dc447eead7c 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -8,7 +8,6 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE * PCIE-TGT -* CPU Memory (CMEM) Latency PMU Driver ---------- @@ -343,27 +342,3 @@ Example usage: 0x10000 to 0x100FF on socket 0's PCIE RC-1:: perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ - -CPU Memory (CMEM) Latency PMU ------------------------------ - -This PMU monitors latency events of memory read requests to local -CPU DRAM: - - * RD_REQ counters: count read requests (32B per request). - * RD_CUM_OUTS counters: accumulated outstanding request counter, which track - how many cycles the read requests are in flight. - * CYCLES counter: counts the number of elapsed cycles. - -The average latency is calculated as:: - - FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS - AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ - AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ - -The events and configuration options of this PMU device are described in sysfs, -see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. - -Example usage:: - - perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 9fed3c41d5ea0..638321fc9800c 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -311,11 +311,4 @@ config MARVELL_PEM_PMU Enable support for PCIe Interface performance monitoring on Marvell platform. -config NVIDIA_TEGRA410_CMEM_LATENCY_PMU - tristate "NVIDIA Tegra410 CPU Memory Latency PMU" - depends on ARM64 - help - Enable perf support for CPU memory latency counters monitoring on - NVIDIA Tegra410 SoC. - endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index 4aa6aad393c2d..ea52711a87e32 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -35,4 +35,3 @@ obj-$(CONFIG_DWC_PCIE_PMU) += dwc_pcie_pmu.o obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o -obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o diff --git a/drivers/perf/nvidia_t410_cmem_latency_pmu.c b/drivers/perf/nvidia_t410_cmem_latency_pmu.c deleted file mode 100644 index 9b466581c8fcc..0000000000000 --- a/drivers/perf/nvidia_t410_cmem_latency_pmu.c +++ /dev/null @@ -1,727 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * NVIDIA Tegra410 CPU Memory (CMEM) Latency PMU driver. - * - * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#define NUM_INSTANCES 14 -#define BCAST(pmu) pmu->base[NUM_INSTANCES] - -/* Register offsets. */ -#define CG_CTRL 0x800 -#define CTRL 0x808 -#define STATUS 0x810 -#define CYCLE_CNTR 0x818 -#define MC0_REQ_CNTR 0x820 -#define MC0_AOR_CNTR 0x830 -#define MC1_REQ_CNTR 0x838 -#define MC1_AOR_CNTR 0x848 -#define MC2_REQ_CNTR 0x850 -#define MC2_AOR_CNTR 0x860 - -/* CTRL values. */ -#define CTRL_DISABLE 0x0ULL -#define CTRL_ENABLE 0x1ULL -#define CTRL_CLR 0x2ULL - -/* CG_CTRL values. */ -#define CG_CTRL_DISABLE 0x0ULL -#define CG_CTRL_ENABLE 0x1ULL - -/* STATUS register field. */ -#define STATUS_CYCLE_OVF BIT(0) -#define STATUS_MC0_AOR_OVF BIT(1) -#define STATUS_MC0_REQ_OVF BIT(3) -#define STATUS_MC1_AOR_OVF BIT(4) -#define STATUS_MC1_REQ_OVF BIT(6) -#define STATUS_MC2_AOR_OVF BIT(7) -#define STATUS_MC2_REQ_OVF BIT(9) - -/* Events. */ -#define EVENT_CYCLES 0x0 -#define EVENT_REQ 0x1 -#define EVENT_AOR 0x2 - -#define NUM_EVENTS 0x3 -#define MASK_EVENT 0x3 -#define MAX_ACTIVE_EVENTS 32 - -#define ACTIVE_CPU_MASK 0x0 -#define ASSOCIATED_CPU_MASK 0x1 - -static unsigned long cmem_lat_pmu_cpuhp_state; - -struct cmem_lat_pmu_hw_events { - struct perf_event *events[MAX_ACTIVE_EVENTS]; - DECLARE_BITMAP(used_ctrs, MAX_ACTIVE_EVENTS); -}; - -struct cmem_lat_pmu { - struct pmu pmu; - struct device *dev; - const char *name; - const char *identifier; - void __iomem *base[NUM_INSTANCES + 1]; - cpumask_t associated_cpus; - cpumask_t active_cpu; - struct hlist_node node; - struct cmem_lat_pmu_hw_events hw_events; -}; - -#define to_cmem_lat_pmu(p) \ - container_of(p, struct cmem_lat_pmu, pmu) - - -/* Get event type from perf_event. */ -static inline u32 get_event_type(struct perf_event *event) -{ - return (event->attr.config) & MASK_EVENT; -} - -/* PMU operations. */ -static int cmem_lat_pmu_get_event_idx(struct cmem_lat_pmu_hw_events *hw_events, - struct perf_event *event) -{ - unsigned int idx; - - idx = find_first_zero_bit(hw_events->used_ctrs, MAX_ACTIVE_EVENTS); - if (idx >= MAX_ACTIVE_EVENTS) - return -EAGAIN; - - set_bit(idx, hw_events->used_ctrs); - - return idx; -} - -static bool cmem_lat_pmu_validate_event(struct pmu *pmu, - struct cmem_lat_pmu_hw_events *hw_events, - struct perf_event *event) -{ - if (is_software_event(event)) - return true; - - /* Reject groups spanning multiple HW PMUs. */ - if (event->pmu != pmu) - return false; - - return (cmem_lat_pmu_get_event_idx(hw_events, event) >= 0); -} - -/* - * Make sure the group of events can be scheduled at once - * on the PMU. - */ -static bool cmem_lat_pmu_validate_group(struct perf_event *event) -{ - struct perf_event *sibling, *leader = event->group_leader; - struct cmem_lat_pmu_hw_events fake_hw_events; - - if (event->group_leader == event) - return true; - - memset(&fake_hw_events, 0, sizeof(fake_hw_events)); - - if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, leader)) - return false; - - for_each_sibling_event(sibling, leader) { - if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, - sibling)) - return false; - } - - return cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, event); -} - -static int cmem_lat_pmu_event_init(struct perf_event *event) -{ - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct hw_perf_event *hwc = &event->hw; - u32 event_type = get_event_type(event); - - if (event->attr.type != event->pmu->type || - event_type >= NUM_EVENTS) - return -ENOENT; - - /* - * Following other "uncore" PMUs, we do not support sampling mode or - * attach to a task (per-process mode). - */ - if (is_sampling_event(event)) { - dev_dbg(cmem_lat_pmu->pmu.dev, - "Can't support sampling events\n"); - return -EOPNOTSUPP; - } - - if (event->cpu < 0 || event->attach_state & PERF_ATTACH_TASK) { - dev_dbg(cmem_lat_pmu->pmu.dev, - "Can't support per-task counters\n"); - return -EINVAL; - } - - /* - * Make sure the CPU assignment is on one of the CPUs associated with - * this PMU. - */ - if (!cpumask_test_cpu(event->cpu, &cmem_lat_pmu->associated_cpus)) { - dev_dbg(cmem_lat_pmu->pmu.dev, - "Requested cpu is not associated with the PMU\n"); - return -EINVAL; - } - - /* Enforce the current active CPU to handle the events in this PMU. */ - event->cpu = cpumask_first(&cmem_lat_pmu->active_cpu); - if (event->cpu >= nr_cpu_ids) - return -EINVAL; - - if (!cmem_lat_pmu_validate_group(event)) - return -EINVAL; - - hwc->idx = -1; - hwc->config = event_type; - - return 0; -} - -static u64 cmem_lat_pmu_read_status(struct cmem_lat_pmu *cmem_lat_pmu, - unsigned int inst) -{ - return readq(cmem_lat_pmu->base[inst] + STATUS); -} - -static u64 cmem_lat_pmu_read_cycle_counter(struct perf_event *event) -{ - const unsigned int instance = 0; - u64 status; - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct device *dev = cmem_lat_pmu->dev; - - /* - * Use the reading from first instance since all instances are - * identical. - */ - status = cmem_lat_pmu_read_status(cmem_lat_pmu, instance); - if (status & STATUS_CYCLE_OVF) - dev_warn(dev, "Cycle counter overflow\n"); - - return readq(cmem_lat_pmu->base[instance] + CYCLE_CNTR); -} - -static u64 cmem_lat_pmu_read_req_counter(struct perf_event *event) -{ - unsigned int i; - u64 status, val = 0; - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct device *dev = cmem_lat_pmu->dev; - - /* Sum up the counts from all instances. */ - for (i = 0; i < NUM_INSTANCES; i++) { - status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); - if (status & STATUS_MC0_REQ_OVF) - dev_warn(dev, "MC0 request counter overflow\n"); - if (status & STATUS_MC1_REQ_OVF) - dev_warn(dev, "MC1 request counter overflow\n"); - if (status & STATUS_MC2_REQ_OVF) - dev_warn(dev, "MC2 request counter overflow\n"); - - val += readq(cmem_lat_pmu->base[i] + MC0_REQ_CNTR); - val += readq(cmem_lat_pmu->base[i] + MC1_REQ_CNTR); - val += readq(cmem_lat_pmu->base[i] + MC2_REQ_CNTR); - } - - return val; -} - -static u64 cmem_lat_pmu_read_aor_counter(struct perf_event *event) -{ - unsigned int i; - u64 status, val = 0; - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct device *dev = cmem_lat_pmu->dev; - - /* Sum up the counts from all instances. */ - for (i = 0; i < NUM_INSTANCES; i++) { - status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); - if (status & STATUS_MC0_AOR_OVF) - dev_warn(dev, "MC0 AOR counter overflow\n"); - if (status & STATUS_MC1_AOR_OVF) - dev_warn(dev, "MC1 AOR counter overflow\n"); - if (status & STATUS_MC2_AOR_OVF) - dev_warn(dev, "MC2 AOR counter overflow\n"); - - val += readq(cmem_lat_pmu->base[i] + MC0_AOR_CNTR); - val += readq(cmem_lat_pmu->base[i] + MC1_AOR_CNTR); - val += readq(cmem_lat_pmu->base[i] + MC2_AOR_CNTR); - } - - return val; -} - -static u64 (*read_counter_fn[NUM_EVENTS])(struct perf_event *) = { - [EVENT_CYCLES] = cmem_lat_pmu_read_cycle_counter, - [EVENT_REQ] = cmem_lat_pmu_read_req_counter, - [EVENT_AOR] = cmem_lat_pmu_read_aor_counter, -}; - -static void cmem_lat_pmu_event_update(struct perf_event *event) -{ - u32 event_type; - u64 prev, now; - struct hw_perf_event *hwc = &event->hw; - - if (hwc->state & PERF_HES_STOPPED) - return; - - event_type = hwc->config; - - do { - prev = local64_read(&hwc->prev_count); - now = read_counter_fn[event_type](event); - } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); - - local64_add(now - prev, &event->count); - - hwc->state |= PERF_HES_UPTODATE; -} - -static void cmem_lat_pmu_start(struct perf_event *event, int pmu_flags) -{ - event->hw.state = 0; -} - -static void cmem_lat_pmu_stop(struct perf_event *event, int pmu_flags) -{ - event->hw.state |= PERF_HES_STOPPED; -} - -static int cmem_lat_pmu_add(struct perf_event *event, int flags) -{ - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; - struct hw_perf_event *hwc = &event->hw; - int idx; - - if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), - &cmem_lat_pmu->associated_cpus))) - return -ENOENT; - - idx = cmem_lat_pmu_get_event_idx(hw_events, event); - if (idx < 0) - return idx; - - hw_events->events[idx] = event; - hwc->idx = idx; - hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; - - if (flags & PERF_EF_START) - cmem_lat_pmu_start(event, PERF_EF_RELOAD); - - /* Propagate changes to the userspace mapping. */ - perf_event_update_userpage(event); - - return 0; -} - -static void cmem_lat_pmu_del(struct perf_event *event, int flags) -{ - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); - struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; - struct hw_perf_event *hwc = &event->hw; - int idx = hwc->idx; - - cmem_lat_pmu_stop(event, PERF_EF_UPDATE); - - hw_events->events[idx] = NULL; - - clear_bit(idx, hw_events->used_ctrs); - - perf_event_update_userpage(event); -} - -static void cmem_lat_pmu_read(struct perf_event *event) -{ - cmem_lat_pmu_event_update(event); -} - -static inline void cmem_lat_pmu_cg_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, u64 val) -{ - writeq(val, BCAST(cmem_lat_pmu) + CG_CTRL); -} - -static inline void cmem_lat_pmu_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, u64 val) -{ - writeq(val, BCAST(cmem_lat_pmu) + CTRL); -} - -static void cmem_lat_pmu_enable(struct pmu *pmu) -{ - bool disabled; - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); - - disabled = bitmap_empty( - cmem_lat_pmu->hw_events.used_ctrs, MAX_ACTIVE_EVENTS); - - if (disabled) - return; - - /* Enable all the counters. */ - cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_ENABLE); - cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_ENABLE); -} - -static void cmem_lat_pmu_disable(struct pmu *pmu) -{ - int idx; - struct perf_event *event; - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); - - /* Disable all the counters. */ - cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_DISABLE); - - /* - * The counters will start from 0 again on restart. - * Update the events immediately to avoid losing the counts. - */ - for_each_set_bit( - idx, cmem_lat_pmu->hw_events.used_ctrs, MAX_ACTIVE_EVENTS) { - event = cmem_lat_pmu->hw_events.events[idx]; - - if (!event) - continue; - - cmem_lat_pmu_event_update(event); - - local64_set(&event->hw.prev_count, 0ULL); - } - - cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_CLR); - cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_DISABLE); -} - -/* PMU identifier attribute. */ - -static ssize_t cmem_lat_pmu_identifier_show(struct device *dev, - struct device_attribute *attr, - char *page) -{ - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(dev_get_drvdata(dev)); - - return sysfs_emit(page, "%s\n", cmem_lat_pmu->identifier); -} - -static struct device_attribute cmem_lat_pmu_identifier_attr = - __ATTR(identifier, 0444, cmem_lat_pmu_identifier_show, NULL); - -static struct attribute *cmem_lat_pmu_identifier_attrs[] = { - &cmem_lat_pmu_identifier_attr.attr, - NULL, -}; - -static struct attribute_group cmem_lat_pmu_identifier_attr_group = { - .attrs = cmem_lat_pmu_identifier_attrs, -}; - -/* Format attributes. */ - -#define NV_PMU_EXT_ATTR(_name, _func, _config) \ - (&((struct dev_ext_attribute[]){ \ - { \ - .attr = __ATTR(_name, 0444, _func, NULL), \ - .var = (void *)_config \ - } \ - })[0].attr.attr) - -static struct attribute *cmem_lat_pmu_formats[] = { - NV_PMU_EXT_ATTR(event, device_show_string, "config:0-1"), - NULL, -}; - -static const struct attribute_group cmem_lat_pmu_format_group = { - .name = "format", - .attrs = cmem_lat_pmu_formats, -}; - -/* Event attributes. */ - -static ssize_t cmem_lat_pmu_sysfs_event_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct perf_pmu_events_attr *pmu_attr; - - pmu_attr = container_of(attr, typeof(*pmu_attr), attr); - return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); -} - -#define NV_PMU_EVENT_ATTR(_name, _config) \ - PMU_EVENT_ATTR_ID(_name, cmem_lat_pmu_sysfs_event_show, _config) - -static struct attribute *cmem_lat_pmu_events[] = { - NV_PMU_EVENT_ATTR(cycles, EVENT_CYCLES), - NV_PMU_EVENT_ATTR(rd_req, EVENT_REQ), - NV_PMU_EVENT_ATTR(rd_cum_outs, EVENT_AOR), - NULL -}; - -static const struct attribute_group cmem_lat_pmu_events_group = { - .name = "events", - .attrs = cmem_lat_pmu_events, -}; - -/* Cpumask attributes. */ - -static ssize_t cmem_lat_pmu_cpumask_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct pmu *pmu = dev_get_drvdata(dev); - struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); - struct dev_ext_attribute *eattr = - container_of(attr, struct dev_ext_attribute, attr); - unsigned long mask_id = (unsigned long)eattr->var; - const cpumask_t *cpumask; - - switch (mask_id) { - case ACTIVE_CPU_MASK: - cpumask = &cmem_lat_pmu->active_cpu; - break; - case ASSOCIATED_CPU_MASK: - cpumask = &cmem_lat_pmu->associated_cpus; - break; - default: - return 0; - } - return cpumap_print_to_pagebuf(true, buf, cpumask); -} - -#define NV_PMU_CPUMASK_ATTR(_name, _config) \ - NV_PMU_EXT_ATTR(_name, cmem_lat_pmu_cpumask_show, \ - (unsigned long)_config) - -static struct attribute *cmem_lat_pmu_cpumask_attrs[] = { - NV_PMU_CPUMASK_ATTR(cpumask, ACTIVE_CPU_MASK), - NV_PMU_CPUMASK_ATTR(associated_cpus, ASSOCIATED_CPU_MASK), - NULL, -}; - -static const struct attribute_group cmem_lat_pmu_cpumask_attr_group = { - .attrs = cmem_lat_pmu_cpumask_attrs, -}; - -/* Per PMU device attribute groups. */ - -static const struct attribute_group *cmem_lat_pmu_attr_groups[] = { - &cmem_lat_pmu_identifier_attr_group, - &cmem_lat_pmu_format_group, - &cmem_lat_pmu_events_group, - &cmem_lat_pmu_cpumask_attr_group, - NULL, -}; - -static int cmem_lat_pmu_cpu_online(unsigned int cpu, struct hlist_node *node) -{ - struct cmem_lat_pmu *cmem_lat_pmu = - hlist_entry_safe(node, struct cmem_lat_pmu, node); - - if (!cpumask_test_cpu(cpu, &cmem_lat_pmu->associated_cpus)) - return 0; - - /* If the PMU is already managed, there is nothing to do */ - if (!cpumask_empty(&cmem_lat_pmu->active_cpu)) - return 0; - - /* Use this CPU for event counting */ - cpumask_set_cpu(cpu, &cmem_lat_pmu->active_cpu); - - return 0; -} - -static int cmem_lat_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) -{ - unsigned int dst; - - struct cmem_lat_pmu *cmem_lat_pmu = - hlist_entry_safe(node, struct cmem_lat_pmu, node); - - /* Nothing to do if this CPU doesn't own the PMU */ - if (!cpumask_test_and_clear_cpu(cpu, &cmem_lat_pmu->active_cpu)) - return 0; - - /* Choose a new CPU to migrate ownership of the PMU to */ - dst = cpumask_any_and_but(&cmem_lat_pmu->associated_cpus, - cpu_online_mask, cpu); - if (dst >= nr_cpu_ids) - return 0; - - /* Use this CPU for event counting */ - perf_pmu_migrate_context(&cmem_lat_pmu->pmu, cpu, dst); - cpumask_set_cpu(dst, &cmem_lat_pmu->active_cpu); - - return 0; -} - -static int cmem_lat_pmu_get_cpus(struct cmem_lat_pmu *cmem_lat_pmu, - unsigned int socket) -{ - int ret = 0, cpu; - - for_each_possible_cpu(cpu) { - if (cpu_to_node(cpu) == socket) - cpumask_set_cpu(cpu, &cmem_lat_pmu->associated_cpus); - } - - if (cpumask_empty(&cmem_lat_pmu->associated_cpus)) { - dev_dbg(cmem_lat_pmu->dev, - "No cpu associated with PMU socket-%u\n", socket); - ret = -ENODEV; - } - - return ret; -} - -static int cmem_lat_pmu_probe(struct platform_device *pdev) -{ - struct device *dev = &pdev->dev; - struct acpi_device *acpi_dev; - struct cmem_lat_pmu *cmem_lat_pmu; - char *name, *uid_str; - int ret, i; - u32 socket; - - acpi_dev = ACPI_COMPANION(dev); - if (!acpi_dev) - return -ENODEV; - - uid_str = acpi_device_uid(acpi_dev); - if (!uid_str) - return -ENODEV; - - ret = kstrtou32(uid_str, 0, &socket); - if (ret) - return ret; - - cmem_lat_pmu = devm_kzalloc(dev, sizeof(*cmem_lat_pmu), GFP_KERNEL); - name = devm_kasprintf(dev, GFP_KERNEL, "nvidia_cmem_latency_pmu_%u", socket); - if (!cmem_lat_pmu || !name) - return -ENOMEM; - - cmem_lat_pmu->dev = dev; - cmem_lat_pmu->name = name; - cmem_lat_pmu->identifier = acpi_device_hid(acpi_dev); - platform_set_drvdata(pdev, cmem_lat_pmu); - - cmem_lat_pmu->pmu = (struct pmu) { - .parent = &pdev->dev, - .task_ctx_nr = perf_invalid_context, - .pmu_enable = cmem_lat_pmu_enable, - .pmu_disable = cmem_lat_pmu_disable, - .event_init = cmem_lat_pmu_event_init, - .add = cmem_lat_pmu_add, - .del = cmem_lat_pmu_del, - .start = cmem_lat_pmu_start, - .stop = cmem_lat_pmu_stop, - .read = cmem_lat_pmu_read, - .attr_groups = cmem_lat_pmu_attr_groups, - .capabilities = PERF_PMU_CAP_NO_EXCLUDE | - PERF_PMU_CAP_NO_INTERRUPT, - }; - - /* Map the address of all the instances plus one for the broadcast. */ - for (i = 0; i < NUM_INSTANCES + 1; i++) { - cmem_lat_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); - if (IS_ERR(cmem_lat_pmu->base[i])) { - dev_err(dev, "Failed map address for instance %d\n", i); - return PTR_ERR(cmem_lat_pmu->base[i]); - } - } - - ret = cmem_lat_pmu_get_cpus(cmem_lat_pmu, socket); - if (ret) - return ret; - - ret = cpuhp_state_add_instance(cmem_lat_pmu_cpuhp_state, - &cmem_lat_pmu->node); - if (ret) { - dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); - return ret; - } - - cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_ENABLE); - cmem_lat_pmu_ctrl(cmem_lat_pmu, CTRL_CLR); - cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CG_CTRL_DISABLE); - - ret = perf_pmu_register(&cmem_lat_pmu->pmu, name, -1); - if (ret) { - dev_err(&pdev->dev, "Failed to register PMU: %d\n", ret); - cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, - &cmem_lat_pmu->node); - return ret; - } - - dev_dbg(&pdev->dev, "Registered %s PMU\n", name); - - return 0; -} - -static void cmem_lat_pmu_device_remove(struct platform_device *pdev) -{ - struct cmem_lat_pmu *cmem_lat_pmu = platform_get_drvdata(pdev); - - perf_pmu_unregister(&cmem_lat_pmu->pmu); - cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, - &cmem_lat_pmu->node); -} - -static const struct acpi_device_id cmem_lat_pmu_acpi_match[] = { - { "NVDA2021", }, - { } -}; -MODULE_DEVICE_TABLE(acpi, cmem_lat_pmu_acpi_match); - -static struct platform_driver cmem_lat_pmu_driver = { - .driver = { - .name = "nvidia-t410-cmem-latency-pmu", - .acpi_match_table = ACPI_PTR(cmem_lat_pmu_acpi_match), - .suppress_bind_attrs = true, - }, - .probe = cmem_lat_pmu_probe, - .remove = cmem_lat_pmu_device_remove, -}; - -static int __init cmem_lat_pmu_init(void) -{ - int ret; - - ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, - "perf/nvidia/cmem_latency:online", - cmem_lat_pmu_cpu_online, - cmem_lat_pmu_cpu_teardown); - if (ret < 0) - return ret; - - cmem_lat_pmu_cpuhp_state = ret; - - return platform_driver_register(&cmem_lat_pmu_driver); -} - -static void __exit cmem_lat_pmu_exit(void) -{ - platform_driver_unregister(&cmem_lat_pmu_driver); - cpuhp_remove_multi_state(cmem_lat_pmu_cpuhp_state); -} - -module_init(cmem_lat_pmu_init); -module_exit(cmem_lat_pmu_exit); - -MODULE_LICENSE("GPL"); -MODULE_DESCRIPTION("NVIDIA Tegra410 CPU Memory Latency PMU driver"); -MODULE_AUTHOR("Besar Wicaksono "); From 26b0c2138931f05a12170f631b6c6b26a94b9678 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:44 -0700 Subject: [PATCH 191/464] Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT PMU" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit ba06e256db01b5c4135d062bc163c78c6256f105. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 76 ---- drivers/perf/arm_cspmu/nvidia_cspmu.c | 324 ------------------ 2 files changed, 400 deletions(-) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 07dc447eead7c..8528685ddb61e 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -7,7 +7,6 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE -* PCIE-TGT PMU Driver ---------- @@ -212,11 +211,6 @@ Example usage: perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ -.. _NVIDIA_T410_PCIE_PMU_RC_Mapping_Section: - -Mapping the RC# to lspci segment number -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register @@ -272,73 +266,3 @@ Example output:: 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 - -PCIE-TGT PMU ------------- - -The PCIE-TGT PMU monitors traffic targeting PCIE BAR and CXL HDM ranges. -There is one PCIE-TGT PMU per PCIE root complex (RC) in the SoC. Each RC in -Tegra410 SoC can have up to 16 lanes that can be bifurcated into up to 8 root -ports (RP). The PMU provides RP filter to count PCIE BAR traffic to each RP and -address filter to count access to PCIE BAR or CXL HDM ranges. The details -of the filters are described in the following sections. - -Mapping the RC# to lspci segment number is similar to the PCIE PMU. -Please see :ref:`NVIDIA_T410_PCIE_PMU_RC_Mapping_Section` for more info. - -The events and configuration options of this PMU device are available in sysfs, -see /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_. - -The events in this PMU can be used to measure bandwidth and utilization: - - * rd_req: count the number of read requests to PCIE. - * wr_req: count the number of write requests to PCIE. - * rd_bytes: count the number of bytes transferred by rd_req. - * wr_bytes: count the number of bytes transferred by wr_req. - * cycles: counts the PCIE cycles. - -The average bandwidth is calculated as:: - - AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS - AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS - -The average request rate is calculated as:: - - AVG_RD_REQUEST_RATE = RD_REQ / CYCLES - AVG_WR_REQUEST_RATE = WR_REQ / CYCLES - -The PMU events can be filtered based on the destination root port or target -address range. Filtering based on RP is only available for PCIE BAR traffic. -Address filter works for both PCIE BAR and CXL HDM ranges. These filters can be -found in sysfs, see -/sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_/format/. - -Destination filter settings: - -* dst_rp_mask: bitmask to select the root port(s) to monitor. E.g. "dst_rp_mask=0xFF" - corresponds to all root ports (from 0 to 7) in the PCIE RC. Note that this filter is - only available for PCIE BAR traffic. -* dst_addr_base: BAR or CXL HDM filter base address. -* dst_addr_mask: BAR or CXL HDM filter address mask. -* dst_addr_en: enable BAR or CXL HDM address range filter. If this is set, the - address range specified by "dst_addr_base" and "dst_addr_mask" will be used to filter - the PCIE BAR and CXL HDM traffic address. The PMU uses the following comparison - to determine if the traffic destination address falls within the filter range:: - - (txn's addr & dst_addr_mask) == (dst_addr_base & dst_addr_mask) - - If the comparison succeeds, then the event will be counted. - -If the destination filter is not specified, the RP filter will be configured by default -to count PCIE BAR traffic to all root ports. - -Example usage: - -* Count event id 0x0 to root port 0 and 1 of PCIE RC-0 on socket 0:: - - perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_0/event=0x0,dst_rp_mask=0x3/ - -* Count event id 0x1 for accesses to PCIE BAR or CXL HDM address range - 0x10000 to 0x100FF on socket 0's PCIE RC-1:: - - perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index 095d2f322c6f9..3a5531d1f94c7 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -42,24 +42,6 @@ #define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) #define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST -#define NV_PCIE_TGT_PORT_COUNT 8ULL -#define NV_PCIE_TGT_EV_TYPE_CC 0x4 -#define NV_PCIE_TGT_EV_TYPE_COUNT 3ULL -#define NV_PCIE_TGT_EV_TYPE_MASK GENMASK_ULL(NV_PCIE_TGT_EV_TYPE_COUNT - 1, 0) -#define NV_PCIE_TGT_FILTER2_MASK GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT, 0) -#define NV_PCIE_TGT_FILTER2_PORT GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT - 1, 0) -#define NV_PCIE_TGT_FILTER2_ADDR_EN BIT(NV_PCIE_TGT_PORT_COUNT) -#define NV_PCIE_TGT_FILTER2_ADDR GENMASK_ULL(15, NV_PCIE_TGT_PORT_COUNT) -#define NV_PCIE_TGT_FILTER2_DEFAULT NV_PCIE_TGT_FILTER2_PORT - -#define NV_PCIE_TGT_ADDR_COUNT 8ULL -#define NV_PCIE_TGT_ADDR_STRIDE 20 -#define NV_PCIE_TGT_ADDR_CTRL 0xD38 -#define NV_PCIE_TGT_ADDR_BASE_LO 0xD3C -#define NV_PCIE_TGT_ADDR_BASE_HI 0xD40 -#define NV_PCIE_TGT_ADDR_MASK_LO 0xD44 -#define NV_PCIE_TGT_ADDR_MASK_HI 0xD48 - #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -204,15 +186,6 @@ static struct attribute *pcie_v2_pmu_event_attrs[] = { NULL, }; -static struct attribute *pcie_tgt_pmu_event_attrs[] = { - ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), - ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), - ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), - ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), - ARM_CSPMU_EVENT_ATTR(cycles, NV_PCIE_TGT_EV_TYPE_CC), - NULL, -}; - static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -266,15 +239,6 @@ static struct attribute *pcie_v2_pmu_format_attrs[] = { NULL, }; -static struct attribute *pcie_tgt_pmu_format_attrs[] = { - ARM_CSPMU_FORMAT_ATTR(event, "config:0-2"), - ARM_CSPMU_FORMAT_ATTR(dst_rp_mask, "config:3-10"), - ARM_CSPMU_FORMAT_ATTR(dst_addr_en, "config:11"), - ARM_CSPMU_FORMAT_ATTR(dst_addr_base, "config1:0-63"), - ARM_CSPMU_FORMAT_ATTR(dst_addr_mask, "config2:0-63"), - NULL, -}; - static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -514,268 +478,6 @@ static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, return 0; } -struct pcie_tgt_addr_filter { - u32 refcount; - u64 base; - u64 mask; -}; - -struct pcie_tgt_data { - struct pcie_tgt_addr_filter addr_filter[NV_PCIE_TGT_ADDR_COUNT]; - void __iomem *addr_filter_reg; -}; - -#if defined(CONFIG_ACPI) -static int pcie_tgt_init_data(struct arm_cspmu *cspmu) -{ - int ret; - struct acpi_device *adev; - struct pcie_tgt_data *data; - struct list_head resource_list; - struct resource_entry *rentry; - struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); - struct device *dev = cspmu->dev; - - data = devm_kzalloc(dev, sizeof(struct pcie_tgt_data), GFP_KERNEL); - if (!data) - return -ENOMEM; - - adev = arm_cspmu_acpi_dev_get(cspmu); - if (!adev) { - dev_err(dev, "failed to get associated PCIE-TGT device\n"); - return -ENODEV; - } - - INIT_LIST_HEAD(&resource_list); - ret = acpi_dev_get_memory_resources(adev, &resource_list); - if (ret < 0) { - dev_err(dev, "failed to get PCIE-TGT device memory resources\n"); - acpi_dev_put(adev); - return ret; - } - - rentry = list_first_entry_or_null( - &resource_list, struct resource_entry, node); - if (rentry) { - data->addr_filter_reg = devm_ioremap_resource(dev, rentry->res); - ret = 0; - } - - if (IS_ERR(data->addr_filter_reg)) { - dev_err(dev, "failed to get address filter resource\n"); - ret = PTR_ERR(data->addr_filter_reg); - } - - acpi_dev_free_resource_list(&resource_list); - acpi_dev_put(adev); - - ctx->data = data; - - return ret; -} -#else -static int pcie_tgt_init_data(struct arm_cspmu *cspmu) -{ - return -ENODEV; -} -#endif - -static struct pcie_tgt_data *pcie_tgt_get_data(struct arm_cspmu *cspmu) -{ - struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); - - return ctx->data; -} - -/* Find the first available address filter slot. */ -static int pcie_tgt_find_addr_idx(struct arm_cspmu *cspmu, u64 base, u64 mask, - bool is_reset) -{ - int i; - struct pcie_tgt_data *data = pcie_tgt_get_data(cspmu); - - for (i = 0; i < NV_PCIE_TGT_ADDR_COUNT; i++) { - if (!is_reset && data->addr_filter[i].refcount == 0) - return i; - - if (data->addr_filter[i].base == base && - data->addr_filter[i].mask == mask) - return i; - } - - return -ENODEV; -} - -static u32 pcie_tgt_pmu_event_filter(const struct perf_event *event) -{ - u32 filter; - - filter = (event->attr.config >> NV_PCIE_TGT_EV_TYPE_COUNT) & - NV_PCIE_TGT_FILTER2_MASK; - - return filter; -} - -static bool pcie_tgt_pmu_addr_en(const struct perf_event *event) -{ - u32 filter = pcie_tgt_pmu_event_filter(event); - - return FIELD_GET(NV_PCIE_TGT_FILTER2_ADDR_EN, filter) != 0; -} - -static u32 pcie_tgt_pmu_port_filter(const struct perf_event *event) -{ - u32 filter = pcie_tgt_pmu_event_filter(event); - - return FIELD_GET(NV_PCIE_TGT_FILTER2_PORT, filter); -} - -static u64 pcie_tgt_pmu_dst_addr_base(const struct perf_event *event) -{ - return event->attr.config1; -} - -static u64 pcie_tgt_pmu_dst_addr_mask(const struct perf_event *event) -{ - return event->attr.config2; -} - -static int pcie_tgt_pmu_validate_event(struct arm_cspmu *cspmu, - struct perf_event *new_ev) -{ - u64 base, mask; - int idx; - - if (!pcie_tgt_pmu_addr_en(new_ev)) - return 0; - - /* Make sure there is a slot available for the address filter. */ - base = pcie_tgt_pmu_dst_addr_base(new_ev); - mask = pcie_tgt_pmu_dst_addr_mask(new_ev); - idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); - if (idx < 0) - return -EINVAL; - - return 0; -} - -static void pcie_tgt_pmu_config_addr_filter(struct arm_cspmu *cspmu, - bool en, u64 base, u64 mask, int idx) -{ - struct pcie_tgt_data *data; - struct pcie_tgt_addr_filter *filter; - void __iomem *filter_reg; - - data = pcie_tgt_get_data(cspmu); - filter = &data->addr_filter[idx]; - filter_reg = data->addr_filter_reg + (idx * NV_PCIE_TGT_ADDR_STRIDE); - - if (en) { - filter->refcount++; - if (filter->refcount == 1) { - filter->base = base; - filter->mask = mask; - - writel(lower_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); - writel(upper_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); - writel(lower_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); - writel(upper_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); - writel(1, filter_reg + NV_PCIE_TGT_ADDR_CTRL); - } - } else { - filter->refcount--; - if (filter->refcount == 0) { - writel(0, filter_reg + NV_PCIE_TGT_ADDR_CTRL); - writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); - writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); - writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); - writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); - - filter->base = 0; - filter->mask = 0; - } - } -} - -static void pcie_tgt_pmu_set_ev_filter(struct arm_cspmu *cspmu, - const struct perf_event *event) -{ - bool addr_filter_en; - int idx; - u32 filter2_val, filter2_offset, port_filter; - u64 base, mask; - - filter2_val = 0; - filter2_offset = PMEVFILT2R + (4 * event->hw.idx); - - addr_filter_en = pcie_tgt_pmu_addr_en(event); - if (addr_filter_en) { - base = pcie_tgt_pmu_dst_addr_base(event); - mask = pcie_tgt_pmu_dst_addr_mask(event); - idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); - - if (idx < 0) { - dev_err(cspmu->dev, - "Unable to find a slot for address filtering\n"); - writel(0, cspmu->base0 + filter2_offset); - return; - } - - /* Configure address range filter registers.*/ - pcie_tgt_pmu_config_addr_filter(cspmu, true, base, mask, idx); - - /* Config the counter to use the selected address filter slot. */ - filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_ADDR, 1U << idx); - } - - port_filter = pcie_tgt_pmu_port_filter(event); - - /* Monitor all ports if no filter is selected. */ - if (!addr_filter_en && port_filter == 0) - port_filter = NV_PCIE_TGT_FILTER2_PORT; - - filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_PORT, port_filter); - - writel(filter2_val, cspmu->base0 + filter2_offset); -} - -static void pcie_tgt_pmu_reset_ev_filter(struct arm_cspmu *cspmu, - const struct perf_event *event) -{ - bool addr_filter_en; - u64 base, mask; - int idx; - - addr_filter_en = pcie_tgt_pmu_addr_en(event); - if (!addr_filter_en) - return; - - base = pcie_tgt_pmu_dst_addr_base(event); - mask = pcie_tgt_pmu_dst_addr_mask(event); - idx = pcie_tgt_find_addr_idx(cspmu, base, mask, true); - - if (idx < 0) { - dev_err(cspmu->dev, - "Unable to find the address filter slot to reset\n"); - return; - } - - pcie_tgt_pmu_config_addr_filter( - cspmu, false, base, mask, idx); -} - -static u32 pcie_tgt_pmu_event_type(const struct perf_event *event) -{ - return event->attr.config & NV_PCIE_TGT_EV_TYPE_MASK; -} - -static bool pcie_tgt_pmu_is_cycle_counter_event(const struct perf_event *event) -{ - u32 event_type = pcie_tgt_pmu_event_type(event); - - return event_type == NV_PCIE_TGT_EV_TYPE_CC; -} - enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, NAME_FMT_SOCKET, @@ -920,30 +622,6 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .reset_ev_filter = nv_cspmu_reset_ev_filter, } }, - { - .prodid = 0x10700000, - .prodid_mask = NV_PRODID_MASK, - .name_pattern = "nvidia_pcie_tgt_pmu_%u_rc_%u", - .name_fmt = NAME_FMT_SOCKET_INST, - .template_ctx = { - .event_attr = pcie_tgt_pmu_event_attrs, - .format_attr = pcie_tgt_pmu_format_attrs, - .filter_mask = 0x0, - .filter_default_val = 0x0, - .filter2_mask = NV_PCIE_TGT_FILTER2_MASK, - .filter2_default_val = NV_PCIE_TGT_FILTER2_DEFAULT, - .get_filter = NULL, - .get_filter2 = NULL, - .init_data = pcie_tgt_init_data - }, - .ops = { - .is_cycle_counter_event = pcie_tgt_pmu_is_cycle_counter_event, - .event_type = pcie_tgt_pmu_event_type, - .validate_event = pcie_tgt_pmu_validate_event, - .set_ev_filter = pcie_tgt_pmu_set_ev_filter, - .reset_ev_filter = pcie_tgt_pmu_reset_ev_filter, - } - }, { .prodid = 0, .prodid_mask = 0, @@ -1039,8 +717,6 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) /* NVIDIA specific callbacks. */ SET_OP(validate_event, impl_ops, match, NULL); - SET_OP(event_type, impl_ops, match, NULL); - SET_OP(is_cycle_counter_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); SET_OP(reset_ev_filter, impl_ops, match, NULL); From 8d12953a89301bba300852bf01a06eb26a1f026e Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:48 -0700 Subject: [PATCH 192/464] Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE PMU" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit 6984fc5cbdb379ca9d4239e0b573b845a7a69b95. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 162 -------------- drivers/perf/arm_cspmu/nvidia_cspmu.c | 208 +----------------- 2 files changed, 2 insertions(+), 368 deletions(-) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 8528685ddb61e..7b7ba5700ca19 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -6,7 +6,6 @@ The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) -* PCIE PMU Driver ---------- @@ -105,164 +104,3 @@ Example usage: destination filter = remote memory:: perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ - -PCIE PMU --------- - -This PMU monitors all read/write traffic from the root port(s) or a particular -BDF in a PCIE root complex (RC) to local or remote memory. There is one PMU per -PCIE RC in the SoC. Each RC can have up to 16 lanes that can be bifurcated into -up to 8 root ports. The traffic from each root port can be filtered using RP or -BDF filter. For example, specifying "src_rp_mask=0xFF" means the PMU counter will -capture traffic from all RPs. Please see below for more details. - -The events and configuration options of this PMU device are described in sysfs, -see /sys/bus/event_source/devices/nvidia_pcie_pmu__rc_. - -The events in this PMU can be used to measure bandwidth, utilization, and -latency: - - * rd_req: count the number of read requests by PCIE device. - * wr_req: count the number of write requests by PCIE device. - * rd_bytes: count the number of bytes transferred by rd_req. - * wr_bytes: count the number of bytes transferred by wr_req. - * rd_cum_outs: count outstanding rd_req each cycle. - * cycles: counts the PCIE cycles. - -The average bandwidth is calculated as:: - - AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS - AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS - -The average request rate is calculated as:: - - AVG_RD_REQUEST_RATE = RD_REQ / CYCLES - AVG_WR_REQUEST_RATE = WR_REQ / CYCLES - - -The average latency is calculated as:: - - FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS - AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ - AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ - -The PMU events can be filtered based on the traffic source and destination. -The source filter indicates the PCIE devices that will be monitored. The -destination filter specifies the destination memory type, e.g. local system -memory (CMEM), local GPU memory (GMEM), or remote memory. The local/remote -classification of the destination filter is based on the home socket of the -address, not where the data actually resides. These filters can be found in -/sys/bus/event_source/devices/nvidia_pcie_pmu__rc_/format/. - -The list of event filters: - -* Source filter: - - * src_rp_mask: bitmask of root ports that will be monitored. Each bit in this - bitmask represents the RP index in the RC. If the bit is set, all devices under - the associated RP will be monitored. E.g "src_rp_mask=0xF" will monitor - devices in root port 0 to 3. - * src_bdf: the BDF that will be monitored. This is a 16-bit value that - follows formula: (bus << 8) + (device << 3) + (function). For example, the - value of BDF 27:01.1 is 0x2781. - * src_bdf_en: enable the BDF filter. If this is set, the BDF filter value in - "src_bdf" is used to filter the traffic. - - Note that Root-Port and BDF filters are mutually exclusive and the PMU in - each RC can only have one BDF filter for the whole counters. If BDF filter - is enabled, the BDF filter value will be applied to all events. - -* Destination filter: - - * dst_loc_cmem: if set, count events to local system memory (CMEM) address - * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address - * dst_loc_pcie_p2p: if set, count events to local PCIE peer address - * dst_loc_pcie_cxl: if set, count events to local CXL memory address - * dst_rem: if set, count events to remote memory address - -If the source filter is not specified, the PMU will count events from all root -ports. If the destination filter is not specified, the PMU will count events -to all destinations. - -Example usage: - -* Count event id 0x0 from root port 0 of PCIE RC-0 on socket 0 targeting all - destinations:: - - perf stat -a -e nvidia_pcie_pmu_0_rc_0/event=0x0,src_rp_mask=0x1/ - -* Count event id 0x1 from root port 0 and 1 of PCIE RC-1 on socket 0 and - targeting just local CMEM of socket 0:: - - perf stat -a -e nvidia_pcie_pmu_0_rc_1/event=0x1,src_rp_mask=0x3,dst_loc_cmem=0x1/ - -* Count event id 0x2 from root port 0 of PCIE RC-2 on socket 1 targeting all - destinations:: - - perf stat -a -e nvidia_pcie_pmu_1_rc_2/event=0x2,src_rp_mask=0x1/ - -* Count event id 0x3 from root port 0 and 1 of PCIE RC-3 on socket 1 and - targeting just local CMEM of socket 1:: - - perf stat -a -e nvidia_pcie_pmu_1_rc_3/event=0x3,src_rp_mask=0x3,dst_loc_cmem=0x1/ - -* Count event id 0x4 from BDF 01:01.0 of PCIE RC-4 on socket 0 targeting all - destinations:: - - perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ - -Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA -Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space -for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register -contains the following information to map PCIE devices under the RP back to its RC# : - - - Bus# (byte 0xc) : bus number as reported by the lspci output - - Segment# (byte 0xd) : segment number as reported by the lspci output - - RP# (byte 0xe) : port number as reported by LnkCap attribute from lspci for a device with Root Port capability - - RC# (byte 0xf): root complex number associated with the RP - - Socket# (byte 0x10): socket number associated with the RP - -Example script for mapping lspci BDF to RC# and socket#:: - - #!/bin/bash - while read bdf rest; do - dvsec4_reg=$(lspci -vv -s $bdf | awk ' - /Designated Vendor-Specific: Vendor=10de ID=0004/ { - match($0, /\[([0-9a-fA-F]+)/, arr); - print "0x" arr[1]; - exit - } - ') - if [ -n "$dvsec4_reg" ]; then - bus=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xc))).b) - segment=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xd))).b) - rp=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xe))).b) - rc=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xf))).b) - socket=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0x10))).b) - echo "$bdf: Bus=$bus, Segment=$segment, RP=$rp, RC=$rc, Socket=$socket" - fi - done < <(lspci -d 10de:) - -Example output:: - - 0001:00:00.0: Bus=00, Segment=01, RP=00, RC=00, Socket=00 - 0002:80:00.0: Bus=80, Segment=02, RP=01, RC=01, Socket=00 - 0002:a0:00.0: Bus=a0, Segment=02, RP=02, RC=01, Socket=00 - 0002:c0:00.0: Bus=c0, Segment=02, RP=03, RC=01, Socket=00 - 0002:e0:00.0: Bus=e0, Segment=02, RP=04, RC=01, Socket=00 - 0003:00:00.0: Bus=00, Segment=03, RP=00, RC=02, Socket=00 - 0004:00:00.0: Bus=00, Segment=04, RP=00, RC=03, Socket=00 - 0005:00:00.0: Bus=00, Segment=05, RP=00, RC=04, Socket=00 - 0005:40:00.0: Bus=40, Segment=05, RP=01, RC=04, Socket=00 - 0005:c0:00.0: Bus=c0, Segment=05, RP=02, RC=04, Socket=00 - 0006:00:00.0: Bus=00, Segment=06, RP=00, RC=05, Socket=00 - 0009:00:00.0: Bus=00, Segment=09, RP=00, RC=00, Socket=01 - 000a:80:00.0: Bus=80, Segment=0a, RP=01, RC=01, Socket=01 - 000a:a0:00.0: Bus=a0, Segment=0a, RP=02, RC=01, Socket=01 - 000a:e0:00.0: Bus=e0, Segment=0a, RP=03, RC=01, Socket=01 - 000b:00:00.0: Bus=00, Segment=0b, RP=00, RC=02, Socket=01 - 000c:00:00.0: Bus=00, Segment=0c, RP=00, RC=03, Socket=01 - 000d:00:00.0: Bus=00, Segment=0d, RP=00, RC=04, Socket=01 - 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 - 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 - 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index 3a5531d1f94c7..c67667097a3cd 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -8,7 +8,6 @@ #include #include -#include #include #include "arm_cspmu.h" @@ -29,19 +28,6 @@ #define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) #define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) -#define NV_PCIE_V2_PORT_COUNT 8ULL -#define NV_PCIE_V2_FILTER_ID_MASK GENMASK_ULL(24, 0) -#define NV_PCIE_V2_FILTER_PORT GENMASK_ULL(NV_PCIE_V2_PORT_COUNT - 1, 0) -#define NV_PCIE_V2_FILTER_BDF_VAL GENMASK_ULL(23, NV_PCIE_V2_PORT_COUNT) -#define NV_PCIE_V2_FILTER_BDF_EN BIT(24) -#define NV_PCIE_V2_FILTER_BDF_VAL_EN GENMASK_ULL(24, NV_PCIE_V2_PORT_COUNT) -#define NV_PCIE_V2_FILTER_DEFAULT NV_PCIE_V2_FILTER_PORT - -#define NV_PCIE_V2_DST_COUNT 5ULL -#define NV_PCIE_V2_FILTER2_ID_MASK GENMASK_ULL(4, 0) -#define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) -#define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST - #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -176,16 +162,6 @@ static struct attribute *ucf_pmu_event_attrs[] = { NULL, }; -static struct attribute *pcie_v2_pmu_event_attrs[] = { - ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), - ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), - ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), - ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), - ARM_CSPMU_EVENT_ATTR(rd_cum_outs, 0x4), - ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), - NULL, -}; - static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -226,19 +202,6 @@ static struct attribute *ucf_pmu_format_attrs[] = { NULL, }; -static struct attribute *pcie_v2_pmu_format_attrs[] = { - ARM_CSPMU_FORMAT_EVENT_ATTR, - ARM_CSPMU_FORMAT_ATTR(src_rp_mask, "config1:0-7"), - ARM_CSPMU_FORMAT_ATTR(src_bdf, "config1:8-23"), - ARM_CSPMU_FORMAT_ATTR(src_bdf_en, "config1:24"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config2:0"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config2:1"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_p2p, "config2:2"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_cxl, "config2:3"), - ARM_CSPMU_FORMAT_ATTR(dst_rem, "config2:4"), - NULL, -}; - static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -270,32 +233,6 @@ nv_cspmu_get_name(const struct arm_cspmu *cspmu) return ctx->name; } -#if defined(CONFIG_ACPI) -static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) -{ - struct fwnode_handle *fwnode; - struct acpi_device *adev; - int ret; - - adev = arm_cspmu_acpi_dev_get(cspmu); - if (!adev) - return -ENODEV; - - fwnode = acpi_fwnode_handle(adev); - ret = fwnode_property_read_u32(fwnode, "instance_id", id); - if (ret) - dev_err(cspmu->dev, "Failed to get instance ID\n"); - - acpi_dev_put(adev); - return ret; -} -#else -static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) -{ - return -EINVAL; -} -#endif - static u32 nv_cspmu_event_filter(const struct perf_event *event) { const struct nv_cspmu_ctx *ctx = @@ -341,20 +278,6 @@ static void nv_cspmu_set_ev_filter(struct arm_cspmu *cspmu, } } -static void nv_cspmu_reset_ev_filter(struct arm_cspmu *cspmu, - const struct perf_event *event) -{ - const struct nv_cspmu_ctx *ctx = - to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); - const u32 offset = 4 * event->hw.idx; - - if (ctx->get_filter) - writel(0, cspmu->base0 + PMEVFILTR + offset); - - if (ctx->get_filter2) - writel(0, cspmu->base0 + PMEVFILT2R + offset); -} - static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, const struct perf_event *event) { @@ -385,103 +308,9 @@ static u32 ucf_pmu_event_filter(const struct perf_event *event) return ret; } -static u32 pcie_v2_pmu_bdf_val_en(u32 filter) -{ - const u32 bdf_en = FIELD_GET(NV_PCIE_V2_FILTER_BDF_EN, filter); - - /* Returns both BDF value and enable bit if BDF filtering is enabled. */ - if (bdf_en) - return FIELD_GET(NV_PCIE_V2_FILTER_BDF_VAL_EN, filter); - - /* Ignore the BDF value if BDF filter is not enabled. */ - return 0; -} - -static u32 pcie_v2_pmu_event_filter(const struct perf_event *event) -{ - u32 filter, lead_filter, lead_bdf; - struct perf_event *leader; - const struct nv_cspmu_ctx *ctx = - to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); - - filter = event->attr.config1 & ctx->filter_mask; - if (filter != 0) - return filter; - - leader = event->group_leader; - - /* Use leader's filter value if its BDF filtering is enabled. */ - if (event != leader) { - lead_filter = pcie_v2_pmu_event_filter(leader); - lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); - if (lead_bdf != 0) - return lead_filter; - } - - /* Otherwise, return default filter value. */ - return ctx->filter_default_val; -} - -static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, - struct perf_event *new_ev) -{ - /* - * Make sure the events are using same BDF filter since the PCIE-SRC PMU - * only supports one common BDF filter setting for all of the counters. - */ - - int idx; - u32 new_filter, new_rp, new_bdf, new_lead_filter, new_lead_bdf; - struct perf_event *leader, *new_leader; - - if (cspmu->impl.ops.is_cycle_counter_event(new_ev)) - return 0; - - new_leader = new_ev->group_leader; - - new_filter = pcie_v2_pmu_event_filter(new_ev); - new_lead_filter = pcie_v2_pmu_event_filter(new_leader); - - new_bdf = pcie_v2_pmu_bdf_val_en(new_filter); - new_lead_bdf = pcie_v2_pmu_bdf_val_en(new_lead_filter); - - new_rp = FIELD_GET(NV_PCIE_V2_FILTER_PORT, new_filter); - - if (new_rp != 0 && new_bdf != 0) { - dev_err(cspmu->dev, - "RP and BDF filtering are mutually exclusive\n"); - return -EINVAL; - } - - if (new_bdf != new_lead_bdf) { - dev_err(cspmu->dev, - "sibling and leader BDF value should be equal\n"); - return -EINVAL; - } - - /* Compare BDF filter on existing events. */ - idx = find_first_bit(cspmu->hw_events.used_ctrs, - cspmu->cycle_counter_logical_idx); - - if (idx != cspmu->cycle_counter_logical_idx) { - leader = cspmu->hw_events.events[idx]->group_leader; - - const u32 lead_filter = pcie_v2_pmu_event_filter(leader); - const u32 lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); - - if (new_lead_bdf != lead_bdf) { - dev_err(cspmu->dev, "only one BDF value is supported\n"); - return -EINVAL; - } - } - - return 0; -} - enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, - NAME_FMT_SOCKET, - NAME_FMT_SOCKET_INST + NAME_FMT_SOCKET }; struct nv_cspmu_match { @@ -601,27 +430,6 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .init_data = NULL }, }, - { - .prodid = 0x10301000, - .prodid_mask = NV_PRODID_MASK, - .name_pattern = "nvidia_pcie_pmu_%u_rc_%u", - .name_fmt = NAME_FMT_SOCKET_INST, - .template_ctx = { - .event_attr = pcie_v2_pmu_event_attrs, - .format_attr = pcie_v2_pmu_format_attrs, - .filter_mask = NV_PCIE_V2_FILTER_ID_MASK, - .filter_default_val = NV_PCIE_V2_FILTER_DEFAULT, - .filter2_mask = NV_PCIE_V2_FILTER2_ID_MASK, - .filter2_default_val = NV_PCIE_V2_FILTER2_DEFAULT, - .get_filter = pcie_v2_pmu_event_filter, - .get_filter2 = nv_cspmu_event_filter2, - .init_data = NULL - }, - .ops = { - .validate_event = pcie_v2_pmu_validate_event, - .reset_ev_filter = nv_cspmu_reset_ev_filter, - } - }, { .prodid = 0, .prodid_mask = 0, @@ -645,7 +453,7 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, const struct nv_cspmu_match *match) { - char *name = NULL; + char *name; struct device *dev = cspmu->dev; static atomic_t pmu_generic_idx = {0}; @@ -659,16 +467,6 @@ static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, socket); break; } - case NAME_FMT_SOCKET_INST: { - const int cpu = cpumask_first(&cspmu->associated_cpus); - const int socket = cpu_to_node(cpu); - u32 inst_id; - - if (!nv_cspmu_get_inst_id(cspmu, &inst_id)) - name = devm_kasprintf(dev, GFP_KERNEL, - match->name_pattern, socket, inst_id); - break; - } case NAME_FMT_GENERIC: name = devm_kasprintf(dev, GFP_KERNEL, match->name_pattern, atomic_fetch_inc(&pmu_generic_idx)); @@ -716,10 +514,8 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) cspmu->impl.ctx = ctx; /* NVIDIA specific callbacks. */ - SET_OP(validate_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); - SET_OP(reset_ev_filter, impl_ops, match, NULL); SET_OP(get_event_attrs, impl_ops, match, nv_cspmu_get_event_attrs); SET_OP(get_format_attrs, impl_ops, match, nv_cspmu_get_format_attrs); SET_OP(get_name, impl_ops, match, nv_cspmu_get_name); From a15e1d98c8d124163354c6a8e7fdd0cdf081beab Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:52 -0700 Subject: [PATCH 193/464] Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: Add arm_cspmu_acpi_dev_get" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit a2ab08daa7b3ad98f34c89edb451053ede1f549d. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/perf/arm_cspmu/arm_cspmu.c | 24 +----------------------- drivers/perf/arm_cspmu/arm_cspmu.h | 17 +---------------- 2 files changed, 2 insertions(+), 39 deletions(-) diff --git a/drivers/perf/arm_cspmu/arm_cspmu.c b/drivers/perf/arm_cspmu/arm_cspmu.c index dadc9b765d801..34430b68f6025 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.c +++ b/drivers/perf/arm_cspmu/arm_cspmu.c @@ -16,7 +16,7 @@ * The user should refer to the vendor technical documentation to get details * about the supported events. * - * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -1132,28 +1132,6 @@ static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) return 0; } - -struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) -{ - char hid[16]; - char uid[16]; - struct acpi_device *adev; - const struct acpi_apmt_node *apmt_node; - - apmt_node = arm_cspmu_apmt_node(cspmu->dev); - if (!apmt_node || apmt_node->type != ACPI_APMT_NODE_TYPE_ACPI) - return NULL; - - memset(hid, 0, sizeof(hid)); - memset(uid, 0, sizeof(uid)); - - memcpy(hid, &apmt_node->inst_primary, sizeof(apmt_node->inst_primary)); - snprintf(uid, sizeof(uid), "%u", apmt_node->inst_secondary); - - adev = acpi_dev_get_first_match_dev(hid, uid, -1); - return adev; -} -EXPORT_SYMBOL_GPL(arm_cspmu_acpi_dev_get); #else static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) { diff --git a/drivers/perf/arm_cspmu/arm_cspmu.h b/drivers/perf/arm_cspmu/arm_cspmu.h index 3200966732003..cd65a58dbd884 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.h +++ b/drivers/perf/arm_cspmu/arm_cspmu.h @@ -1,14 +1,13 @@ /* SPDX-License-Identifier: GPL-2.0 * * ARM CoreSight Architecture PMU driver. - * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ #ifndef __ARM_CSPMU_H__ #define __ARM_CSPMU_H__ -#include #include #include #include @@ -256,18 +255,4 @@ int arm_cspmu_impl_register(const struct arm_cspmu_impl_match *impl_match); /* Unregister vendor backend. */ void arm_cspmu_impl_unregister(const struct arm_cspmu_impl_match *impl_match); -#if defined(CONFIG_ACPI) -/** - * Get ACPI device associated with the PMU. - * The caller is responsible for calling acpi_dev_put() on the returned device. - */ -struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu); -#else -static inline struct acpi_device * -arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) -{ - return NULL; -} -#endif - #endif /* __ARM_CSPMU_H__ */ From e0364a79efa10e187d227da1f2aaf2786ecbec39 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:25:56 -0700 Subject: [PATCH 194/464] Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit e12d030d5a967372345b960ac4130e4f112bf1d5. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/admin-guide/perf/index.rst | 1 - .../admin-guide/perf/nvidia-tegra410-pmu.rst | 106 ------------------ drivers/perf/arm_cspmu/nvidia_cspmu.c | 90 +-------------- 3 files changed, 1 insertion(+), 196 deletions(-) delete mode 100644 Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index aa12708ddb965..c407bb44b08e3 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -25,7 +25,6 @@ Performance monitor support alibaba_pmu dwc_pcie_pmu nvidia-tegra241-pmu - nvidia-tegra410-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst deleted file mode 100644 index 7b7ba5700ca19..0000000000000 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ /dev/null @@ -1,106 +0,0 @@ -===================================================================== -NVIDIA Tegra410 SoC Uncore Performance Monitoring Unit (PMU) -===================================================================== - -The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance -metrics like memory bandwidth, latency, and utilization: - -* Unified Coherence Fabric (UCF) - -PMU Driver ----------- - -The PMU driver describes the available events and configuration of each PMU in -sysfs. Please see the sections below to get the sysfs path of each PMU. Like -other uncore PMU drivers, the driver provides "cpumask" sysfs attribute to show -the CPU id used to handle the PMU event. There is also "associated_cpus" -sysfs attribute, which contains a list of CPUs associated with the PMU instance. - -UCF PMU -------- - -The Unified Coherence Fabric (UCF) in the NVIDIA Tegra410 SoC serves as a -distributed cache, last level for CPU Memory and CXL Memory, and cache coherent -interconnect that supports hardware coherence across multiple coherently caching -agents, including: - - * CPU clusters - * GPU - * PCIe Ordering Controller Unit (OCU) - * Other IO-coherent requesters - -The events and configuration options of this PMU device are described in sysfs, -see /sys/bus/event_source/devices/nvidia_ucf_pmu_. - -Some of the events available in this PMU can be used to measure bandwidth and -utilization: - - * slc_access_rd: count the number of read requests to SLC. - * slc_access_wr: count the number of write requests to SLC. - * slc_bytes_rd: count the number of bytes transferred by slc_access_rd. - * slc_bytes_wr: count the number of bytes transferred by slc_access_wr. - * mem_access_rd: count the number of read requests to local or remote memory. - * mem_access_wr: count the number of write requests to local or remote memory. - * mem_bytes_rd: count the number of bytes transferred by mem_access_rd. - * mem_bytes_wr: count the number of bytes transferred by mem_access_wr. - * cycles: counts the UCF cycles. - -The average bandwidth is calculated as:: - - AVG_SLC_READ_BANDWIDTH_IN_GBPS = SLC_BYTES_RD / ELAPSED_TIME_IN_NS - AVG_SLC_WRITE_BANDWIDTH_IN_GBPS = SLC_BYTES_WR / ELAPSED_TIME_IN_NS - AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS - AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS - -The average request rate is calculated as:: - - AVG_SLC_READ_REQUEST_RATE = SLC_ACCESS_RD / CYCLES - AVG_SLC_WRITE_REQUEST_RATE = SLC_ACCESS_WR / CYCLES - AVG_MEM_READ_REQUEST_RATE = MEM_ACCESS_RD / CYCLES - AVG_MEM_WRITE_REQUEST_RATE = MEM_ACCESS_WR / CYCLES - -More details about what other events are available can be found in Tegra410 SoC -technical reference manual. - -The events can be filtered based on source or destination. The source filter -indicates the traffic initiator to the SLC, e.g local CPU, non-CPU device, or -remote socket. The destination filter specifies the destination memory type, -e.g. local system memory (CMEM), local GPU memory (GMEM), or remote memory. The -local/remote classification of the destination filter is based on the home -socket of the address, not where the data actually resides. The available -filters are described in -/sys/bus/event_source/devices/nvidia_ucf_pmu_/format/. - -The list of UCF PMU event filters: - -* Source filter: - - * src_loc_cpu: if set, count events from local CPU - * src_loc_noncpu: if set, count events from local non-CPU device - * src_rem: if set, count events from CPU, GPU, PCIE devices of remote socket - -* Destination filter: - - * dst_loc_cmem: if set, count events to local system memory (CMEM) address - * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address - * dst_loc_other: if set, count events to local CXL memory address - * dst_rem: if set, count events to CPU, GPU, and CXL memory address of remote socket - -If the source is not specified, the PMU will count events from all sources. If -the destination is not specified, the PMU will count events to all destinations. - -Example usage: - -* Count event id 0x0 in socket 0 from all sources and to all destinations:: - - perf stat -a -e nvidia_ucf_pmu_0/event=0x0/ - -* Count event id 0x0 in socket 0 with source filter = local CPU and destination - filter = local system memory (CMEM):: - - perf stat -a -e nvidia_ucf_pmu_0/event=0x0,src_loc_cpu=0x1,dst_loc_cmem=0x1/ - -* Count event id 0x0 in socket 1 with source filter = local non-CPU device and - destination filter = remote memory:: - - perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index c67667097a3cd..e06a06d3407b1 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -21,13 +21,6 @@ #define NV_CNVL_PORT_COUNT 4ULL #define NV_CNVL_FILTER_ID_MASK GENMASK_ULL(NV_CNVL_PORT_COUNT - 1, 0) -#define NV_UCF_SRC_COUNT 3ULL -#define NV_UCF_DST_COUNT 4ULL -#define NV_UCF_FILTER_ID_MASK GENMASK_ULL(11, 0) -#define NV_UCF_FILTER_SRC GENMASK_ULL(2, 0) -#define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) -#define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) - #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -131,37 +124,6 @@ static struct attribute *mcf_pmu_event_attrs[] = { NULL, }; -static struct attribute *ucf_pmu_event_attrs[] = { - ARM_CSPMU_EVENT_ATTR(bus_cycles, 0x1D), - - ARM_CSPMU_EVENT_ATTR(slc_allocate, 0xF0), - ARM_CSPMU_EVENT_ATTR(slc_wb, 0xF3), - ARM_CSPMU_EVENT_ATTR(slc_refill_rd, 0x109), - ARM_CSPMU_EVENT_ATTR(slc_refill_wr, 0x10A), - ARM_CSPMU_EVENT_ATTR(slc_hit_rd, 0x119), - - ARM_CSPMU_EVENT_ATTR(slc_access_dataless, 0x183), - ARM_CSPMU_EVENT_ATTR(slc_access_atomic, 0x184), - - ARM_CSPMU_EVENT_ATTR(slc_access, 0xF2), - ARM_CSPMU_EVENT_ATTR(slc_access_rd, 0x111), - ARM_CSPMU_EVENT_ATTR(slc_access_wr, 0x112), - ARM_CSPMU_EVENT_ATTR(slc_bytes_rd, 0x113), - ARM_CSPMU_EVENT_ATTR(slc_bytes_wr, 0x114), - - ARM_CSPMU_EVENT_ATTR(mem_access_rd, 0x121), - ARM_CSPMU_EVENT_ATTR(mem_access_wr, 0x122), - ARM_CSPMU_EVENT_ATTR(mem_bytes_rd, 0x123), - ARM_CSPMU_EVENT_ATTR(mem_bytes_wr, 0x124), - - ARM_CSPMU_EVENT_ATTR(local_snoop, 0x180), - ARM_CSPMU_EVENT_ATTR(ext_snp_access, 0x181), - ARM_CSPMU_EVENT_ATTR(ext_snp_evict, 0x182), - - ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), - NULL, -}; - static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -190,18 +152,6 @@ static struct attribute *cnvlink_pmu_format_attrs[] = { NULL, }; -static struct attribute *ucf_pmu_format_attrs[] = { - ARM_CSPMU_FORMAT_EVENT_ATTR, - ARM_CSPMU_FORMAT_ATTR(src_loc_noncpu, "config1:0"), - ARM_CSPMU_FORMAT_ATTR(src_loc_cpu, "config1:1"), - ARM_CSPMU_FORMAT_ATTR(src_rem, "config1:2"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config1:8"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config1:9"), - ARM_CSPMU_FORMAT_ATTR(dst_loc_other, "config1:10"), - ARM_CSPMU_FORMAT_ATTR(dst_rem, "config1:11"), - NULL, -}; - static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -286,27 +236,6 @@ static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, writel(filter, cspmu->base0 + PMCCFILTR); } -static u32 ucf_pmu_event_filter(const struct perf_event *event) -{ - u32 ret, filter, src, dst; - - filter = nv_cspmu_event_filter(event); - - /* Monitor all sources if none is selected. */ - src = FIELD_GET(NV_UCF_FILTER_SRC, filter); - if (src == 0) - src = GENMASK_ULL(NV_UCF_SRC_COUNT - 1, 0); - - /* Monitor all destinations if none is selected. */ - dst = FIELD_GET(NV_UCF_FILTER_DST, filter); - if (dst == 0) - dst = GENMASK_ULL(NV_UCF_DST_COUNT - 1, 0); - - ret = FIELD_PREP(NV_UCF_FILTER_SRC, src); - ret |= FIELD_PREP(NV_UCF_FILTER_DST, dst); - - return ret; -} enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, @@ -413,23 +342,6 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .init_data = NULL }, }, - { - .prodid = 0x2CF20000, - .prodid_mask = NV_PRODID_MASK, - .name_pattern = "nvidia_ucf_pmu_%u", - .name_fmt = NAME_FMT_SOCKET, - .template_ctx = { - .event_attr = ucf_pmu_event_attrs, - .format_attr = ucf_pmu_format_attrs, - .filter_mask = NV_UCF_FILTER_ID_MASK, - .filter_default_val = NV_UCF_FILTER_DEFAULT, - .filter2_mask = 0x0, - .filter2_default_val = 0x0, - .get_filter = ucf_pmu_event_filter, - .get_filter2 = NULL, - .init_data = NULL - }, - }, { .prodid = 0, .prodid_mask = 0, From 522e42e1776d15bb3187d7a8111b8d044e2b49df Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 16 Apr 2026 17:26:00 -0700 Subject: [PATCH 195/464] Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Rename doc to Tegra241" BugLink: https://bugs.launchpad.net/bugs/2149756 This reverts commit 575f7ef512986af745ef7346bc3badda9a4a7ec8. This will be replaced by the equivalent patch from v7.1. Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/admin-guide/perf/index.rst | 2 +- .../perf/{nvidia-tegra241-pmu.rst => nvidia-pmu.rst} | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) rename Documentation/admin-guide/perf/{nvidia-tegra241-pmu.rst => nvidia-pmu.rst} (98%) diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index c407bb44b08e3..47d9a3df6329b 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -24,7 +24,7 @@ Performance monitor support thunderx2-pmu alibaba_pmu dwc_pcie_pmu - nvidia-tegra241-pmu + nvidia-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst b/Documentation/admin-guide/perf/nvidia-pmu.rst similarity index 98% rename from Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst rename to Documentation/admin-guide/perf/nvidia-pmu.rst index fad5bc4cee6c0..f538ef67e0e8f 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-pmu.rst @@ -1,8 +1,8 @@ -============================================================ -NVIDIA Tegra241 SoC Uncore Performance Monitoring Unit (PMU) -============================================================ +========================================================= +NVIDIA Tegra SoC Uncore Performance Monitoring Unit (PMU) +========================================================= -The NVIDIA Tegra241 SoC includes various system PMUs to measure key performance +The NVIDIA Tegra SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Scalable Coherency Fabric (SCF) From ca8694a0a9fb2bb0542e9a275fead553c30a2eac Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:45 +0000 Subject: [PATCH 196/464] perf/arm_cspmu: nvidia: Rename doc to Tegra241 BugLink: https://bugs.launchpad.net/bugs/2149756 The documentation in nvidia-pmu.rst contains PMUs specific to NVIDIA Tegra241 SoC. Rename the file for this specific SoC to have better distinction with other NVIDIA SoC. Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit d332424d1d06a9fb03ca04ba3f1092c3990125e8) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/admin-guide/perf/index.rst | 2 +- .../perf/{nvidia-pmu.rst => nvidia-tegra241-pmu.rst} | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) rename Documentation/admin-guide/perf/{nvidia-pmu.rst => nvidia-tegra241-pmu.rst} (98%) diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index 47d9a3df6329b..c407bb44b08e3 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -24,7 +24,7 @@ Performance monitor support thunderx2-pmu alibaba_pmu dwc_pcie_pmu - nvidia-pmu + nvidia-tegra241-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst similarity index 98% rename from Documentation/admin-guide/perf/nvidia-pmu.rst rename to Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst index f538ef67e0e8f..fad5bc4cee6c0 100644 --- a/Documentation/admin-guide/perf/nvidia-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra241-pmu.rst @@ -1,8 +1,8 @@ -========================================================= -NVIDIA Tegra SoC Uncore Performance Monitoring Unit (PMU) -========================================================= +============================================================ +NVIDIA Tegra241 SoC Uncore Performance Monitoring Unit (PMU) +============================================================ -The NVIDIA Tegra SoC includes various system PMUs to measure key performance +The NVIDIA Tegra241 SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Scalable Coherency Fabric (SCF) From 9b9cedef3180550922c30b77facba74e9a6b3c38 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:46 +0000 Subject: [PATCH 197/464] perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU BugLink: https://bugs.launchpad.net/bugs/2149756 The Unified Coherence Fabric (UCF) contains last level cache and cache coherent interconnect in Tegra410 SOC. The PMU in this device can be used to capture events related to access to the last level cache and memory from different sources. Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit f5caf26fd6c71294d0fb254404ed66f8cff6f7f7) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/admin-guide/perf/index.rst | 1 + .../admin-guide/perf/nvidia-tegra410-pmu.rst | 106 ++++++++++++++++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 87 +++++++++++++- 3 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst index c407bb44b08e3..aa12708ddb965 100644 --- a/Documentation/admin-guide/perf/index.rst +++ b/Documentation/admin-guide/perf/index.rst @@ -25,6 +25,7 @@ Performance monitor support alibaba_pmu dwc_pcie_pmu nvidia-tegra241-pmu + nvidia-tegra410-pmu meson-ddr-pmu cxl ampere_cspmu diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst new file mode 100644 index 0000000000000..7b7ba5700ca19 --- /dev/null +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -0,0 +1,106 @@ +===================================================================== +NVIDIA Tegra410 SoC Uncore Performance Monitoring Unit (PMU) +===================================================================== + +The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance +metrics like memory bandwidth, latency, and utilization: + +* Unified Coherence Fabric (UCF) + +PMU Driver +---------- + +The PMU driver describes the available events and configuration of each PMU in +sysfs. Please see the sections below to get the sysfs path of each PMU. Like +other uncore PMU drivers, the driver provides "cpumask" sysfs attribute to show +the CPU id used to handle the PMU event. There is also "associated_cpus" +sysfs attribute, which contains a list of CPUs associated with the PMU instance. + +UCF PMU +------- + +The Unified Coherence Fabric (UCF) in the NVIDIA Tegra410 SoC serves as a +distributed cache, last level for CPU Memory and CXL Memory, and cache coherent +interconnect that supports hardware coherence across multiple coherently caching +agents, including: + + * CPU clusters + * GPU + * PCIe Ordering Controller Unit (OCU) + * Other IO-coherent requesters + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_ucf_pmu_. + +Some of the events available in this PMU can be used to measure bandwidth and +utilization: + + * slc_access_rd: count the number of read requests to SLC. + * slc_access_wr: count the number of write requests to SLC. + * slc_bytes_rd: count the number of bytes transferred by slc_access_rd. + * slc_bytes_wr: count the number of bytes transferred by slc_access_wr. + * mem_access_rd: count the number of read requests to local or remote memory. + * mem_access_wr: count the number of write requests to local or remote memory. + * mem_bytes_rd: count the number of bytes transferred by mem_access_rd. + * mem_bytes_wr: count the number of bytes transferred by mem_access_wr. + * cycles: counts the UCF cycles. + +The average bandwidth is calculated as:: + + AVG_SLC_READ_BANDWIDTH_IN_GBPS = SLC_BYTES_RD / ELAPSED_TIME_IN_NS + AVG_SLC_WRITE_BANDWIDTH_IN_GBPS = SLC_BYTES_WR / ELAPSED_TIME_IN_NS + AVG_MEM_READ_BANDWIDTH_IN_GBPS = MEM_BYTES_RD / ELAPSED_TIME_IN_NS + AVG_MEM_WRITE_BANDWIDTH_IN_GBPS = MEM_BYTES_WR / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_SLC_READ_REQUEST_RATE = SLC_ACCESS_RD / CYCLES + AVG_SLC_WRITE_REQUEST_RATE = SLC_ACCESS_WR / CYCLES + AVG_MEM_READ_REQUEST_RATE = MEM_ACCESS_RD / CYCLES + AVG_MEM_WRITE_REQUEST_RATE = MEM_ACCESS_WR / CYCLES + +More details about what other events are available can be found in Tegra410 SoC +technical reference manual. + +The events can be filtered based on source or destination. The source filter +indicates the traffic initiator to the SLC, e.g local CPU, non-CPU device, or +remote socket. The destination filter specifies the destination memory type, +e.g. local system memory (CMEM), local GPU memory (GMEM), or remote memory. The +local/remote classification of the destination filter is based on the home +socket of the address, not where the data actually resides. The available +filters are described in +/sys/bus/event_source/devices/nvidia_ucf_pmu_/format/. + +The list of UCF PMU event filters: + +* Source filter: + + * src_loc_cpu: if set, count events from local CPU + * src_loc_noncpu: if set, count events from local non-CPU device + * src_rem: if set, count events from CPU, GPU, PCIE devices of remote socket + +* Destination filter: + + * dst_loc_cmem: if set, count events to local system memory (CMEM) address + * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address + * dst_loc_other: if set, count events to local CXL memory address + * dst_rem: if set, count events to CPU, GPU, and CXL memory address of remote socket + +If the source is not specified, the PMU will count events from all sources. If +the destination is not specified, the PMU will count events to all destinations. + +Example usage: + +* Count event id 0x0 in socket 0 from all sources and to all destinations:: + + perf stat -a -e nvidia_ucf_pmu_0/event=0x0/ + +* Count event id 0x0 in socket 0 with source filter = local CPU and destination + filter = local system memory (CMEM):: + + perf stat -a -e nvidia_ucf_pmu_0/event=0x0,src_loc_cpu=0x1,dst_loc_cmem=0x1/ + +* Count event id 0x0 in socket 1 with source filter = local non-CPU device and + destination filter = remote memory:: + + perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index e06a06d3407b1..8e37cbe3bae99 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -21,6 +21,13 @@ #define NV_CNVL_PORT_COUNT 4ULL #define NV_CNVL_FILTER_ID_MASK GENMASK_ULL(NV_CNVL_PORT_COUNT - 1, 0) +#define NV_UCF_SRC_COUNT 3ULL +#define NV_UCF_DST_COUNT 4ULL +#define NV_UCF_FILTER_ID_MASK GENMASK_ULL(11, 0) +#define NV_UCF_FILTER_SRC GENMASK_ULL(2, 0) +#define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) +#define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -124,6 +131,36 @@ static struct attribute *mcf_pmu_event_attrs[] = { NULL, }; +static struct attribute *ucf_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(bus_cycles, 0x1D), + + ARM_CSPMU_EVENT_ATTR(slc_allocate, 0xF0), + ARM_CSPMU_EVENT_ATTR(slc_wb, 0xF3), + ARM_CSPMU_EVENT_ATTR(slc_refill_rd, 0x109), + ARM_CSPMU_EVENT_ATTR(slc_refill_wr, 0x10A), + ARM_CSPMU_EVENT_ATTR(slc_hit_rd, 0x119), + + ARM_CSPMU_EVENT_ATTR(slc_access_dataless, 0x183), + ARM_CSPMU_EVENT_ATTR(slc_access_atomic, 0x184), + + ARM_CSPMU_EVENT_ATTR(slc_access_rd, 0x111), + ARM_CSPMU_EVENT_ATTR(slc_access_wr, 0x112), + ARM_CSPMU_EVENT_ATTR(slc_bytes_rd, 0x113), + ARM_CSPMU_EVENT_ATTR(slc_bytes_wr, 0x114), + + ARM_CSPMU_EVENT_ATTR(mem_access_rd, 0x121), + ARM_CSPMU_EVENT_ATTR(mem_access_wr, 0x122), + ARM_CSPMU_EVENT_ATTR(mem_bytes_rd, 0x123), + ARM_CSPMU_EVENT_ATTR(mem_bytes_wr, 0x124), + + ARM_CSPMU_EVENT_ATTR(local_snoop, 0x180), + ARM_CSPMU_EVENT_ATTR(ext_snp_access, 0x181), + ARM_CSPMU_EVENT_ATTR(ext_snp_evict, 0x182), + + ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), + NULL +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -152,6 +189,18 @@ static struct attribute *cnvlink_pmu_format_attrs[] = { NULL, }; +static struct attribute *ucf_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_EVENT_ATTR, + ARM_CSPMU_FORMAT_ATTR(src_loc_noncpu, "config1:0"), + ARM_CSPMU_FORMAT_ATTR(src_loc_cpu, "config1:1"), + ARM_CSPMU_FORMAT_ATTR(src_rem, "config1:2"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config1:8"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config1:9"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_other, "config1:10"), + ARM_CSPMU_FORMAT_ATTR(dst_rem, "config1:11"), + NULL +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -236,6 +285,27 @@ static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, writel(filter, cspmu->base0 + PMCCFILTR); } +static u32 ucf_pmu_event_filter(const struct perf_event *event) +{ + u32 ret, filter, src, dst; + + filter = nv_cspmu_event_filter(event); + + /* Monitor all sources if none is selected. */ + src = FIELD_GET(NV_UCF_FILTER_SRC, filter); + if (src == 0) + src = GENMASK_ULL(NV_UCF_SRC_COUNT - 1, 0); + + /* Monitor all destinations if none is selected. */ + dst = FIELD_GET(NV_UCF_FILTER_DST, filter); + if (dst == 0) + dst = GENMASK_ULL(NV_UCF_DST_COUNT - 1, 0); + + ret = FIELD_PREP(NV_UCF_FILTER_SRC, src); + ret |= FIELD_PREP(NV_UCF_FILTER_DST, dst); + + return ret; +} enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, @@ -342,6 +412,21 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .init_data = NULL }, }, + { + .prodid = 0x2CF20000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_ucf_pmu_%u", + .name_fmt = NAME_FMT_SOCKET, + .template_ctx = { + .event_attr = ucf_pmu_event_attrs, + .format_attr = ucf_pmu_format_attrs, + .filter_mask = NV_UCF_FILTER_ID_MASK, + .filter_default_val = NV_UCF_FILTER_DEFAULT, + .filter2_mask = 0x0, + .filter2_default_val = 0x0, + .get_filter = ucf_pmu_event_filter, + }, + }, { .prodid = 0, .prodid_mask = 0, From df2e770800aa06e690c52d24272faf1832d0c686 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:47 +0000 Subject: [PATCH 198/464] perf/arm_cspmu: Add arm_cspmu_acpi_dev_get BugLink: https://bugs.launchpad.net/bugs/2149756 Add interface to get ACPI device associated with the PMU. This ACPI device may contain additional properties not covered by the standard properties. Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit bc86281fe4bd5d4a78be2f370e8319c9517e40ff) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/perf/arm_cspmu/arm_cspmu.c | 19 ++++++++++++++++++- drivers/perf/arm_cspmu/arm_cspmu.h | 17 ++++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/drivers/perf/arm_cspmu/arm_cspmu.c b/drivers/perf/arm_cspmu/arm_cspmu.c index 34430b68f6025..49e8a1f381319 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.c +++ b/drivers/perf/arm_cspmu/arm_cspmu.c @@ -16,7 +16,7 @@ * The user should refer to the vendor technical documentation to get details * about the supported events. * - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ @@ -1132,6 +1132,23 @@ static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) return 0; } + +struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) +{ + char hid[16] = {}; + char uid[16] = {}; + const struct acpi_apmt_node *apmt_node; + + apmt_node = arm_cspmu_apmt_node(cspmu->dev); + if (!apmt_node || apmt_node->type != ACPI_APMT_NODE_TYPE_ACPI) + return NULL; + + memcpy(hid, &apmt_node->inst_primary, sizeof(apmt_node->inst_primary)); + snprintf(uid, sizeof(uid), "%u", apmt_node->inst_secondary); + + return acpi_dev_get_first_match_dev(hid, uid, -1); +} +EXPORT_SYMBOL_GPL(arm_cspmu_acpi_dev_get); #else static int arm_cspmu_acpi_get_cpus(struct arm_cspmu *cspmu) { diff --git a/drivers/perf/arm_cspmu/arm_cspmu.h b/drivers/perf/arm_cspmu/arm_cspmu.h index cd65a58dbd884..3fc5c8d772663 100644 --- a/drivers/perf/arm_cspmu/arm_cspmu.h +++ b/drivers/perf/arm_cspmu/arm_cspmu.h @@ -1,13 +1,14 @@ /* SPDX-License-Identifier: GPL-2.0 * * ARM CoreSight Architecture PMU driver. - * Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. * */ #ifndef __ARM_CSPMU_H__ #define __ARM_CSPMU_H__ +#include #include #include #include @@ -255,4 +256,18 @@ int arm_cspmu_impl_register(const struct arm_cspmu_impl_match *impl_match); /* Unregister vendor backend. */ void arm_cspmu_impl_unregister(const struct arm_cspmu_impl_match *impl_match); +#if defined(CONFIG_ACPI) && defined(CONFIG_ARM64) +/** + * Get ACPI device associated with the PMU. + * The caller is responsible for calling acpi_dev_put() on the returned device. + */ +struct acpi_device *arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu); +#else +static inline struct acpi_device * +arm_cspmu_acpi_dev_get(const struct arm_cspmu *cspmu) +{ + return NULL; +} +#endif + #endif /* __ARM_CSPMU_H__ */ From 604f0170c5283a88ed03ecd31d62c9f0b9f79448 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:48 +0000 Subject: [PATCH 199/464] perf/arm_cspmu: nvidia: Add Tegra410 PCIE PMU BugLink: https://bugs.launchpad.net/bugs/2149756 Adds PCIE PMU support in Tegra410 SOC. This PMU is instanced in each root complex in the SOC and can capture traffic from PCIE device to various memory types. This PMU can filter traffic based on the originating root port or BDF and the target memory types (CPU DRAM, GPU Memory, CXL Memory, or remote Memory). Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit bf585ba14726788335c640512d11186dab573612) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 163 ++++++++++++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 210 +++++++++++++++++- 2 files changed, 368 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 7b7ba5700ca19..b8cfbb80be1c1 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -6,6 +6,7 @@ The NVIDIA Tegra410 SoC includes various system PMUs to measure key performance metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) +* PCIE PMU Driver ---------- @@ -104,3 +105,165 @@ Example usage: destination filter = remote memory:: perf stat -a -e nvidia_ucf_pmu_1/event=0x0,src_loc_noncpu=0x1,dst_rem=0x1/ + +PCIE PMU +-------- + +This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and +the memory subsystem. It monitors all read/write traffic from the root port(s) +or a particular BDF in a PCIE RC to local or remote memory. There is one PMU per +PCIE RC in the SoC. Each RC can have up to 16 lanes that can be bifurcated into +up to 8 root ports. The traffic from each root port can be filtered using RP or +BDF filter. For example, specifying "src_rp_mask=0xFF" means the PMU counter will +capture traffic from all RPs. Please see below for more details. + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_pcie_pmu__rc_. + +The events in this PMU can be used to measure bandwidth, utilization, and +latency: + + * rd_req: count the number of read requests by PCIE device. + * wr_req: count the number of write requests by PCIE device. + * rd_bytes: count the number of bytes transferred by rd_req. + * wr_bytes: count the number of bytes transferred by wr_req. + * rd_cum_outs: count outstanding rd_req each cycle. + * cycles: count the clock cycles of SOC fabric connected to the PCIE interface. + +The average bandwidth is calculated as:: + + AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS + AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_RD_REQUEST_RATE = RD_REQ / CYCLES + AVG_WR_REQUEST_RATE = WR_REQ / CYCLES + + +The average latency is calculated as:: + + FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ + AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ + +The PMU events can be filtered based on the traffic source and destination. +The source filter indicates the PCIE devices that will be monitored. The +destination filter specifies the destination memory type, e.g. local system +memory (CMEM), local GPU memory (GMEM), or remote memory. The local/remote +classification of the destination filter is based on the home socket of the +address, not where the data actually resides. These filters can be found in +/sys/bus/event_source/devices/nvidia_pcie_pmu__rc_/format/. + +The list of event filters: + +* Source filter: + + * src_rp_mask: bitmask of root ports that will be monitored. Each bit in this + bitmask represents the RP index in the RC. If the bit is set, all devices under + the associated RP will be monitored. E.g "src_rp_mask=0xF" will monitor + devices in root port 0 to 3. + * src_bdf: the BDF that will be monitored. This is a 16-bit value that + follows formula: (bus << 8) + (device << 3) + (function). For example, the + value of BDF 27:01.1 is 0x2781. + * src_bdf_en: enable the BDF filter. If this is set, the BDF filter value in + "src_bdf" is used to filter the traffic. + + Note that Root-Port and BDF filters are mutually exclusive and the PMU in + each RC can only have one BDF filter for the whole counters. If BDF filter + is enabled, the BDF filter value will be applied to all events. + +* Destination filter: + + * dst_loc_cmem: if set, count events to local system memory (CMEM) address + * dst_loc_gmem: if set, count events to local GPU memory (GMEM) address + * dst_loc_pcie_p2p: if set, count events to local PCIE peer address + * dst_loc_pcie_cxl: if set, count events to local CXL memory address + * dst_rem: if set, count events to remote memory address + +If the source filter is not specified, the PMU will count events from all root +ports. If the destination filter is not specified, the PMU will count events +to all destinations. + +Example usage: + +* Count event id 0x0 from root port 0 of PCIE RC-0 on socket 0 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_0/event=0x0,src_rp_mask=0x1/ + +* Count event id 0x1 from root port 0 and 1 of PCIE RC-1 on socket 0 and + targeting just local CMEM of socket 0:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_1/event=0x1,src_rp_mask=0x3,dst_loc_cmem=0x1/ + +* Count event id 0x2 from root port 0 of PCIE RC-2 on socket 1 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_1_rc_2/event=0x2,src_rp_mask=0x1/ + +* Count event id 0x3 from root port 0 and 1 of PCIE RC-3 on socket 1 and + targeting just local CMEM of socket 1:: + + perf stat -a -e nvidia_pcie_pmu_1_rc_3/event=0x3,src_rp_mask=0x3,dst_loc_cmem=0x1/ + +* Count event id 0x4 from BDF 01:01.0 of PCIE RC-4 on socket 0 targeting all + destinations:: + + perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ + +Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA +Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space +for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register +contains the following information to map PCIE devices under the RP back to its RC# : + + - Bus# (byte 0xc) : bus number as reported by the lspci output + - Segment# (byte 0xd) : segment number as reported by the lspci output + - RP# (byte 0xe) : port number as reported by LnkCap attribute from lspci for a device with Root Port capability + - RC# (byte 0xf): root complex number associated with the RP + - Socket# (byte 0x10): socket number associated with the RP + +Example script for mapping lspci BDF to RC# and socket#:: + + #!/bin/bash + while read bdf rest; do + dvsec4_reg=$(lspci -vv -s $bdf | awk ' + /Designated Vendor-Specific: Vendor=10de ID=0004/ { + match($0, /\[([0-9a-fA-F]+)/, arr); + print "0x" arr[1]; + exit + } + ') + if [ -n "$dvsec4_reg" ]; then + bus=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xc))).b) + segment=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xd))).b) + rp=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xe))).b) + rc=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0xf))).b) + socket=$(setpci -s $bdf $(printf '0x%x' $((${dvsec4_reg} + 0x10))).b) + echo "$bdf: Bus=$bus, Segment=$segment, RP=$rp, RC=$rc, Socket=$socket" + fi + done < <(lspci -d 10de:) + +Example output:: + + 0001:00:00.0: Bus=00, Segment=01, RP=00, RC=00, Socket=00 + 0002:80:00.0: Bus=80, Segment=02, RP=01, RC=01, Socket=00 + 0002:a0:00.0: Bus=a0, Segment=02, RP=02, RC=01, Socket=00 + 0002:c0:00.0: Bus=c0, Segment=02, RP=03, RC=01, Socket=00 + 0002:e0:00.0: Bus=e0, Segment=02, RP=04, RC=01, Socket=00 + 0003:00:00.0: Bus=00, Segment=03, RP=00, RC=02, Socket=00 + 0004:00:00.0: Bus=00, Segment=04, RP=00, RC=03, Socket=00 + 0005:00:00.0: Bus=00, Segment=05, RP=00, RC=04, Socket=00 + 0005:40:00.0: Bus=40, Segment=05, RP=01, RC=04, Socket=00 + 0005:c0:00.0: Bus=c0, Segment=05, RP=02, RC=04, Socket=00 + 0006:00:00.0: Bus=00, Segment=06, RP=00, RC=05, Socket=00 + 0009:00:00.0: Bus=00, Segment=09, RP=00, RC=00, Socket=01 + 000a:80:00.0: Bus=80, Segment=0a, RP=01, RC=01, Socket=01 + 000a:a0:00.0: Bus=a0, Segment=0a, RP=02, RC=01, Socket=01 + 000a:e0:00.0: Bus=e0, Segment=0a, RP=03, RC=01, Socket=01 + 000b:00:00.0: Bus=00, Segment=0b, RP=00, RC=02, Socket=01 + 000c:00:00.0: Bus=00, Segment=0c, RP=00, RC=03, Socket=01 + 000d:00:00.0: Bus=00, Segment=0d, RP=00, RC=04, Socket=01 + 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 + 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 + 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index 8e37cbe3bae99..61fde84ea3434 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -8,6 +8,7 @@ #include #include +#include #include #include "arm_cspmu.h" @@ -28,6 +29,19 @@ #define NV_UCF_FILTER_DST GENMASK_ULL(11, 8) #define NV_UCF_FILTER_DEFAULT (NV_UCF_FILTER_SRC | NV_UCF_FILTER_DST) +#define NV_PCIE_V2_PORT_COUNT 8ULL +#define NV_PCIE_V2_FILTER_ID_MASK GENMASK_ULL(24, 0) +#define NV_PCIE_V2_FILTER_PORT GENMASK_ULL(NV_PCIE_V2_PORT_COUNT - 1, 0) +#define NV_PCIE_V2_FILTER_BDF_VAL GENMASK_ULL(23, NV_PCIE_V2_PORT_COUNT) +#define NV_PCIE_V2_FILTER_BDF_EN BIT(24) +#define NV_PCIE_V2_FILTER_BDF_VAL_EN GENMASK_ULL(24, NV_PCIE_V2_PORT_COUNT) +#define NV_PCIE_V2_FILTER_DEFAULT NV_PCIE_V2_FILTER_PORT + +#define NV_PCIE_V2_DST_COUNT 5ULL +#define NV_PCIE_V2_FILTER2_ID_MASK GENMASK_ULL(4, 0) +#define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) +#define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -161,6 +175,16 @@ static struct attribute *ucf_pmu_event_attrs[] = { NULL }; +static struct attribute *pcie_v2_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), + ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), + ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), + ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), + ARM_CSPMU_EVENT_ATTR(rd_cum_outs, 0x4), + ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), + NULL +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -201,6 +225,19 @@ static struct attribute *ucf_pmu_format_attrs[] = { NULL }; +static struct attribute *pcie_v2_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_EVENT_ATTR, + ARM_CSPMU_FORMAT_ATTR(src_rp_mask, "config1:0-7"), + ARM_CSPMU_FORMAT_ATTR(src_bdf, "config1:8-23"), + ARM_CSPMU_FORMAT_ATTR(src_bdf_en, "config1:24"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_cmem, "config2:0"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_gmem, "config2:1"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_p2p, "config2:2"), + ARM_CSPMU_FORMAT_ATTR(dst_loc_pcie_cxl, "config2:3"), + ARM_CSPMU_FORMAT_ATTR(dst_rem, "config2:4"), + NULL +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -232,6 +269,32 @@ nv_cspmu_get_name(const struct arm_cspmu *cspmu) return ctx->name; } +#if defined(CONFIG_ACPI) && defined(CONFIG_ARM64) +static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) +{ + struct fwnode_handle *fwnode; + struct acpi_device *adev; + int ret; + + adev = arm_cspmu_acpi_dev_get(cspmu); + if (!adev) + return -ENODEV; + + fwnode = acpi_fwnode_handle(adev); + ret = fwnode_property_read_u32(fwnode, "instance_id", id); + if (ret) + dev_err(cspmu->dev, "Failed to get instance ID\n"); + + acpi_dev_put(adev); + return ret; +} +#else +static int nv_cspmu_get_inst_id(const struct arm_cspmu *cspmu, u32 *id) +{ + return -EINVAL; +} +#endif + static u32 nv_cspmu_event_filter(const struct perf_event *event) { const struct nv_cspmu_ctx *ctx = @@ -277,6 +340,20 @@ static void nv_cspmu_set_ev_filter(struct arm_cspmu *cspmu, } } +static void nv_cspmu_reset_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + const struct nv_cspmu_ctx *ctx = + to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); + const u32 offset = 4 * event->hw.idx; + + if (ctx->get_filter) + writel(0, cspmu->base0 + PMEVFILTR + offset); + + if (ctx->get_filter2) + writel(0, cspmu->base0 + PMEVFILT2R + offset); +} + static void nv_cspmu_set_cc_filter(struct arm_cspmu *cspmu, const struct perf_event *event) { @@ -307,9 +384,103 @@ static u32 ucf_pmu_event_filter(const struct perf_event *event) return ret; } +static u32 pcie_v2_pmu_bdf_val_en(u32 filter) +{ + const u32 bdf_en = FIELD_GET(NV_PCIE_V2_FILTER_BDF_EN, filter); + + /* Returns both BDF value and enable bit if BDF filtering is enabled. */ + if (bdf_en) + return FIELD_GET(NV_PCIE_V2_FILTER_BDF_VAL_EN, filter); + + /* Ignore the BDF value if BDF filter is not enabled. */ + return 0; +} + +static u32 pcie_v2_pmu_event_filter(const struct perf_event *event) +{ + u32 filter, lead_filter, lead_bdf; + struct perf_event *leader; + const struct nv_cspmu_ctx *ctx = + to_nv_cspmu_ctx(to_arm_cspmu(event->pmu)); + + filter = event->attr.config1 & ctx->filter_mask; + if (filter != 0) + return filter; + + leader = event->group_leader; + + /* Use leader's filter value if its BDF filtering is enabled. */ + if (event != leader) { + lead_filter = pcie_v2_pmu_event_filter(leader); + lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); + if (lead_bdf != 0) + return lead_filter; + } + + /* Otherwise, return default filter value. */ + return ctx->filter_default_val; +} + +static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, + struct perf_event *new_ev) +{ + /* + * Make sure the events are using same BDF filter since the PCIE-SRC PMU + * only supports one common BDF filter setting for all of the counters. + */ + + int idx; + u32 new_filter, new_rp, new_bdf, new_lead_filter, new_lead_bdf; + struct perf_event *new_leader; + + if (cspmu->impl.ops.is_cycle_counter_event(new_ev)) + return 0; + + new_leader = new_ev->group_leader; + + new_filter = pcie_v2_pmu_event_filter(new_ev); + new_lead_filter = pcie_v2_pmu_event_filter(new_leader); + + new_bdf = pcie_v2_pmu_bdf_val_en(new_filter); + new_lead_bdf = pcie_v2_pmu_bdf_val_en(new_lead_filter); + + new_rp = FIELD_GET(NV_PCIE_V2_FILTER_PORT, new_filter); + + if (new_rp != 0 && new_bdf != 0) { + dev_err(cspmu->dev, + "RP and BDF filtering are mutually exclusive\n"); + return -EINVAL; + } + + if (new_bdf != new_lead_bdf) { + dev_err(cspmu->dev, + "sibling and leader BDF value should be equal\n"); + return -EINVAL; + } + + /* Compare BDF filter on existing events. */ + idx = find_first_bit(cspmu->hw_events.used_ctrs, + cspmu->cycle_counter_logical_idx); + + if (idx != cspmu->cycle_counter_logical_idx) { + struct perf_event *leader = cspmu->hw_events.events[idx]->group_leader; + + const u32 lead_filter = pcie_v2_pmu_event_filter(leader); + const u32 lead_bdf = pcie_v2_pmu_bdf_val_en(lead_filter); + + if (new_lead_bdf != lead_bdf) { + dev_err(cspmu->dev, "only one BDF value is supported\n"); + return -EINVAL; + } + } + + return 0; +} + enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, - NAME_FMT_SOCKET + NAME_FMT_SOCKET, + NAME_FMT_SOCKET_INST, }; struct nv_cspmu_match { @@ -427,6 +598,26 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .get_filter = ucf_pmu_event_filter, }, }, + { + .prodid = 0x10301000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_pcie_pmu_%u_rc_%u", + .name_fmt = NAME_FMT_SOCKET_INST, + .template_ctx = { + .event_attr = pcie_v2_pmu_event_attrs, + .format_attr = pcie_v2_pmu_format_attrs, + .filter_mask = NV_PCIE_V2_FILTER_ID_MASK, + .filter_default_val = NV_PCIE_V2_FILTER_DEFAULT, + .filter2_mask = NV_PCIE_V2_FILTER2_ID_MASK, + .filter2_default_val = NV_PCIE_V2_FILTER2_DEFAULT, + .get_filter = pcie_v2_pmu_event_filter, + .get_filter2 = nv_cspmu_event_filter2, + }, + .ops = { + .validate_event = pcie_v2_pmu_validate_event, + .reset_ev_filter = nv_cspmu_reset_ev_filter, + } + }, { .prodid = 0, .prodid_mask = 0, @@ -450,7 +641,7 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, const struct nv_cspmu_match *match) { - char *name; + char *name = NULL; struct device *dev = cspmu->dev; static atomic_t pmu_generic_idx = {0}; @@ -464,13 +655,20 @@ static char *nv_cspmu_format_name(const struct arm_cspmu *cspmu, socket); break; } + case NAME_FMT_SOCKET_INST: { + const int cpu = cpumask_first(&cspmu->associated_cpus); + const int socket = cpu_to_node(cpu); + u32 inst_id; + + if (!nv_cspmu_get_inst_id(cspmu, &inst_id)) + name = devm_kasprintf(dev, GFP_KERNEL, + match->name_pattern, socket, inst_id); + break; + } case NAME_FMT_GENERIC: name = devm_kasprintf(dev, GFP_KERNEL, match->name_pattern, atomic_fetch_inc(&pmu_generic_idx)); break; - default: - name = NULL; - break; } return name; @@ -511,8 +709,10 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) cspmu->impl.ctx = ctx; /* NVIDIA specific callbacks. */ + SET_OP(validate_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); + SET_OP(reset_ev_filter, impl_ops, match, NULL); SET_OP(get_event_attrs, impl_ops, match, nv_cspmu_get_event_attrs); SET_OP(get_format_attrs, impl_ops, match, nv_cspmu_get_format_attrs); SET_OP(get_name, impl_ops, match, nv_cspmu_get_name); From 5b0641fd0c91bb887f0d04efd46001337b621b8d Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:49 +0000 Subject: [PATCH 200/464] perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT PMU BugLink: https://bugs.launchpad.net/bugs/2149756 Adds PCIE-TGT PMU support in Tegra410 SOC. This PMU is instanced in each root complex in the SOC and it captures traffic originating from any source towards PCIE BAR and CXL HDM range. The traffic can be filtered based on the destination root port or target address range. Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit 3dd73022306bfdb29b1c33cb106fe337f46a6105) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 77 +++++ drivers/perf/arm_cspmu/nvidia_cspmu.c | 321 ++++++++++++++++++ 2 files changed, 398 insertions(+) diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index b8cfbb80be1c1..c065764d41fea 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -7,6 +7,7 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE +* PCIE-TGT PMU Driver ---------- @@ -212,6 +213,11 @@ Example usage: perf stat -a -e nvidia_pcie_pmu_0_rc_4/event=0x4,src_bdf=0x0180,src_bdf_en=0x1/ +.. _NVIDIA_T410_PCIE_PMU_RC_Mapping_Section: + +Mapping the RC# to lspci segment number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Mapping the RC# to lspci segment number can be non-trivial; hence a new NVIDIA Designated Vendor Specific Capability (DVSEC) register is added into the PCIE config space for each RP. This DVSEC has vendor id "10de" and DVSEC id of "0x4". The DVSEC register @@ -267,3 +273,74 @@ Example output:: 000d:40:00.0: Bus=40, Segment=0d, RP=01, RC=04, Socket=01 000d:c0:00.0: Bus=c0, Segment=0d, RP=02, RC=04, Socket=01 000e:00:00.0: Bus=00, Segment=0e, RP=00, RC=05, Socket=01 + +PCIE-TGT PMU +------------ + +This PMU is located in the SOC fabric connecting the PCIE root complex (RC) and +the memory subsystem. It monitors traffic targeting PCIE BAR and CXL HDM ranges. +There is one PCIE-TGT PMU per PCIE RC in the SoC. Each RC in Tegra410 SoC can +have up to 16 lanes that can be bifurcated into up to 8 root ports (RP). The PMU +provides RP filter to count PCIE BAR traffic to each RP and address filter to +count access to PCIE BAR or CXL HDM ranges. The details of the filters are +described in the following sections. + +Mapping the RC# to lspci segment number is similar to the PCIE PMU. Please see +:ref:`NVIDIA_T410_PCIE_PMU_RC_Mapping_Section` for more info. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_. + +The events in this PMU can be used to measure bandwidth and utilization: + + * rd_req: count the number of read requests to PCIE. + * wr_req: count the number of write requests to PCIE. + * rd_bytes: count the number of bytes transferred by rd_req. + * wr_bytes: count the number of bytes transferred by wr_req. + * cycles: count the clock cycles of SOC fabric connected to the PCIE interface. + +The average bandwidth is calculated as:: + + AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS + AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS + +The average request rate is calculated as:: + + AVG_RD_REQUEST_RATE = RD_REQ / CYCLES + AVG_WR_REQUEST_RATE = WR_REQ / CYCLES + +The PMU events can be filtered based on the destination root port or target +address range. Filtering based on RP is only available for PCIE BAR traffic. +Address filter works for both PCIE BAR and CXL HDM ranges. These filters can be +found in sysfs, see +/sys/bus/event_source/devices/nvidia_pcie_tgt_pmu__rc_/format/. + +Destination filter settings: + +* dst_rp_mask: bitmask to select the root port(s) to monitor. E.g. "dst_rp_mask=0xFF" + corresponds to all root ports (from 0 to 7) in the PCIE RC. Note that this filter is + only available for PCIE BAR traffic. +* dst_addr_base: BAR or CXL HDM filter base address. +* dst_addr_mask: BAR or CXL HDM filter address mask. +* dst_addr_en: enable BAR or CXL HDM address range filter. If this is set, the + address range specified by "dst_addr_base" and "dst_addr_mask" will be used to filter + the PCIE BAR and CXL HDM traffic address. The PMU uses the following comparison + to determine if the traffic destination address falls within the filter range:: + + (txn's addr & dst_addr_mask) == (dst_addr_base & dst_addr_mask) + + If the comparison succeeds, then the event will be counted. + +If the destination filter is not specified, the RP filter will be configured by default +to count PCIE BAR traffic to all root ports. + +Example usage: + +* Count event id 0x0 to root port 0 and 1 of PCIE RC-0 on socket 0:: + + perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_0/event=0x0,dst_rp_mask=0x3/ + +* Count event id 0x1 for accesses to PCIE BAR or CXL HDM address range + 0x10000 to 0x100FF on socket 0's PCIE RC-1:: + + perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ diff --git a/drivers/perf/arm_cspmu/nvidia_cspmu.c b/drivers/perf/arm_cspmu/nvidia_cspmu.c index 61fde84ea3434..bac83e424d6dc 100644 --- a/drivers/perf/arm_cspmu/nvidia_cspmu.c +++ b/drivers/perf/arm_cspmu/nvidia_cspmu.c @@ -42,6 +42,24 @@ #define NV_PCIE_V2_FILTER2_DST GENMASK_ULL(NV_PCIE_V2_DST_COUNT - 1, 0) #define NV_PCIE_V2_FILTER2_DEFAULT NV_PCIE_V2_FILTER2_DST +#define NV_PCIE_TGT_PORT_COUNT 8ULL +#define NV_PCIE_TGT_EV_TYPE_CC 0x4 +#define NV_PCIE_TGT_EV_TYPE_COUNT 3ULL +#define NV_PCIE_TGT_EV_TYPE_MASK GENMASK_ULL(NV_PCIE_TGT_EV_TYPE_COUNT - 1, 0) +#define NV_PCIE_TGT_FILTER2_MASK GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT, 0) +#define NV_PCIE_TGT_FILTER2_PORT GENMASK_ULL(NV_PCIE_TGT_PORT_COUNT - 1, 0) +#define NV_PCIE_TGT_FILTER2_ADDR_EN BIT(NV_PCIE_TGT_PORT_COUNT) +#define NV_PCIE_TGT_FILTER2_ADDR GENMASK_ULL(15, NV_PCIE_TGT_PORT_COUNT) +#define NV_PCIE_TGT_FILTER2_DEFAULT NV_PCIE_TGT_FILTER2_PORT + +#define NV_PCIE_TGT_ADDR_COUNT 8ULL +#define NV_PCIE_TGT_ADDR_STRIDE 20 +#define NV_PCIE_TGT_ADDR_CTRL 0xD38 +#define NV_PCIE_TGT_ADDR_BASE_LO 0xD3C +#define NV_PCIE_TGT_ADDR_BASE_HI 0xD40 +#define NV_PCIE_TGT_ADDR_MASK_LO 0xD44 +#define NV_PCIE_TGT_ADDR_MASK_HI 0xD48 + #define NV_GENERIC_FILTER_ID_MASK GENMASK_ULL(31, 0) #define NV_PRODID_MASK (PMIIDR_PRODUCTID | PMIIDR_VARIANT | PMIIDR_REVISION) @@ -185,6 +203,15 @@ static struct attribute *pcie_v2_pmu_event_attrs[] = { NULL }; +static struct attribute *pcie_tgt_pmu_event_attrs[] = { + ARM_CSPMU_EVENT_ATTR(rd_bytes, 0x0), + ARM_CSPMU_EVENT_ATTR(wr_bytes, 0x1), + ARM_CSPMU_EVENT_ATTR(rd_req, 0x2), + ARM_CSPMU_EVENT_ATTR(wr_req, 0x3), + ARM_CSPMU_EVENT_ATTR(cycles, NV_PCIE_TGT_EV_TYPE_CC), + NULL +}; + static struct attribute *generic_pmu_event_attrs[] = { ARM_CSPMU_EVENT_ATTR(cycles, ARM_CSPMU_EVT_CYCLES_DEFAULT), NULL, @@ -238,6 +265,15 @@ static struct attribute *pcie_v2_pmu_format_attrs[] = { NULL }; +static struct attribute *pcie_tgt_pmu_format_attrs[] = { + ARM_CSPMU_FORMAT_ATTR(event, "config:0-2"), + ARM_CSPMU_FORMAT_ATTR(dst_rp_mask, "config:3-10"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_en, "config:11"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_base, "config1:0-63"), + ARM_CSPMU_FORMAT_ATTR(dst_addr_mask, "config2:0-63"), + NULL +}; + static struct attribute *generic_pmu_format_attrs[] = { ARM_CSPMU_FORMAT_EVENT_ATTR, ARM_CSPMU_FORMAT_FILTER_ATTR, @@ -477,6 +513,267 @@ static int pcie_v2_pmu_validate_event(struct arm_cspmu *cspmu, return 0; } +struct pcie_tgt_addr_filter { + u32 refcount; + u64 base; + u64 mask; +}; + +struct pcie_tgt_data { + struct pcie_tgt_addr_filter addr_filter[NV_PCIE_TGT_ADDR_COUNT]; + void __iomem *addr_filter_reg; +}; + +#if defined(CONFIG_ACPI) && defined(CONFIG_ARM64) +static int pcie_tgt_init_data(struct arm_cspmu *cspmu) +{ + int ret; + struct acpi_device *adev; + struct pcie_tgt_data *data; + struct list_head resource_list; + struct resource_entry *rentry; + struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); + struct device *dev = cspmu->dev; + + data = devm_kzalloc(dev, sizeof(struct pcie_tgt_data), GFP_KERNEL); + if (!data) + return -ENOMEM; + + adev = arm_cspmu_acpi_dev_get(cspmu); + if (!adev) { + dev_err(dev, "failed to get associated PCIE-TGT device\n"); + return -ENODEV; + } + + INIT_LIST_HEAD(&resource_list); + ret = acpi_dev_get_memory_resources(adev, &resource_list); + if (ret < 0) { + dev_err(dev, "failed to get PCIE-TGT device memory resources\n"); + acpi_dev_put(adev); + return ret; + } + + rentry = list_first_entry_or_null( + &resource_list, struct resource_entry, node); + if (rentry) { + data->addr_filter_reg = devm_ioremap_resource(dev, rentry->res); + ret = 0; + } + + if (IS_ERR(data->addr_filter_reg)) { + dev_err(dev, "failed to get address filter resource\n"); + ret = PTR_ERR(data->addr_filter_reg); + } + + acpi_dev_free_resource_list(&resource_list); + acpi_dev_put(adev); + + ctx->data = data; + + return ret; +} +#else +static int pcie_tgt_init_data(struct arm_cspmu *cspmu) +{ + return -ENODEV; +} +#endif + +static struct pcie_tgt_data *pcie_tgt_get_data(struct arm_cspmu *cspmu) +{ + struct nv_cspmu_ctx *ctx = to_nv_cspmu_ctx(cspmu); + + return ctx->data; +} + +/* Find the first available address filter slot. */ +static int pcie_tgt_find_addr_idx(struct arm_cspmu *cspmu, u64 base, u64 mask, + bool is_reset) +{ + int i; + struct pcie_tgt_data *data = pcie_tgt_get_data(cspmu); + + for (i = 0; i < NV_PCIE_TGT_ADDR_COUNT; i++) { + if (!is_reset && data->addr_filter[i].refcount == 0) + return i; + + if (data->addr_filter[i].base == base && + data->addr_filter[i].mask == mask) + return i; + } + + return -ENODEV; +} + +static u32 pcie_tgt_pmu_event_filter(const struct perf_event *event) +{ + u32 filter; + + filter = (event->attr.config >> NV_PCIE_TGT_EV_TYPE_COUNT) & + NV_PCIE_TGT_FILTER2_MASK; + + return filter; +} + +static bool pcie_tgt_pmu_addr_en(const struct perf_event *event) +{ + u32 filter = pcie_tgt_pmu_event_filter(event); + + return FIELD_GET(NV_PCIE_TGT_FILTER2_ADDR_EN, filter) != 0; +} + +static u32 pcie_tgt_pmu_port_filter(const struct perf_event *event) +{ + u32 filter = pcie_tgt_pmu_event_filter(event); + + return FIELD_GET(NV_PCIE_TGT_FILTER2_PORT, filter); +} + +static u64 pcie_tgt_pmu_dst_addr_base(const struct perf_event *event) +{ + return event->attr.config1; +} + +static u64 pcie_tgt_pmu_dst_addr_mask(const struct perf_event *event) +{ + return event->attr.config2; +} + +static int pcie_tgt_pmu_validate_event(struct arm_cspmu *cspmu, + struct perf_event *new_ev) +{ + u64 base, mask; + int idx; + + if (!pcie_tgt_pmu_addr_en(new_ev)) + return 0; + + /* Make sure there is a slot available for the address filter. */ + base = pcie_tgt_pmu_dst_addr_base(new_ev); + mask = pcie_tgt_pmu_dst_addr_mask(new_ev); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); + if (idx < 0) + return -EINVAL; + + return 0; +} + +static void pcie_tgt_pmu_config_addr_filter(struct arm_cspmu *cspmu, + bool en, u64 base, u64 mask, int idx) +{ + struct pcie_tgt_data *data; + struct pcie_tgt_addr_filter *filter; + void __iomem *filter_reg; + + data = pcie_tgt_get_data(cspmu); + filter = &data->addr_filter[idx]; + filter_reg = data->addr_filter_reg + (idx * NV_PCIE_TGT_ADDR_STRIDE); + + if (en) { + filter->refcount++; + if (filter->refcount == 1) { + filter->base = base; + filter->mask = mask; + + writel(lower_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); + writel(upper_32_bits(base), filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); + writel(lower_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); + writel(upper_32_bits(mask), filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); + writel(1, filter_reg + NV_PCIE_TGT_ADDR_CTRL); + } + } else { + filter->refcount--; + if (filter->refcount == 0) { + writel(0, filter_reg + NV_PCIE_TGT_ADDR_CTRL); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_LO); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_BASE_HI); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_LO); + writel(0, filter_reg + NV_PCIE_TGT_ADDR_MASK_HI); + + filter->base = 0; + filter->mask = 0; + } + } +} + +static void pcie_tgt_pmu_set_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + bool addr_filter_en; + int idx; + u32 filter2_val, filter2_offset, port_filter; + u64 base, mask; + + filter2_val = 0; + filter2_offset = PMEVFILT2R + (4 * event->hw.idx); + + addr_filter_en = pcie_tgt_pmu_addr_en(event); + if (addr_filter_en) { + base = pcie_tgt_pmu_dst_addr_base(event); + mask = pcie_tgt_pmu_dst_addr_mask(event); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, false); + + if (idx < 0) { + dev_err(cspmu->dev, + "Unable to find a slot for address filtering\n"); + writel(0, cspmu->base0 + filter2_offset); + return; + } + + /* Configure address range filter registers.*/ + pcie_tgt_pmu_config_addr_filter(cspmu, true, base, mask, idx); + + /* Config the counter to use the selected address filter slot. */ + filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_ADDR, 1U << idx); + } + + port_filter = pcie_tgt_pmu_port_filter(event); + + /* Monitor all ports if no filter is selected. */ + if (!addr_filter_en && port_filter == 0) + port_filter = NV_PCIE_TGT_FILTER2_PORT; + + filter2_val |= FIELD_PREP(NV_PCIE_TGT_FILTER2_PORT, port_filter); + + writel(filter2_val, cspmu->base0 + filter2_offset); +} + +static void pcie_tgt_pmu_reset_ev_filter(struct arm_cspmu *cspmu, + const struct perf_event *event) +{ + bool addr_filter_en; + u64 base, mask; + int idx; + + addr_filter_en = pcie_tgt_pmu_addr_en(event); + if (!addr_filter_en) + return; + + base = pcie_tgt_pmu_dst_addr_base(event); + mask = pcie_tgt_pmu_dst_addr_mask(event); + idx = pcie_tgt_find_addr_idx(cspmu, base, mask, true); + + if (idx < 0) { + dev_err(cspmu->dev, + "Unable to find the address filter slot to reset\n"); + return; + } + + pcie_tgt_pmu_config_addr_filter(cspmu, false, base, mask, idx); +} + +static u32 pcie_tgt_pmu_event_type(const struct perf_event *event) +{ + return event->attr.config & NV_PCIE_TGT_EV_TYPE_MASK; +} + +static bool pcie_tgt_pmu_is_cycle_counter_event(const struct perf_event *event) +{ + u32 event_type = pcie_tgt_pmu_event_type(event); + + return event_type == NV_PCIE_TGT_EV_TYPE_CC; +} + enum nv_cspmu_name_fmt { NAME_FMT_GENERIC, NAME_FMT_SOCKET, @@ -618,6 +915,28 @@ static const struct nv_cspmu_match nv_cspmu_match[] = { .reset_ev_filter = nv_cspmu_reset_ev_filter, } }, + { + .prodid = 0x10700000, + .prodid_mask = NV_PRODID_MASK, + .name_pattern = "nvidia_pcie_tgt_pmu_%u_rc_%u", + .name_fmt = NAME_FMT_SOCKET_INST, + .template_ctx = { + .event_attr = pcie_tgt_pmu_event_attrs, + .format_attr = pcie_tgt_pmu_format_attrs, + .filter_mask = 0x0, + .filter_default_val = 0x0, + .filter2_mask = NV_PCIE_TGT_FILTER2_MASK, + .filter2_default_val = NV_PCIE_TGT_FILTER2_DEFAULT, + .init_data = pcie_tgt_init_data + }, + .ops = { + .is_cycle_counter_event = pcie_tgt_pmu_is_cycle_counter_event, + .event_type = pcie_tgt_pmu_event_type, + .validate_event = pcie_tgt_pmu_validate_event, + .set_ev_filter = pcie_tgt_pmu_set_ev_filter, + .reset_ev_filter = pcie_tgt_pmu_reset_ev_filter, + } + }, { .prodid = 0, .prodid_mask = 0, @@ -710,6 +1029,8 @@ static int nv_cspmu_init_ops(struct arm_cspmu *cspmu) /* NVIDIA specific callbacks. */ SET_OP(validate_event, impl_ops, match, NULL); + SET_OP(event_type, impl_ops, match, NULL); + SET_OP(is_cycle_counter_event, impl_ops, match, NULL); SET_OP(set_cc_filter, impl_ops, match, nv_cspmu_set_cc_filter); SET_OP(set_ev_filter, impl_ops, match, nv_cspmu_set_ev_filter); SET_OP(reset_ev_filter, impl_ops, match, NULL); From 4934e4e29e289fd834531f0d41c2bd7efdc2ed86 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:50 +0000 Subject: [PATCH 201/464] perf: add NVIDIA Tegra410 CPU Memory Latency PMU BugLink: https://bugs.launchpad.net/bugs/2149756 Adds CPU Memory (CMEM) Latency PMU support in Tegra410 SOC. The PMU is used to measure latency between the edge of the Unified Coherence Fabric to the local system DRAM. Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit 429b7638b2df5538e945aaa2cc189cf0d6e8fb3a) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 25 + drivers/perf/Kconfig | 7 + drivers/perf/Makefile | 1 + drivers/perf/nvidia_t410_cmem_latency_pmu.c | 736 ++++++++++++++++++ 4 files changed, 769 insertions(+) create mode 100644 drivers/perf/nvidia_t410_cmem_latency_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index c065764d41fea..9945c43f6a7a5 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -8,6 +8,7 @@ metrics like memory bandwidth, latency, and utilization: * Unified Coherence Fabric (UCF) * PCIE * PCIE-TGT +* CPU Memory (CMEM) Latency PMU Driver ---------- @@ -344,3 +345,27 @@ Example usage: 0x10000 to 0x100FF on socket 0's PCIE RC-1:: perf stat -a -e nvidia_pcie_tgt_pmu_0_rc_1/event=0x1,dst_addr_base=0x10000,dst_addr_mask=0xFFF00,dst_addr_en=0x1/ + +CPU Memory (CMEM) Latency PMU +----------------------------- + +This PMU monitors latency events of memory read requests from the edge of the +Unified Coherence Fabric (UCF) to local CPU DRAM: + + * RD_REQ counters: count read requests (32B per request). + * RD_CUM_OUTS counters: accumulated outstanding request counter, which track + how many cycles the read requests are in flight. + * CYCLES counter: counts the number of elapsed cycles. + +The average latency is calculated as:: + + FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + AVG_LATENCY_IN_CYCLES = RD_CUM_OUTS / RD_REQ + AVERAGE_LATENCY_IN_NS = AVG_LATENCY_IN_CYCLES / FREQ_IN_GHZ + +The events and configuration options of this PMU device are described in sysfs, +see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. + +Example usage:: + + perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 638321fc9800c..26e86067d8f93 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -311,4 +311,11 @@ config MARVELL_PEM_PMU Enable support for PCIe Interface performance monitoring on Marvell platform. +config NVIDIA_TEGRA410_CMEM_LATENCY_PMU + tristate "NVIDIA Tegra410 CPU Memory Latency PMU" + depends on ARM64 && ACPI + help + Enable perf support for CPU memory latency counters monitoring on + NVIDIA Tegra410 SoC. + endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index ea52711a87e32..4aa6aad393c2d 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -35,3 +35,4 @@ obj-$(CONFIG_DWC_PCIE_PMU) += dwc_pcie_pmu.o obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o +obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o diff --git a/drivers/perf/nvidia_t410_cmem_latency_pmu.c b/drivers/perf/nvidia_t410_cmem_latency_pmu.c new file mode 100644 index 0000000000000..acb8f5571522c --- /dev/null +++ b/drivers/perf/nvidia_t410_cmem_latency_pmu.c @@ -0,0 +1,736 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * NVIDIA Tegra410 CPU Memory (CMEM) Latency PMU driver. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NUM_INSTANCES 14 + +/* Register offsets. */ +#define CMEM_LAT_CG_CTRL 0x800 +#define CMEM_LAT_CTRL 0x808 +#define CMEM_LAT_STATUS 0x810 +#define CMEM_LAT_CYCLE_CNTR 0x818 +#define CMEM_LAT_MC0_REQ_CNTR 0x820 +#define CMEM_LAT_MC0_AOR_CNTR 0x830 +#define CMEM_LAT_MC1_REQ_CNTR 0x838 +#define CMEM_LAT_MC1_AOR_CNTR 0x848 +#define CMEM_LAT_MC2_REQ_CNTR 0x850 +#define CMEM_LAT_MC2_AOR_CNTR 0x860 + +/* CMEM_LAT_CTRL values. */ +#define CMEM_LAT_CTRL_DISABLE 0x0ULL +#define CMEM_LAT_CTRL_ENABLE 0x1ULL +#define CMEM_LAT_CTRL_CLR 0x2ULL + +/* CMEM_LAT_CG_CTRL values. */ +#define CMEM_LAT_CG_CTRL_DISABLE 0x0ULL +#define CMEM_LAT_CG_CTRL_ENABLE 0x1ULL + +/* CMEM_LAT_STATUS register field. */ +#define CMEM_LAT_STATUS_CYCLE_OVF BIT(0) +#define CMEM_LAT_STATUS_MC0_AOR_OVF BIT(1) +#define CMEM_LAT_STATUS_MC0_REQ_OVF BIT(3) +#define CMEM_LAT_STATUS_MC1_AOR_OVF BIT(4) +#define CMEM_LAT_STATUS_MC1_REQ_OVF BIT(6) +#define CMEM_LAT_STATUS_MC2_AOR_OVF BIT(7) +#define CMEM_LAT_STATUS_MC2_REQ_OVF BIT(9) + +/* Events. */ +#define CMEM_LAT_EVENT_CYCLES 0x0 +#define CMEM_LAT_EVENT_REQ 0x1 +#define CMEM_LAT_EVENT_AOR 0x2 + +#define CMEM_LAT_NUM_EVENTS 0x3 +#define CMEM_LAT_MASK_EVENT 0x3 +#define CMEM_LAT_MAX_ACTIVE_EVENTS 32 + +#define CMEM_LAT_ACTIVE_CPU_MASK 0x0 +#define CMEM_LAT_ASSOCIATED_CPU_MASK 0x1 + +static unsigned long cmem_lat_pmu_cpuhp_state; + +struct cmem_lat_pmu_hw_events { + struct perf_event *events[CMEM_LAT_MAX_ACTIVE_EVENTS]; + DECLARE_BITMAP(used_ctrs, CMEM_LAT_MAX_ACTIVE_EVENTS); +}; + +struct cmem_lat_pmu { + struct pmu pmu; + struct device *dev; + const char *name; + const char *identifier; + void __iomem *base_broadcast; + void __iomem *base[NUM_INSTANCES]; + cpumask_t associated_cpus; + cpumask_t active_cpu; + struct hlist_node node; + struct cmem_lat_pmu_hw_events hw_events; +}; + +#define to_cmem_lat_pmu(p) \ + container_of(p, struct cmem_lat_pmu, pmu) + + +/* Get event type from perf_event. */ +static inline u32 get_event_type(struct perf_event *event) +{ + return (event->attr.config) & CMEM_LAT_MASK_EVENT; +} + +/* PMU operations. */ +static int cmem_lat_pmu_get_event_idx(struct cmem_lat_pmu_hw_events *hw_events, + struct perf_event *event) +{ + unsigned int idx; + + idx = find_first_zero_bit(hw_events->used_ctrs, CMEM_LAT_MAX_ACTIVE_EVENTS); + if (idx >= CMEM_LAT_MAX_ACTIVE_EVENTS) + return -EAGAIN; + + set_bit(idx, hw_events->used_ctrs); + + return idx; +} + +static bool cmem_lat_pmu_validate_event(struct pmu *pmu, + struct cmem_lat_pmu_hw_events *hw_events, + struct perf_event *event) +{ + int ret; + + if (is_software_event(event)) + return true; + + /* Reject groups spanning multiple HW PMUs. */ + if (event->pmu != pmu) + return false; + + ret = cmem_lat_pmu_get_event_idx(hw_events, event); + if (ret < 0) + return false; + + return true; +} + +/* Make sure the group of events can be scheduled at once on the PMU. */ +static bool cmem_lat_pmu_validate_group(struct perf_event *event) +{ + struct perf_event *sibling, *leader = event->group_leader; + struct cmem_lat_pmu_hw_events fake_hw_events; + + if (event->group_leader == event) + return true; + + memset(&fake_hw_events, 0, sizeof(fake_hw_events)); + + if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, leader)) + return false; + + for_each_sibling_event(sibling, leader) { + if (!cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, sibling)) + return false; + } + + return cmem_lat_pmu_validate_event(event->pmu, &fake_hw_events, event); +} + +static int cmem_lat_pmu_event_init(struct perf_event *event) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u32 event_type = get_event_type(event); + + if (event->attr.type != event->pmu->type || + event_type >= CMEM_LAT_NUM_EVENTS) + return -ENOENT; + + /* + * Sampling, per-process mode, and per-task counters are not supported + * since this PMU is shared across all CPUs. + */ + if (is_sampling_event(event) || event->attach_state & PERF_ATTACH_TASK) { + dev_dbg(cmem_lat_pmu->pmu.dev, + "Can't support sampling and per-process mode\n"); + return -EOPNOTSUPP; + } + + if (event->cpu < 0) { + dev_dbg(cmem_lat_pmu->pmu.dev, "Can't support per-task counters\n"); + return -EINVAL; + } + + /* + * Make sure the CPU assignment is on one of the CPUs associated with + * this PMU. + */ + if (!cpumask_test_cpu(event->cpu, &cmem_lat_pmu->associated_cpus)) { + dev_dbg(cmem_lat_pmu->pmu.dev, + "Requested cpu is not associated with the PMU\n"); + return -EINVAL; + } + + /* Enforce the current active CPU to handle the events in this PMU. */ + event->cpu = cpumask_first(&cmem_lat_pmu->active_cpu); + if (event->cpu >= nr_cpu_ids) + return -EINVAL; + + if (!cmem_lat_pmu_validate_group(event)) + return -EINVAL; + + hwc->idx = -1; + hwc->config = event_type; + + return 0; +} + +static u64 cmem_lat_pmu_read_status(struct cmem_lat_pmu *cmem_lat_pmu, + unsigned int inst) +{ + return readq(cmem_lat_pmu->base[inst] + CMEM_LAT_STATUS); +} + +static u64 cmem_lat_pmu_read_cycle_counter(struct perf_event *event) +{ + const unsigned int instance = 0; + u64 status; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* + * Use the reading from first instance since all instances are + * identical. + */ + status = cmem_lat_pmu_read_status(cmem_lat_pmu, instance); + if (status & CMEM_LAT_STATUS_CYCLE_OVF) + dev_warn(dev, "Cycle counter overflow\n"); + + return readq(cmem_lat_pmu->base[instance] + CMEM_LAT_CYCLE_CNTR); +} + +static u64 cmem_lat_pmu_read_req_counter(struct perf_event *event) +{ + unsigned int i; + u64 status, val = 0; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* Sum up the counts from all instances. */ + for (i = 0; i < NUM_INSTANCES; i++) { + status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); + if (status & CMEM_LAT_STATUS_MC0_REQ_OVF) + dev_warn(dev, "MC0 request counter overflow\n"); + if (status & CMEM_LAT_STATUS_MC1_REQ_OVF) + dev_warn(dev, "MC1 request counter overflow\n"); + if (status & CMEM_LAT_STATUS_MC2_REQ_OVF) + dev_warn(dev, "MC2 request counter overflow\n"); + + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC0_REQ_CNTR); + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC1_REQ_CNTR); + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC2_REQ_CNTR); + } + + return val; +} + +static u64 cmem_lat_pmu_read_aor_counter(struct perf_event *event) +{ + unsigned int i; + u64 status, val = 0; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct device *dev = cmem_lat_pmu->dev; + + /* Sum up the counts from all instances. */ + for (i = 0; i < NUM_INSTANCES; i++) { + status = cmem_lat_pmu_read_status(cmem_lat_pmu, i); + if (status & CMEM_LAT_STATUS_MC0_AOR_OVF) + dev_warn(dev, "MC0 AOR counter overflow\n"); + if (status & CMEM_LAT_STATUS_MC1_AOR_OVF) + dev_warn(dev, "MC1 AOR counter overflow\n"); + if (status & CMEM_LAT_STATUS_MC2_AOR_OVF) + dev_warn(dev, "MC2 AOR counter overflow\n"); + + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC0_AOR_CNTR); + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC1_AOR_CNTR); + val += readq(cmem_lat_pmu->base[i] + CMEM_LAT_MC2_AOR_CNTR); + } + + return val; +} + +static u64 (*read_counter_fn[CMEM_LAT_NUM_EVENTS])(struct perf_event *) = { + [CMEM_LAT_EVENT_CYCLES] = cmem_lat_pmu_read_cycle_counter, + [CMEM_LAT_EVENT_REQ] = cmem_lat_pmu_read_req_counter, + [CMEM_LAT_EVENT_AOR] = cmem_lat_pmu_read_aor_counter, +}; + +static void cmem_lat_pmu_event_update(struct perf_event *event) +{ + u32 event_type; + u64 prev, now; + struct hw_perf_event *hwc = &event->hw; + + if (hwc->state & PERF_HES_STOPPED) + return; + + event_type = hwc->config; + + do { + prev = local64_read(&hwc->prev_count); + now = read_counter_fn[event_type](event); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); + + local64_add(now - prev, &event->count); + + hwc->state |= PERF_HES_UPTODATE; +} + +static void cmem_lat_pmu_start(struct perf_event *event, int pmu_flags) +{ + event->hw.state = 0; +} + +static void cmem_lat_pmu_stop(struct perf_event *event, int pmu_flags) +{ + event->hw.state |= PERF_HES_STOPPED; +} + +static int cmem_lat_pmu_add(struct perf_event *event, int flags) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx; + + if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &cmem_lat_pmu->associated_cpus))) + return -ENOENT; + + idx = cmem_lat_pmu_get_event_idx(hw_events, event); + if (idx < 0) + return idx; + + hw_events->events[idx] = event; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + cmem_lat_pmu_start(event, PERF_EF_RELOAD); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void cmem_lat_pmu_del(struct perf_event *event, int flags) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(event->pmu); + struct cmem_lat_pmu_hw_events *hw_events = &cmem_lat_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + cmem_lat_pmu_stop(event, PERF_EF_UPDATE); + + hw_events->events[idx] = NULL; + + clear_bit(idx, hw_events->used_ctrs); + + perf_event_update_userpage(event); +} + +static void cmem_lat_pmu_read(struct perf_event *event) +{ + cmem_lat_pmu_event_update(event); +} + +static inline void cmem_lat_pmu_cg_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, + u64 val) +{ + writeq(val, cmem_lat_pmu->base_broadcast + CMEM_LAT_CG_CTRL); +} + +static inline void cmem_lat_pmu_ctrl(struct cmem_lat_pmu *cmem_lat_pmu, u64 val) +{ + writeq(val, cmem_lat_pmu->base_broadcast + CMEM_LAT_CTRL); +} + +static void cmem_lat_pmu_enable(struct pmu *pmu) +{ + bool disabled; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + + disabled = bitmap_empty(cmem_lat_pmu->hw_events.used_ctrs, + CMEM_LAT_MAX_ACTIVE_EVENTS); + + if (disabled) + return; + + /* Enable all the counters. */ + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CMEM_LAT_CG_CTRL_ENABLE); + cmem_lat_pmu_ctrl(cmem_lat_pmu, CMEM_LAT_CTRL_ENABLE); +} + +static void cmem_lat_pmu_disable(struct pmu *pmu) +{ + int idx; + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + + /* Disable all the counters. */ + cmem_lat_pmu_ctrl(cmem_lat_pmu, CMEM_LAT_CTRL_DISABLE); + + /* + * The counters will start from 0 again on restart. + * Update the events immediately to avoid losing the counts. + */ + for_each_set_bit(idx, cmem_lat_pmu->hw_events.used_ctrs, + CMEM_LAT_MAX_ACTIVE_EVENTS) { + struct perf_event *event = cmem_lat_pmu->hw_events.events[idx]; + + if (!event) + continue; + + cmem_lat_pmu_event_update(event); + + local64_set(&event->hw.prev_count, 0ULL); + } + + cmem_lat_pmu_ctrl(cmem_lat_pmu, CMEM_LAT_CTRL_CLR); + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CMEM_LAT_CG_CTRL_DISABLE); +} + +/* PMU identifier attribute. */ + +static ssize_t cmem_lat_pmu_identifier_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(dev_get_drvdata(dev)); + + return sysfs_emit(page, "%s\n", cmem_lat_pmu->identifier); +} + +static struct device_attribute cmem_lat_pmu_identifier_attr = + __ATTR(identifier, 0444, cmem_lat_pmu_identifier_show, NULL); + +static struct attribute *cmem_lat_pmu_identifier_attrs[] = { + &cmem_lat_pmu_identifier_attr.attr, + NULL +}; + +static struct attribute_group cmem_lat_pmu_identifier_attr_group = { + .attrs = cmem_lat_pmu_identifier_attrs, +}; + +/* Format attributes. */ + +#define NV_PMU_EXT_ATTR(_name, _func, _config) \ + (&((struct dev_ext_attribute[]){ \ + { \ + .attr = __ATTR(_name, 0444, _func, NULL), \ + .var = (void *)_config \ + } \ + })[0].attr.attr) + +static struct attribute *cmem_lat_pmu_formats[] = { + NV_PMU_EXT_ATTR(event, device_show_string, "config:0-1"), + NULL +}; + +static const struct attribute_group cmem_lat_pmu_format_group = { + .name = "format", + .attrs = cmem_lat_pmu_formats, +}; + +/* Event attributes. */ + +static ssize_t cmem_lat_pmu_sysfs_event_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, typeof(*pmu_attr), attr); + return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); +} + +#define NV_PMU_EVENT_ATTR(_name, _config) \ + PMU_EVENT_ATTR_ID(_name, cmem_lat_pmu_sysfs_event_show, _config) + +static struct attribute *cmem_lat_pmu_events[] = { + NV_PMU_EVENT_ATTR(cycles, CMEM_LAT_EVENT_CYCLES), + NV_PMU_EVENT_ATTR(rd_req, CMEM_LAT_EVENT_REQ), + NV_PMU_EVENT_ATTR(rd_cum_outs, CMEM_LAT_EVENT_AOR), + NULL +}; + +static const struct attribute_group cmem_lat_pmu_events_group = { + .name = "events", + .attrs = cmem_lat_pmu_events, +}; + +/* Cpumask attributes. */ + +static ssize_t cmem_lat_pmu_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pmu *pmu = dev_get_drvdata(dev); + struct cmem_lat_pmu *cmem_lat_pmu = to_cmem_lat_pmu(pmu); + struct dev_ext_attribute *eattr = + container_of(attr, struct dev_ext_attribute, attr); + unsigned long mask_id = (unsigned long)eattr->var; + const cpumask_t *cpumask; + + switch (mask_id) { + case CMEM_LAT_ACTIVE_CPU_MASK: + cpumask = &cmem_lat_pmu->active_cpu; + break; + case CMEM_LAT_ASSOCIATED_CPU_MASK: + cpumask = &cmem_lat_pmu->associated_cpus; + break; + default: + return 0; + } + return cpumap_print_to_pagebuf(true, buf, cpumask); +} + +#define NV_PMU_CPUMASK_ATTR(_name, _config) \ + NV_PMU_EXT_ATTR(_name, cmem_lat_pmu_cpumask_show, \ + (unsigned long)_config) + +static struct attribute *cmem_lat_pmu_cpumask_attrs[] = { + NV_PMU_CPUMASK_ATTR(cpumask, CMEM_LAT_ACTIVE_CPU_MASK), + NV_PMU_CPUMASK_ATTR(associated_cpus, CMEM_LAT_ASSOCIATED_CPU_MASK), + NULL +}; + +static const struct attribute_group cmem_lat_pmu_cpumask_attr_group = { + .attrs = cmem_lat_pmu_cpumask_attrs, +}; + +/* Per PMU device attribute groups. */ + +static const struct attribute_group *cmem_lat_pmu_attr_groups[] = { + &cmem_lat_pmu_identifier_attr_group, + &cmem_lat_pmu_format_group, + &cmem_lat_pmu_events_group, + &cmem_lat_pmu_cpumask_attr_group, + NULL +}; + +static int cmem_lat_pmu_cpu_online(unsigned int cpu, struct hlist_node *node) +{ + struct cmem_lat_pmu *cmem_lat_pmu = + hlist_entry_safe(node, struct cmem_lat_pmu, node); + + if (!cpumask_test_cpu(cpu, &cmem_lat_pmu->associated_cpus)) + return 0; + + /* If the PMU is already managed, there is nothing to do */ + if (!cpumask_empty(&cmem_lat_pmu->active_cpu)) + return 0; + + /* Use this CPU for event counting */ + cpumask_set_cpu(cpu, &cmem_lat_pmu->active_cpu); + + return 0; +} + +static int cmem_lat_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) +{ + unsigned int dst; + + struct cmem_lat_pmu *cmem_lat_pmu = + hlist_entry_safe(node, struct cmem_lat_pmu, node); + + /* Nothing to do if this CPU doesn't own the PMU */ + if (!cpumask_test_and_clear_cpu(cpu, &cmem_lat_pmu->active_cpu)) + return 0; + + /* Choose a new CPU to migrate ownership of the PMU to */ + dst = cpumask_any_and_but(&cmem_lat_pmu->associated_cpus, + cpu_online_mask, cpu); + if (dst >= nr_cpu_ids) + return 0; + + /* Use this CPU for event counting */ + perf_pmu_migrate_context(&cmem_lat_pmu->pmu, cpu, dst); + cpumask_set_cpu(dst, &cmem_lat_pmu->active_cpu); + + return 0; +} + +static int cmem_lat_pmu_get_cpus(struct cmem_lat_pmu *cmem_lat_pmu, + unsigned int socket) +{ + int cpu; + + for_each_possible_cpu(cpu) { + if (cpu_to_node(cpu) == socket) + cpumask_set_cpu(cpu, &cmem_lat_pmu->associated_cpus); + } + + if (cpumask_empty(&cmem_lat_pmu->associated_cpus)) { + dev_dbg(cmem_lat_pmu->dev, + "No cpu associated with PMU socket-%u\n", socket); + return -ENODEV; + } + + return 0; +} + +static int cmem_lat_pmu_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct acpi_device *acpi_dev; + struct cmem_lat_pmu *cmem_lat_pmu; + char *name, *uid_str; + int ret, i; + u32 socket; + + acpi_dev = ACPI_COMPANION(dev); + if (!acpi_dev) + return -ENODEV; + + uid_str = acpi_device_uid(acpi_dev); + if (!uid_str) + return -ENODEV; + + ret = kstrtou32(uid_str, 0, &socket); + if (ret) + return ret; + + cmem_lat_pmu = devm_kzalloc(dev, sizeof(*cmem_lat_pmu), GFP_KERNEL); + name = devm_kasprintf(dev, GFP_KERNEL, "nvidia_cmem_latency_pmu_%u", socket); + if (!cmem_lat_pmu || !name) + return -ENOMEM; + + cmem_lat_pmu->dev = dev; + cmem_lat_pmu->name = name; + cmem_lat_pmu->identifier = acpi_device_hid(acpi_dev); + platform_set_drvdata(pdev, cmem_lat_pmu); + + cmem_lat_pmu->pmu = (struct pmu) { + .parent = &pdev->dev, + .task_ctx_nr = perf_invalid_context, + .pmu_enable = cmem_lat_pmu_enable, + .pmu_disable = cmem_lat_pmu_disable, + .event_init = cmem_lat_pmu_event_init, + .add = cmem_lat_pmu_add, + .del = cmem_lat_pmu_del, + .start = cmem_lat_pmu_start, + .stop = cmem_lat_pmu_stop, + .read = cmem_lat_pmu_read, + .attr_groups = cmem_lat_pmu_attr_groups, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | + PERF_PMU_CAP_NO_INTERRUPT, + }; + + /* Map the address of all the instances. */ + for (i = 0; i < NUM_INSTANCES; i++) { + cmem_lat_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); + if (IS_ERR(cmem_lat_pmu->base[i])) { + dev_err(dev, "Failed map address for instance %d\n", i); + return PTR_ERR(cmem_lat_pmu->base[i]); + } + } + + /* Map broadcast address. */ + cmem_lat_pmu->base_broadcast = devm_platform_ioremap_resource(pdev, + NUM_INSTANCES); + if (IS_ERR(cmem_lat_pmu->base_broadcast)) { + dev_err(dev, "Failed map broadcast address\n"); + return PTR_ERR(cmem_lat_pmu->base_broadcast); + } + + ret = cmem_lat_pmu_get_cpus(cmem_lat_pmu, socket); + if (ret) + return ret; + + ret = cpuhp_state_add_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); + if (ret) { + dev_err(&pdev->dev, "Error %d registering hotplug\n", ret); + return ret; + } + + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CMEM_LAT_CG_CTRL_ENABLE); + cmem_lat_pmu_ctrl(cmem_lat_pmu, CMEM_LAT_CTRL_CLR); + cmem_lat_pmu_cg_ctrl(cmem_lat_pmu, CMEM_LAT_CG_CTRL_DISABLE); + + ret = perf_pmu_register(&cmem_lat_pmu->pmu, name, -1); + if (ret) { + dev_err(&pdev->dev, "Failed to register PMU: %d\n", ret); + cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); + return ret; + } + + dev_dbg(&pdev->dev, "Registered %s PMU\n", name); + + return 0; +} + +static void cmem_lat_pmu_device_remove(struct platform_device *pdev) +{ + struct cmem_lat_pmu *cmem_lat_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&cmem_lat_pmu->pmu); + cpuhp_state_remove_instance(cmem_lat_pmu_cpuhp_state, + &cmem_lat_pmu->node); +} + +static const struct acpi_device_id cmem_lat_pmu_acpi_match[] = { + { "NVDA2021" }, + { } +}; +MODULE_DEVICE_TABLE(acpi, cmem_lat_pmu_acpi_match); + +static struct platform_driver cmem_lat_pmu_driver = { + .driver = { + .name = "nvidia-t410-cmem-latency-pmu", + .acpi_match_table = ACPI_PTR(cmem_lat_pmu_acpi_match), + .suppress_bind_attrs = true, + }, + .probe = cmem_lat_pmu_probe, + .remove = cmem_lat_pmu_device_remove, +}; + +static int __init cmem_lat_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/nvidia/cmem_latency:online", + cmem_lat_pmu_cpu_online, + cmem_lat_pmu_cpu_teardown); + if (ret < 0) + return ret; + + cmem_lat_pmu_cpuhp_state = ret; + + return platform_driver_register(&cmem_lat_pmu_driver); +} + +static void __exit cmem_lat_pmu_exit(void) +{ + platform_driver_unregister(&cmem_lat_pmu_driver); + cpuhp_remove_multi_state(cmem_lat_pmu_cpuhp_state); +} + +module_init(cmem_lat_pmu_init); +module_exit(cmem_lat_pmu_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("NVIDIA Tegra410 CPU Memory Latency PMU driver"); +MODULE_AUTHOR("Besar Wicaksono "); From a1352e85b67c1ba8512254b468cf1a52d0fc3ffe Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Tue, 24 Mar 2026 01:29:51 +0000 Subject: [PATCH 202/464] perf: add NVIDIA Tegra410 C2C PMU BugLink: https://bugs.launchpad.net/bugs/2149756 Adds NVIDIA C2C PMU support in Tegra410 SOC. This PMU is used to measure memory latency between the SOC and device memory, e.g GPU Memory (GMEM), CXL Memory, or memory on remote Tegra410 SOC. Reviewed-by: Ilkka Koskinen Signed-off-by: Besar Wicaksono Signed-off-by: Will Deacon (cherry picked from commit 2f89b7f78c50ca973ca035ceb30426f78d9e0996) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../admin-guide/perf/nvidia-tegra410-pmu.rst | 151 +++ drivers/perf/Kconfig | 7 + drivers/perf/Makefile | 1 + drivers/perf/nvidia_t410_c2c_pmu.c | 1051 +++++++++++++++++ 4 files changed, 1210 insertions(+) create mode 100644 drivers/perf/nvidia_t410_c2c_pmu.c diff --git a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst index 9945c43f6a7a5..0656223b61d47 100644 --- a/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst +++ b/Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst @@ -9,6 +9,9 @@ metrics like memory bandwidth, latency, and utilization: * PCIE * PCIE-TGT * CPU Memory (CMEM) Latency +* NVLink-C2C +* NV-CLink +* NV-DLink PMU Driver ---------- @@ -369,3 +372,151 @@ see /sys/bus/event_source/devices/nvidia_cmem_latency_pmu_. Example usage:: perf stat -a -e '{nvidia_cmem_latency_pmu_0/rd_req/,nvidia_cmem_latency_pmu_0/rd_cum_outs/,nvidia_cmem_latency_pmu_0/cycles/}' + +NVLink-C2C PMU +-------------- + +This PMU monitors latency events of memory read/write requests that pass through +the NVIDIA Chip-to-Chip (C2C) interface. Bandwidth events are not available +in this PMU, unlike the C2C PMU in Grace (Tegra241 SoC). + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. + * IN_RD_REQ: the number of incoming read requests. + * IN_WR_CUM_OUTS: accumulated outstanding request (in cycles) of incoming write requests. + * IN_WR_REQ: the number of incoming write requests. + * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. + * OUT_RD_REQ: the number of outgoing read requests. + * OUT_WR_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing write requests. + * OUT_WR_REQ: the number of outgoing write requests. + * CYCLES: NVLink-C2C interface cycle counts. + +The incoming events count the reads/writes from remote device to the SoC. +The outgoing events count the reads/writes from the SoC to remote device. + +The sysfs /sys/bus/event_source/devices/nvidia_nvlink_c2c_pmu_/peer +contains the information about the connected device. + +When the C2C interface is connected to GPU(s), the user can use the +"gpu_mask" parameter to filter traffic to/from specific GPU(s). Each bit represents the GPU +index, e.g. "gpu_mask=0x1" corresponds to GPU 0 and "gpu_mask=0x3" is for GPU 0 and 1. +The PMU will monitor all GPUs by default if not specified. + +When connected to another SoC, only the read events are available. + +The events can be used to calculate the average latency of the read/write requests:: + + C2C_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + IN_WR_AVG_LATENCY_IN_CYCLES = IN_WR_CUM_OUTS / IN_WR_REQ + IN_WR_AVG_LATENCY_IN_NS = IN_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ + OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + + OUT_WR_AVG_LATENCY_IN_CYCLES = OUT_WR_CUM_OUTS / OUT_WR_REQ + OUT_WR_AVG_LATENCY_IN_NS = OUT_WR_AVG_LATENCY_IN_CYCLES / C2C_FREQ_IN_GHZ + +Example usage: + + * Count incoming traffic from all GPUs connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_req/ + + * Count incoming traffic from GPU 0 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x1/ + + * Count incoming traffic from GPU 1 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/in_rd_cum_outs,gpu_mask=0x2/ + + * Count outgoing traffic to all GPUs connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_req/ + + * Count outgoing traffic to GPU 0 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x1/ + + * Count outgoing traffic to GPU 1 connected via NVLink-C2C:: + + perf stat -a -e nvidia_nvlink_c2c_pmu_0/out_rd_cum_outs,gpu_mask=0x2/ + +NV-CLink PMU +------------ + +This PMU monitors latency events of memory read requests that pass through +the NV-CLINK interface. Bandwidth events are not available in this PMU. +In Tegra410 SoC, the NV-CLink interface is used to connect to another Tegra410 +SoC and this PMU only counts read traffic. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvclink_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding request (in cycles) of incoming read requests. + * IN_RD_REQ: the number of incoming read requests. + * OUT_RD_CUM_OUTS: accumulated outstanding request (in cycles) of outgoing read requests. + * OUT_RD_REQ: the number of outgoing read requests. + * CYCLES: NV-CLINK interface cycle counts. + +The incoming events count the reads from remote device to the SoC. +The outgoing events count the reads from the SoC to remote device. + +The events can be used to calculate the average latency of the read requests:: + + CLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ + + OUT_RD_AVG_LATENCY_IN_CYCLES = OUT_RD_CUM_OUTS / OUT_RD_REQ + OUT_RD_AVG_LATENCY_IN_NS = OUT_RD_AVG_LATENCY_IN_CYCLES / CLINK_FREQ_IN_GHZ + +Example usage: + + * Count incoming read traffic from remote SoC connected via NV-CLINK:: + + perf stat -a -e nvidia_nvclink_pmu_0/in_rd_req/ + + * Count outgoing read traffic to remote SoC connected via NV-CLINK:: + + perf stat -a -e nvidia_nvclink_pmu_0/out_rd_req/ + +NV-DLink PMU +------------ + +This PMU monitors latency events of memory read requests that pass through +the NV-DLINK interface. Bandwidth events are not available in this PMU. +In Tegra410 SoC, this PMU only counts CXL memory read traffic. + +The events and configuration options of this PMU device are available in sysfs, +see /sys/bus/event_source/devices/nvidia_nvdlink_pmu_. + +The list of events: + + * IN_RD_CUM_OUTS: accumulated outstanding read requests (in cycles) to CXL memory. + * IN_RD_REQ: the number of read requests to CXL memory. + * CYCLES: NV-DLINK interface cycle counts. + +The events can be used to calculate the average latency of the read requests:: + + DLINK_FREQ_IN_GHZ = CYCLES / ELAPSED_TIME_IN_NS + + IN_RD_AVG_LATENCY_IN_CYCLES = IN_RD_CUM_OUTS / IN_RD_REQ + IN_RD_AVG_LATENCY_IN_NS = IN_RD_AVG_LATENCY_IN_CYCLES / DLINK_FREQ_IN_GHZ + +Example usage: + + * Count read events to CXL memory:: + + perf stat -a -e '{nvidia_nvdlink_pmu_0/in_rd_req/,nvidia_nvdlink_pmu_0/in_rd_cum_outs/}' diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 26e86067d8f93..ab90932fc2d01 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -318,4 +318,11 @@ config NVIDIA_TEGRA410_CMEM_LATENCY_PMU Enable perf support for CPU memory latency counters monitoring on NVIDIA Tegra410 SoC. +config NVIDIA_TEGRA410_C2C_PMU + tristate "NVIDIA Tegra410 C2C PMU" + depends on ARM64 && ACPI + help + Enable perf support for counters in NVIDIA C2C interface of NVIDIA + Tegra410 SoC. + endmenu diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile index 4aa6aad393c2d..eb8a022dad9a7 100644 --- a/drivers/perf/Makefile +++ b/drivers/perf/Makefile @@ -36,3 +36,4 @@ obj-$(CONFIG_ARM_CORESIGHT_PMU_ARCH_SYSTEM_PMU) += arm_cspmu/ obj-$(CONFIG_MESON_DDR_PMU) += amlogic/ obj-$(CONFIG_CXL_PMU) += cxl_pmu.o obj-$(CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU) += nvidia_t410_cmem_latency_pmu.o +obj-$(CONFIG_NVIDIA_TEGRA410_C2C_PMU) += nvidia_t410_c2c_pmu.o diff --git a/drivers/perf/nvidia_t410_c2c_pmu.c b/drivers/perf/nvidia_t410_c2c_pmu.c new file mode 100644 index 0000000000000..411987153ff3f --- /dev/null +++ b/drivers/perf/nvidia_t410_c2c_pmu.c @@ -0,0 +1,1051 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * NVIDIA Tegra410 C2C PMU driver. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* The C2C interface types in Tegra410. */ +#define C2C_TYPE_NVLINK 0x0 +#define C2C_TYPE_NVCLINK 0x1 +#define C2C_TYPE_NVDLINK 0x2 +#define C2C_TYPE_COUNT 0x3 + +/* The type of the peer device connected to the C2C interface. */ +#define C2C_PEER_TYPE_CPU 0x0 +#define C2C_PEER_TYPE_GPU 0x1 +#define C2C_PEER_TYPE_CXLMEM 0x2 +#define C2C_PEER_TYPE_COUNT 0x3 + +/* The number of peer devices can be connected to the C2C interface. */ +#define C2C_NR_PEER_CPU 0x1 +#define C2C_NR_PEER_GPU 0x2 +#define C2C_NR_PEER_CXLMEM 0x1 +#define C2C_NR_PEER_MAX 0x2 + +/* Number of instances on each interface. */ +#define C2C_NR_INST_NVLINK 14 +#define C2C_NR_INST_NVCLINK 12 +#define C2C_NR_INST_NVDLINK 16 +#define C2C_NR_INST_MAX 16 + +/* Register offsets. */ +#define C2C_CTRL 0x864 +#define C2C_IN_STATUS 0x868 +#define C2C_CYCLE_CNTR 0x86c +#define C2C_IN_RD_CUM_OUTS_CNTR 0x874 +#define C2C_IN_RD_REQ_CNTR 0x87c +#define C2C_IN_WR_CUM_OUTS_CNTR 0x884 +#define C2C_IN_WR_REQ_CNTR 0x88c +#define C2C_OUT_STATUS 0x890 +#define C2C_OUT_RD_CUM_OUTS_CNTR 0x898 +#define C2C_OUT_RD_REQ_CNTR 0x8a0 +#define C2C_OUT_WR_CUM_OUTS_CNTR 0x8a8 +#define C2C_OUT_WR_REQ_CNTR 0x8b0 + +/* C2C_IN_STATUS register field. */ +#define C2C_IN_STATUS_CYCLE_OVF BIT(0) +#define C2C_IN_STATUS_IN_RD_CUM_OUTS_OVF BIT(1) +#define C2C_IN_STATUS_IN_RD_REQ_OVF BIT(2) +#define C2C_IN_STATUS_IN_WR_CUM_OUTS_OVF BIT(3) +#define C2C_IN_STATUS_IN_WR_REQ_OVF BIT(4) + +/* C2C_OUT_STATUS register field. */ +#define C2C_OUT_STATUS_OUT_RD_CUM_OUTS_OVF BIT(0) +#define C2C_OUT_STATUS_OUT_RD_REQ_OVF BIT(1) +#define C2C_OUT_STATUS_OUT_WR_CUM_OUTS_OVF BIT(2) +#define C2C_OUT_STATUS_OUT_WR_REQ_OVF BIT(3) + +/* Events. */ +#define C2C_EVENT_CYCLES 0x0 +#define C2C_EVENT_IN_RD_CUM_OUTS 0x1 +#define C2C_EVENT_IN_RD_REQ 0x2 +#define C2C_EVENT_IN_WR_CUM_OUTS 0x3 +#define C2C_EVENT_IN_WR_REQ 0x4 +#define C2C_EVENT_OUT_RD_CUM_OUTS 0x5 +#define C2C_EVENT_OUT_RD_REQ 0x6 +#define C2C_EVENT_OUT_WR_CUM_OUTS 0x7 +#define C2C_EVENT_OUT_WR_REQ 0x8 + +#define C2C_NUM_EVENTS 0x9 +#define C2C_MASK_EVENT 0xFF +#define C2C_MAX_ACTIVE_EVENTS 32 + +#define C2C_ACTIVE_CPU_MASK 0x0 +#define C2C_ASSOCIATED_CPU_MASK 0x1 + +/* + * Maximum poll count for reading counter value using high-low-high sequence. + */ +#define HILOHI_MAX_POLL 1000 + +static unsigned long nv_c2c_pmu_cpuhp_state; + +/* PMU descriptor. */ + +/* C2C type information. */ +struct nv_c2c_pmu_data { + unsigned int c2c_type; + unsigned int nr_inst; + const char *name_fmt; +}; + +static const struct nv_c2c_pmu_data nv_c2c_pmu_data[] = { + [C2C_TYPE_NVLINK] = { + .c2c_type = C2C_TYPE_NVLINK, + .nr_inst = C2C_NR_INST_NVLINK, + .name_fmt = "nvidia_nvlink_c2c_pmu_%u", + }, + [C2C_TYPE_NVCLINK] = { + .c2c_type = C2C_TYPE_NVCLINK, + .nr_inst = C2C_NR_INST_NVCLINK, + .name_fmt = "nvidia_nvclink_pmu_%u", + }, + [C2C_TYPE_NVDLINK] = { + .c2c_type = C2C_TYPE_NVDLINK, + .nr_inst = C2C_NR_INST_NVDLINK, + .name_fmt = "nvidia_nvdlink_pmu_%u", + }, +}; + +/* Tracks the events assigned to the PMU for a given logical index. */ +struct nv_c2c_pmu_hw_events { + /* The events that are active. */ + struct perf_event *events[C2C_MAX_ACTIVE_EVENTS]; + + /* + * Each bit indicates a logical counter is being used (or not) for an + * event. + */ + DECLARE_BITMAP(used_ctrs, C2C_MAX_ACTIVE_EVENTS); +}; + +struct nv_c2c_pmu { + struct pmu pmu; + struct device *dev; + struct acpi_device *acpi_dev; + + const char *name; + const char *identifier; + + const struct nv_c2c_pmu_data *data; + unsigned int peer_type; + unsigned int socket; + unsigned int nr_peer; + unsigned long peer_insts[C2C_NR_PEER_MAX][BITS_TO_LONGS(C2C_NR_INST_MAX)]; + u32 filter_default; + + struct nv_c2c_pmu_hw_events hw_events; + + cpumask_t associated_cpus; + cpumask_t active_cpu; + + struct hlist_node cpuhp_node; + + const struct attribute_group **attr_groups; + + void __iomem *base_broadcast; + void __iomem *base[C2C_NR_INST_MAX]; +}; + +#define to_c2c_pmu(p) (container_of(p, struct nv_c2c_pmu, pmu)) + +/* Get event type from perf_event. */ +static inline u32 get_event_type(struct perf_event *event) +{ + return (event->attr.config) & C2C_MASK_EVENT; +} + +static inline u32 get_filter_mask(struct perf_event *event) +{ + u32 filter; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + + filter = ((u32)event->attr.config1) & c2c_pmu->filter_default; + if (filter == 0) + filter = c2c_pmu->filter_default; + + return filter; +} + +/* PMU operations. */ + +static int nv_c2c_pmu_get_event_idx(struct nv_c2c_pmu_hw_events *hw_events, + struct perf_event *event) +{ + u32 idx; + + idx = find_first_zero_bit(hw_events->used_ctrs, C2C_MAX_ACTIVE_EVENTS); + if (idx >= C2C_MAX_ACTIVE_EVENTS) + return -EAGAIN; + + set_bit(idx, hw_events->used_ctrs); + + return idx; +} + +static bool +nv_c2c_pmu_validate_event(struct pmu *pmu, + struct nv_c2c_pmu_hw_events *hw_events, + struct perf_event *event) +{ + if (is_software_event(event)) + return true; + + /* Reject groups spanning multiple HW PMUs. */ + if (event->pmu != pmu) + return false; + + return nv_c2c_pmu_get_event_idx(hw_events, event) >= 0; +} + +/* + * Make sure the group of events can be scheduled at once + * on the PMU. + */ +static bool nv_c2c_pmu_validate_group(struct perf_event *event) +{ + struct perf_event *sibling, *leader = event->group_leader; + struct nv_c2c_pmu_hw_events fake_hw_events; + + if (event->group_leader == event) + return true; + + memset(&fake_hw_events, 0, sizeof(fake_hw_events)); + + if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, leader)) + return false; + + for_each_sibling_event(sibling, leader) { + if (!nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, + sibling)) + return false; + } + + return nv_c2c_pmu_validate_event(event->pmu, &fake_hw_events, event); +} + +static int nv_c2c_pmu_event_init(struct perf_event *event) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct hw_perf_event *hwc = &event->hw; + u32 event_type = get_event_type(event); + + if (event->attr.type != event->pmu->type || + event_type >= C2C_NUM_EVENTS) + return -ENOENT; + + /* + * Following other "uncore" PMUs, we do not support sampling mode or + * attach to a task (per-process mode). + */ + if (is_sampling_event(event)) { + dev_dbg(c2c_pmu->pmu.dev, "Can't support sampling events\n"); + return -EOPNOTSUPP; + } + + if (event->cpu < 0 || event->attach_state & PERF_ATTACH_TASK) { + dev_dbg(c2c_pmu->pmu.dev, "Can't support per-task counters\n"); + return -EINVAL; + } + + /* + * Make sure the CPU assignment is on one of the CPUs associated with + * this PMU. + */ + if (!cpumask_test_cpu(event->cpu, &c2c_pmu->associated_cpus)) { + dev_dbg(c2c_pmu->pmu.dev, + "Requested cpu is not associated with the PMU\n"); + return -EINVAL; + } + + /* Enforce the current active CPU to handle the events in this PMU. */ + event->cpu = cpumask_first(&c2c_pmu->active_cpu); + if (event->cpu >= nr_cpu_ids) + return -EINVAL; + + if (!nv_c2c_pmu_validate_group(event)) + return -EINVAL; + + hwc->idx = -1; + hwc->config = event_type; + + return 0; +} + +/* + * Read 64-bit register as a pair of 32-bit registers using hi-lo-hi sequence. + */ +static u64 read_reg64_hilohi(const void __iomem *addr, u32 max_poll_count) +{ + u32 val_lo, val_hi; + u64 val; + + /* Use high-low-high sequence to avoid tearing */ + do { + if (max_poll_count-- == 0) { + pr_err("NV C2C PMU: timeout hi-low-high sequence\n"); + return 0; + } + + val_hi = readl(addr + 4); + val_lo = readl(addr); + } while (val_hi != readl(addr + 4)); + + val = (((u64)val_hi << 32) | val_lo); + + return val; +} + +static void nv_c2c_pmu_check_status(struct nv_c2c_pmu *c2c_pmu, u32 instance) +{ + u32 in_status, out_status; + + in_status = readl(c2c_pmu->base[instance] + C2C_IN_STATUS); + out_status = readl(c2c_pmu->base[instance] + C2C_OUT_STATUS); + + if (in_status || out_status) + dev_warn(c2c_pmu->dev, + "C2C PMU overflow in: 0x%x, out: 0x%x\n", + in_status, out_status); +} + +static u32 nv_c2c_ctr_offset[C2C_NUM_EVENTS] = { + [C2C_EVENT_CYCLES] = C2C_CYCLE_CNTR, + [C2C_EVENT_IN_RD_CUM_OUTS] = C2C_IN_RD_CUM_OUTS_CNTR, + [C2C_EVENT_IN_RD_REQ] = C2C_IN_RD_REQ_CNTR, + [C2C_EVENT_IN_WR_CUM_OUTS] = C2C_IN_WR_CUM_OUTS_CNTR, + [C2C_EVENT_IN_WR_REQ] = C2C_IN_WR_REQ_CNTR, + [C2C_EVENT_OUT_RD_CUM_OUTS] = C2C_OUT_RD_CUM_OUTS_CNTR, + [C2C_EVENT_OUT_RD_REQ] = C2C_OUT_RD_REQ_CNTR, + [C2C_EVENT_OUT_WR_CUM_OUTS] = C2C_OUT_WR_CUM_OUTS_CNTR, + [C2C_EVENT_OUT_WR_REQ] = C2C_OUT_WR_REQ_CNTR, +}; + +static u64 nv_c2c_pmu_read_counter(struct perf_event *event) +{ + u32 ctr_id, ctr_offset, filter_mask, filter_idx, inst_idx; + unsigned long *inst_mask; + DECLARE_BITMAP(filter_bitmap, C2C_NR_PEER_MAX); + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + u64 val = 0; + + filter_mask = get_filter_mask(event); + bitmap_from_arr32(filter_bitmap, &filter_mask, c2c_pmu->nr_peer); + + ctr_id = event->hw.config; + ctr_offset = nv_c2c_ctr_offset[ctr_id]; + + for_each_set_bit(filter_idx, filter_bitmap, c2c_pmu->nr_peer) { + inst_mask = c2c_pmu->peer_insts[filter_idx]; + for_each_set_bit(inst_idx, inst_mask, c2c_pmu->data->nr_inst) { + nv_c2c_pmu_check_status(c2c_pmu, inst_idx); + + /* + * Each instance share same clock and the driver always + * enables all instances. So we can use the counts from + * one instance for cycle counter. + */ + if (ctr_id == C2C_EVENT_CYCLES) + return read_reg64_hilohi( + c2c_pmu->base[inst_idx] + ctr_offset, + HILOHI_MAX_POLL); + + /* + * For other events, sum up the counts from all instances. + */ + val += read_reg64_hilohi( + c2c_pmu->base[inst_idx] + ctr_offset, + HILOHI_MAX_POLL); + } + } + + return val; +} + +static void nv_c2c_pmu_event_update(struct perf_event *event) +{ + struct hw_perf_event *hwc = &event->hw; + u64 prev, now; + + do { + prev = local64_read(&hwc->prev_count); + now = nv_c2c_pmu_read_counter(event); + } while (local64_cmpxchg(&hwc->prev_count, prev, now) != prev); + + local64_add(now - prev, &event->count); +} + +static void nv_c2c_pmu_start(struct perf_event *event, int pmu_flags) +{ + event->hw.state = 0; +} + +static void nv_c2c_pmu_stop(struct perf_event *event, int pmu_flags) +{ + event->hw.state |= PERF_HES_STOPPED; +} + +static int nv_c2c_pmu_add(struct perf_event *event, int flags) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx; + + if (WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), + &c2c_pmu->associated_cpus))) + return -ENOENT; + + idx = nv_c2c_pmu_get_event_idx(hw_events, event); + if (idx < 0) + return idx; + + hw_events->events[idx] = event; + hwc->idx = idx; + hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE; + + if (flags & PERF_EF_START) + nv_c2c_pmu_start(event, PERF_EF_RELOAD); + + /* Propagate changes to the userspace mapping. */ + perf_event_update_userpage(event); + + return 0; +} + +static void nv_c2c_pmu_del(struct perf_event *event, int flags) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(event->pmu); + struct nv_c2c_pmu_hw_events *hw_events = &c2c_pmu->hw_events; + struct hw_perf_event *hwc = &event->hw; + int idx = hwc->idx; + + nv_c2c_pmu_stop(event, PERF_EF_UPDATE); + + hw_events->events[idx] = NULL; + + clear_bit(idx, hw_events->used_ctrs); + + perf_event_update_userpage(event); +} + +static void nv_c2c_pmu_read(struct perf_event *event) +{ + nv_c2c_pmu_event_update(event); +} + +static void nv_c2c_pmu_enable(struct pmu *pmu) +{ + void __iomem *bcast; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + + /* Check if any filter is enabled. */ + if (bitmap_empty(c2c_pmu->hw_events.used_ctrs, C2C_MAX_ACTIVE_EVENTS)) + return; + + /* Enable all the counters. */ + bcast = c2c_pmu->base_broadcast; + writel(0x1UL, bcast + C2C_CTRL); +} + +static void nv_c2c_pmu_disable(struct pmu *pmu) +{ + unsigned int idx; + void __iomem *bcast; + struct perf_event *event; + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + + /* Disable all the counters. */ + bcast = c2c_pmu->base_broadcast; + writel(0x0UL, bcast + C2C_CTRL); + + /* + * The counters will start from 0 again on restart. + * Update the events immediately to avoid losing the counts. + */ + for_each_set_bit(idx, c2c_pmu->hw_events.used_ctrs, + C2C_MAX_ACTIVE_EVENTS) { + event = c2c_pmu->hw_events.events[idx]; + + if (!event) + continue; + + nv_c2c_pmu_event_update(event); + + local64_set(&event->hw.prev_count, 0ULL); + } +} + +/* PMU identifier attribute. */ + +static ssize_t nv_c2c_pmu_identifier_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); + + return sysfs_emit(page, "%s\n", c2c_pmu->identifier); +} + +static struct device_attribute nv_c2c_pmu_identifier_attr = + __ATTR(identifier, 0444, nv_c2c_pmu_identifier_show, NULL); + +static struct attribute *nv_c2c_pmu_identifier_attrs[] = { + &nv_c2c_pmu_identifier_attr.attr, + NULL, +}; + +static struct attribute_group nv_c2c_pmu_identifier_attr_group = { + .attrs = nv_c2c_pmu_identifier_attrs, +}; + +/* Peer attribute. */ + +static ssize_t nv_c2c_pmu_peer_show(struct device *dev, + struct device_attribute *attr, + char *page) +{ + const char *peer_type[C2C_PEER_TYPE_COUNT] = { + [C2C_PEER_TYPE_CPU] = "cpu", + [C2C_PEER_TYPE_GPU] = "gpu", + [C2C_PEER_TYPE_CXLMEM] = "cxlmem", + }; + + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(dev_get_drvdata(dev)); + return sysfs_emit(page, "nr_%s=%u\n", peer_type[c2c_pmu->peer_type], + c2c_pmu->nr_peer); +} + +static struct device_attribute nv_c2c_pmu_peer_attr = + __ATTR(peer, 0444, nv_c2c_pmu_peer_show, NULL); + +static struct attribute *nv_c2c_pmu_peer_attrs[] = { + &nv_c2c_pmu_peer_attr.attr, + NULL, +}; + +static struct attribute_group nv_c2c_pmu_peer_attr_group = { + .attrs = nv_c2c_pmu_peer_attrs, +}; + +/* Format attributes. */ + +#define NV_C2C_PMU_EXT_ATTR(_name, _func, _config) \ + (&((struct dev_ext_attribute[]){ \ + { \ + .attr = __ATTR(_name, 0444, _func, NULL), \ + .var = (void *)_config \ + } \ + })[0].attr.attr) + +#define NV_C2C_PMU_FORMAT_ATTR(_name, _config) \ + NV_C2C_PMU_EXT_ATTR(_name, device_show_string, _config) + +#define NV_C2C_PMU_FORMAT_EVENT_ATTR \ + NV_C2C_PMU_FORMAT_ATTR(event, "config:0-3") + +static struct attribute *nv_c2c_pmu_gpu_formats[] = { + NV_C2C_PMU_FORMAT_EVENT_ATTR, + NV_C2C_PMU_FORMAT_ATTR(gpu_mask, "config1:0-1"), + NULL, +}; + +static const struct attribute_group nv_c2c_pmu_gpu_format_group = { + .name = "format", + .attrs = nv_c2c_pmu_gpu_formats, +}; + +static struct attribute *nv_c2c_pmu_formats[] = { + NV_C2C_PMU_FORMAT_EVENT_ATTR, + NULL, +}; + +static const struct attribute_group nv_c2c_pmu_format_group = { + .name = "format", + .attrs = nv_c2c_pmu_formats, +}; + +/* Event attributes. */ + +static ssize_t nv_c2c_pmu_sysfs_event_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct perf_pmu_events_attr *pmu_attr; + + pmu_attr = container_of(attr, typeof(*pmu_attr), attr); + return sysfs_emit(buf, "event=0x%llx\n", pmu_attr->id); +} + +#define NV_C2C_PMU_EVENT_ATTR(_name, _config) \ + PMU_EVENT_ATTR_ID(_name, nv_c2c_pmu_sysfs_event_show, _config) + +static struct attribute *nv_c2c_pmu_gpu_events[] = { + NV_C2C_PMU_EVENT_ATTR(cycles, C2C_EVENT_CYCLES), + NV_C2C_PMU_EVENT_ATTR(in_rd_cum_outs, C2C_EVENT_IN_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_rd_req, C2C_EVENT_IN_RD_REQ), + NV_C2C_PMU_EVENT_ATTR(in_wr_cum_outs, C2C_EVENT_IN_WR_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_wr_req, C2C_EVENT_IN_WR_REQ), + NV_C2C_PMU_EVENT_ATTR(out_rd_cum_outs, C2C_EVENT_OUT_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(out_rd_req, C2C_EVENT_OUT_RD_REQ), + NV_C2C_PMU_EVENT_ATTR(out_wr_cum_outs, C2C_EVENT_OUT_WR_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(out_wr_req, C2C_EVENT_OUT_WR_REQ), + NULL +}; + +static const struct attribute_group nv_c2c_pmu_gpu_events_group = { + .name = "events", + .attrs = nv_c2c_pmu_gpu_events, +}; + +static struct attribute *nv_c2c_pmu_cpu_events[] = { + NV_C2C_PMU_EVENT_ATTR(cycles, C2C_EVENT_CYCLES), + NV_C2C_PMU_EVENT_ATTR(in_rd_cum_outs, C2C_EVENT_IN_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_rd_req, C2C_EVENT_IN_RD_REQ), + NV_C2C_PMU_EVENT_ATTR(out_rd_cum_outs, C2C_EVENT_OUT_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(out_rd_req, C2C_EVENT_OUT_RD_REQ), + NULL +}; + +static const struct attribute_group nv_c2c_pmu_cpu_events_group = { + .name = "events", + .attrs = nv_c2c_pmu_cpu_events, +}; + +static struct attribute *nv_c2c_pmu_cxlmem_events[] = { + NV_C2C_PMU_EVENT_ATTR(cycles, C2C_EVENT_CYCLES), + NV_C2C_PMU_EVENT_ATTR(in_rd_cum_outs, C2C_EVENT_IN_RD_CUM_OUTS), + NV_C2C_PMU_EVENT_ATTR(in_rd_req, C2C_EVENT_IN_RD_REQ), + NULL +}; + +static const struct attribute_group nv_c2c_pmu_cxlmem_events_group = { + .name = "events", + .attrs = nv_c2c_pmu_cxlmem_events, +}; + +/* Cpumask attributes. */ + +static ssize_t nv_c2c_pmu_cpumask_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pmu *pmu = dev_get_drvdata(dev); + struct nv_c2c_pmu *c2c_pmu = to_c2c_pmu(pmu); + struct dev_ext_attribute *eattr = + container_of(attr, struct dev_ext_attribute, attr); + unsigned long mask_id = (unsigned long)eattr->var; + const cpumask_t *cpumask; + + switch (mask_id) { + case C2C_ACTIVE_CPU_MASK: + cpumask = &c2c_pmu->active_cpu; + break; + case C2C_ASSOCIATED_CPU_MASK: + cpumask = &c2c_pmu->associated_cpus; + break; + default: + return 0; + } + return cpumap_print_to_pagebuf(true, buf, cpumask); +} + +#define NV_C2C_PMU_CPUMASK_ATTR(_name, _config) \ + NV_C2C_PMU_EXT_ATTR(_name, nv_c2c_pmu_cpumask_show, \ + (unsigned long)_config) + +static struct attribute *nv_c2c_pmu_cpumask_attrs[] = { + NV_C2C_PMU_CPUMASK_ATTR(cpumask, C2C_ACTIVE_CPU_MASK), + NV_C2C_PMU_CPUMASK_ATTR(associated_cpus, C2C_ASSOCIATED_CPU_MASK), + NULL, +}; + +static const struct attribute_group nv_c2c_pmu_cpumask_attr_group = { + .attrs = nv_c2c_pmu_cpumask_attrs, +}; + +/* Attribute groups for C2C PMU connecting SoC and GPU */ +static const struct attribute_group *nv_c2c_pmu_gpu_attr_groups[] = { + &nv_c2c_pmu_gpu_format_group, + &nv_c2c_pmu_gpu_events_group, + &nv_c2c_pmu_cpumask_attr_group, + &nv_c2c_pmu_identifier_attr_group, + &nv_c2c_pmu_peer_attr_group, + NULL +}; + +/* Attribute groups for C2C PMU connecting multiple SoCs */ +static const struct attribute_group *nv_c2c_pmu_cpu_attr_groups[] = { + &nv_c2c_pmu_format_group, + &nv_c2c_pmu_cpu_events_group, + &nv_c2c_pmu_cpumask_attr_group, + &nv_c2c_pmu_identifier_attr_group, + &nv_c2c_pmu_peer_attr_group, + NULL +}; + +/* Attribute groups for C2C PMU connecting SoC and CXLMEM */ +static const struct attribute_group *nv_c2c_pmu_cxlmem_attr_groups[] = { + &nv_c2c_pmu_format_group, + &nv_c2c_pmu_cxlmem_events_group, + &nv_c2c_pmu_cpumask_attr_group, + &nv_c2c_pmu_identifier_attr_group, + &nv_c2c_pmu_peer_attr_group, + NULL +}; + +static int nv_c2c_pmu_online_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct nv_c2c_pmu *c2c_pmu = + hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); + + if (!cpumask_test_cpu(cpu, &c2c_pmu->associated_cpus)) + return 0; + + /* If the PMU is already managed, there is nothing to do */ + if (!cpumask_empty(&c2c_pmu->active_cpu)) + return 0; + + /* Use this CPU for event counting */ + cpumask_set_cpu(cpu, &c2c_pmu->active_cpu); + + return 0; +} + +static int nv_c2c_pmu_cpu_teardown(unsigned int cpu, struct hlist_node *node) +{ + unsigned int dst; + + struct nv_c2c_pmu *c2c_pmu = + hlist_entry_safe(node, struct nv_c2c_pmu, cpuhp_node); + + /* Nothing to do if this CPU doesn't own the PMU */ + if (!cpumask_test_and_clear_cpu(cpu, &c2c_pmu->active_cpu)) + return 0; + + /* Choose a new CPU to migrate ownership of the PMU to */ + dst = cpumask_any_and_but(&c2c_pmu->associated_cpus, + cpu_online_mask, cpu); + if (dst >= nr_cpu_ids) + return 0; + + /* Use this CPU for event counting */ + perf_pmu_migrate_context(&c2c_pmu->pmu, cpu, dst); + cpumask_set_cpu(dst, &c2c_pmu->active_cpu); + + return 0; +} + +static int nv_c2c_pmu_get_cpus(struct nv_c2c_pmu *c2c_pmu) +{ + int socket = c2c_pmu->socket, cpu; + + for_each_possible_cpu(cpu) { + if (cpu_to_node(cpu) == socket) + cpumask_set_cpu(cpu, &c2c_pmu->associated_cpus); + } + + if (cpumask_empty(&c2c_pmu->associated_cpus)) { + dev_dbg(c2c_pmu->dev, + "No cpu associated with C2C PMU socket-%u\n", socket); + return -ENODEV; + } + + return 0; +} + +static int nv_c2c_pmu_init_socket(struct nv_c2c_pmu *c2c_pmu) +{ + const char *uid_str; + int ret, socket; + + uid_str = acpi_device_uid(c2c_pmu->acpi_dev); + if (!uid_str) { + dev_err(c2c_pmu->dev, "No ACPI device UID\n"); + return -ENODEV; + } + + ret = kstrtou32(uid_str, 0, &socket); + if (ret) { + dev_err(c2c_pmu->dev, "Failed to parse ACPI device UID\n"); + return ret; + } + + c2c_pmu->socket = socket; + return 0; +} + +static int nv_c2c_pmu_init_id(struct nv_c2c_pmu *c2c_pmu) +{ + char *name; + + name = devm_kasprintf(c2c_pmu->dev, GFP_KERNEL, c2c_pmu->data->name_fmt, + c2c_pmu->socket); + if (!name) + return -ENOMEM; + + c2c_pmu->name = name; + + c2c_pmu->identifier = acpi_device_hid(c2c_pmu->acpi_dev); + + return 0; +} + +static int nv_c2c_pmu_init_filter(struct nv_c2c_pmu *c2c_pmu) +{ + u32 cpu_en = 0; + struct device *dev = c2c_pmu->dev; + const struct nv_c2c_pmu_data *data = c2c_pmu->data; + + if (data->c2c_type == C2C_TYPE_NVDLINK) { + c2c_pmu->peer_type = C2C_PEER_TYPE_CXLMEM; + + c2c_pmu->peer_insts[0][0] = (1UL << data->nr_inst) - 1; + + c2c_pmu->nr_peer = C2C_NR_PEER_CXLMEM; + c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; + + c2c_pmu->attr_groups = nv_c2c_pmu_cxlmem_attr_groups; + + return 0; + } + + if (device_property_read_u32(dev, "cpu_en_mask", &cpu_en)) + dev_dbg(dev, "no cpu_en_mask property\n"); + + if (cpu_en) { + c2c_pmu->peer_type = C2C_PEER_TYPE_CPU; + + /* Fill peer_insts bitmap with instances connected to peer CPU. */ + bitmap_from_arr32(c2c_pmu->peer_insts[0], &cpu_en, data->nr_inst); + + c2c_pmu->nr_peer = 1; + c2c_pmu->attr_groups = nv_c2c_pmu_cpu_attr_groups; + } else { + u32 i; + const char *props[C2C_NR_PEER_MAX] = { + "gpu0_en_mask", "gpu1_en_mask" + }; + + for (i = 0; i < C2C_NR_PEER_MAX; i++) { + u32 gpu_en = 0; + + if (device_property_read_u32(dev, props[i], &gpu_en)) + dev_dbg(dev, "no %s property\n", props[i]); + + if (gpu_en) { + /* Fill peer_insts bitmap with instances connected to peer GPU. */ + bitmap_from_arr32(c2c_pmu->peer_insts[i], &gpu_en, + data->nr_inst); + + c2c_pmu->nr_peer++; + } + } + + if (c2c_pmu->nr_peer == 0) { + dev_err(dev, "No GPU is enabled\n"); + return -EINVAL; + } + + c2c_pmu->peer_type = C2C_PEER_TYPE_GPU; + c2c_pmu->attr_groups = nv_c2c_pmu_gpu_attr_groups; + } + + c2c_pmu->filter_default = (1 << c2c_pmu->nr_peer) - 1; + + return 0; +} + +static void *nv_c2c_pmu_init_pmu(struct platform_device *pdev) +{ + int ret; + struct nv_c2c_pmu *c2c_pmu; + struct acpi_device *acpi_dev; + struct device *dev = &pdev->dev; + + acpi_dev = ACPI_COMPANION(dev); + if (!acpi_dev) + return ERR_PTR(-ENODEV); + + c2c_pmu = devm_kzalloc(dev, sizeof(*c2c_pmu), GFP_KERNEL); + if (!c2c_pmu) + return ERR_PTR(-ENOMEM); + + c2c_pmu->dev = dev; + c2c_pmu->acpi_dev = acpi_dev; + c2c_pmu->data = (const struct nv_c2c_pmu_data *)device_get_match_data(dev); + if (!c2c_pmu->data) + return ERR_PTR(-EINVAL); + + platform_set_drvdata(pdev, c2c_pmu); + + ret = nv_c2c_pmu_init_socket(c2c_pmu); + if (ret) + return ERR_PTR(ret); + + ret = nv_c2c_pmu_init_id(c2c_pmu); + if (ret) + return ERR_PTR(ret); + + ret = nv_c2c_pmu_init_filter(c2c_pmu); + if (ret) + return ERR_PTR(ret); + + return c2c_pmu; +} + +static int nv_c2c_pmu_init_mmio(struct nv_c2c_pmu *c2c_pmu) +{ + int i; + struct device *dev = c2c_pmu->dev; + struct platform_device *pdev = to_platform_device(dev); + const struct nv_c2c_pmu_data *data = c2c_pmu->data; + + /* Map the address of all the instances. */ + for (i = 0; i < data->nr_inst; i++) { + c2c_pmu->base[i] = devm_platform_ioremap_resource(pdev, i); + if (IS_ERR(c2c_pmu->base[i])) { + dev_err(dev, "Failed map address for instance %d\n", i); + return PTR_ERR(c2c_pmu->base[i]); + } + } + + /* Map broadcast address. */ + c2c_pmu->base_broadcast = devm_platform_ioremap_resource(pdev, + data->nr_inst); + if (IS_ERR(c2c_pmu->base_broadcast)) { + dev_err(dev, "Failed map broadcast address\n"); + return PTR_ERR(c2c_pmu->base_broadcast); + } + + return 0; +} + +static int nv_c2c_pmu_register_pmu(struct nv_c2c_pmu *c2c_pmu) +{ + int ret; + + ret = cpuhp_state_add_instance(nv_c2c_pmu_cpuhp_state, + &c2c_pmu->cpuhp_node); + if (ret) { + dev_err(c2c_pmu->dev, "Error %d registering hotplug\n", ret); + return ret; + } + + c2c_pmu->pmu = (struct pmu) { + .parent = c2c_pmu->dev, + .task_ctx_nr = perf_invalid_context, + .pmu_enable = nv_c2c_pmu_enable, + .pmu_disable = nv_c2c_pmu_disable, + .event_init = nv_c2c_pmu_event_init, + .add = nv_c2c_pmu_add, + .del = nv_c2c_pmu_del, + .start = nv_c2c_pmu_start, + .stop = nv_c2c_pmu_stop, + .read = nv_c2c_pmu_read, + .attr_groups = c2c_pmu->attr_groups, + .capabilities = PERF_PMU_CAP_NO_EXCLUDE | + PERF_PMU_CAP_NO_INTERRUPT, + }; + + ret = perf_pmu_register(&c2c_pmu->pmu, c2c_pmu->name, -1); + if (ret) { + dev_err(c2c_pmu->dev, "Failed to register C2C PMU: %d\n", ret); + cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, + &c2c_pmu->cpuhp_node); + return ret; + } + + return 0; +} + +static int nv_c2c_pmu_probe(struct platform_device *pdev) +{ + int ret; + struct nv_c2c_pmu *c2c_pmu; + + c2c_pmu = nv_c2c_pmu_init_pmu(pdev); + if (IS_ERR(c2c_pmu)) + return PTR_ERR(c2c_pmu); + + ret = nv_c2c_pmu_init_mmio(c2c_pmu); + if (ret) + return ret; + + ret = nv_c2c_pmu_get_cpus(c2c_pmu); + if (ret) + return ret; + + ret = nv_c2c_pmu_register_pmu(c2c_pmu); + if (ret) + return ret; + + dev_dbg(c2c_pmu->dev, "Registered %s PMU\n", c2c_pmu->name); + + return 0; +} + +static void nv_c2c_pmu_device_remove(struct platform_device *pdev) +{ + struct nv_c2c_pmu *c2c_pmu = platform_get_drvdata(pdev); + + perf_pmu_unregister(&c2c_pmu->pmu); + cpuhp_state_remove_instance(nv_c2c_pmu_cpuhp_state, &c2c_pmu->cpuhp_node); +} + +static const struct acpi_device_id nv_c2c_pmu_acpi_match[] = { + { "NVDA2023", (kernel_ulong_t)&nv_c2c_pmu_data[C2C_TYPE_NVLINK] }, + { "NVDA2022", (kernel_ulong_t)&nv_c2c_pmu_data[C2C_TYPE_NVCLINK] }, + { "NVDA2020", (kernel_ulong_t)&nv_c2c_pmu_data[C2C_TYPE_NVDLINK] }, + { } +}; +MODULE_DEVICE_TABLE(acpi, nv_c2c_pmu_acpi_match); + +static struct platform_driver nv_c2c_pmu_driver = { + .driver = { + .name = "nvidia-t410-c2c-pmu", + .acpi_match_table = nv_c2c_pmu_acpi_match, + .suppress_bind_attrs = true, + }, + .probe = nv_c2c_pmu_probe, + .remove = nv_c2c_pmu_device_remove, +}; + +static int __init nv_c2c_pmu_init(void) +{ + int ret; + + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, + "perf/nvidia/c2c:online", + nv_c2c_pmu_online_cpu, + nv_c2c_pmu_cpu_teardown); + if (ret < 0) + return ret; + + nv_c2c_pmu_cpuhp_state = ret; + return platform_driver_register(&nv_c2c_pmu_driver); +} + +static void __exit nv_c2c_pmu_exit(void) +{ + platform_driver_unregister(&nv_c2c_pmu_driver); + cpuhp_remove_multi_state(nv_c2c_pmu_cpuhp_state); +} + +module_init(nv_c2c_pmu_init); +module_exit(nv_c2c_pmu_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("NVIDIA Tegra410 C2C PMU driver"); +MODULE_AUTHOR("Besar Wicaksono "); From feafe6f224d5ac4366bbaf37034939cd16de3292 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Thu, 12 Feb 2026 23:34:07 +0000 Subject: [PATCH 203/464] perf vendor events arm64: Add Tegra410 Olympus PMU events BugLink: https://bugs.launchpad.net/bugs/2149756 Add JSON files for NVIDIA Tegra410 Olympus core PMU events. Also updated the common-and-microarch.json. Signed-off-by: Besar Wicaksono Reviewed-by: James Clark Signed-off-by: Namhyung Kim (cherry picked from commit 86ff690f45cc034ab32246630b3c7d7a46d1ae6b) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- .../arch/arm64/common-and-microarch.json | 85 +++ tools/perf/pmu-events/arch/arm64/mapfile.csv | 1 + .../arch/arm64/nvidia/t410/branch.json | 45 ++ .../arch/arm64/nvidia/t410/brbe.json | 6 + .../arch/arm64/nvidia/t410/bus.json | 48 ++ .../arch/arm64/nvidia/t410/exception.json | 62 ++ .../arch/arm64/nvidia/t410/fp_operation.json | 78 ++ .../arch/arm64/nvidia/t410/general.json | 15 + .../arch/arm64/nvidia/t410/l1d_cache.json | 122 +++ .../arch/arm64/nvidia/t410/l1i_cache.json | 114 +++ .../arch/arm64/nvidia/t410/l2d_cache.json | 134 ++++ .../arch/arm64/nvidia/t410/ll_cache.json | 107 +++ .../arch/arm64/nvidia/t410/memory.json | 46 ++ .../arch/arm64/nvidia/t410/metrics.json | 722 ++++++++++++++++++ .../arch/arm64/nvidia/t410/misc.json | 642 ++++++++++++++++ .../arch/arm64/nvidia/t410/retired.json | 94 +++ .../arch/arm64/nvidia/t410/spe.json | 42 + .../arm64/nvidia/t410/spec_operation.json | 230 ++++++ .../arch/arm64/nvidia/t410/stall.json | 145 ++++ .../arch/arm64/nvidia/t410/tlb.json | 158 ++++ 20 files changed, 2896 insertions(+) create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json create mode 100644 tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json diff --git a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json index 468cb085d8796..144325d87be44 100644 --- a/tools/perf/pmu-events/arch/arm64/common-and-microarch.json +++ b/tools/perf/pmu-events/arch/arm64/common-and-microarch.json @@ -1512,11 +1512,26 @@ "EventName": "L2D_CACHE_REFILL_PRFM", "BriefDescription": "Level 2 data cache refill, software preload" }, + { + "EventCode": "0x8150", + "EventName": "L3D_CACHE_RW", + "BriefDescription": "Level 3 data cache demand access." + }, + { + "EventCode": "0x8151", + "EventName": "L3D_CACHE_PRFM", + "BriefDescription": "Level 3 data cache software prefetch" + }, { "EventCode": "0x8152", "EventName": "L3D_CACHE_MISS", "BriefDescription": "Level 3 data cache demand access miss" }, + { + "EventCode": "0x8153", + "EventName": "L3D_CACHE_REFILL_PRFM", + "BriefDescription": "Level 3 data cache refill, software prefetch." + }, { "EventCode": "0x8154", "EventName": "L1D_CACHE_HWPRF", @@ -1527,6 +1542,11 @@ "EventName": "L2D_CACHE_HWPRF", "BriefDescription": "Level 2 data cache hardware prefetch." }, + { + "EventCode": "0x8156", + "EventName": "L3D_CACHE_HWPRF", + "BriefDescription": "Level 3 data cache hardware prefetch." + }, { "EventCode": "0x8158", "EventName": "STALL_FRONTEND_MEMBOUND", @@ -1682,6 +1702,11 @@ "EventName": "L2D_CACHE_REFILL_HWPRF", "BriefDescription": "Level 2 data cache refill, hardware prefetch." }, + { + "EventCode": "0x81BE", + "EventName": "L3D_CACHE_REFILL_HWPRF", + "BriefDescription": "Level 3 data cache refill, hardware prefetch." + }, { "EventCode": "0x81C0", "EventName": "L1I_CACHE_HIT_RD", @@ -1712,11 +1737,31 @@ "EventName": "L1I_CACHE_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by software preload" }, + { + "EventCode": "0x81DC", + "EventName": "L1D_CACHE_HIT_RW_FPRFM", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by software prefetch." + }, { "EventCode": "0x81E0", "EventName": "L1I_CACHE_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by hardware prefetcher" }, + { + "EventCode": "0x81EC", + "EventName": "L1D_CACHE_HIT_RW_FHWPRF", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by hardware prefetcher." + }, + { + "EventCode": "0x81F0", + "EventName": "L1I_CACHE_HIT_RD_FPRF", + "BriefDescription": "Level 1 instruction cache demand fetch first hit, fetched by prefetch." + }, + { + "EventCode": "0x81FC", + "EventName": "L1D_CACHE_HIT_RW_FPRF", + "BriefDescription": "Level 1 data cache demand access first hit, fetched by prefetch." + }, { "EventCode": "0x8200", "EventName": "L1I_CACHE_HIT", @@ -1767,11 +1812,26 @@ "EventName": "L1I_LFB_HIT_RD_FPRFM", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by software preload" }, + { + "EventCode": "0x825C", + "EventName": "L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by software prefetch." + }, { "EventCode": "0x8260", "EventName": "L1I_LFB_HIT_RD_FHWPRF", "BriefDescription": "Level 1 instruction cache demand fetch line-fill buffer first hit, recently fetched by hardware prefetcher" }, + { + "EventCode": "0x826C", + "EventName": "L1D_LFB_HIT_RW_FHWPRF", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by hardware prefetcher." + }, + { + "EventCode": "0x827C", + "EventName": "L1D_LFB_HIT_RW_FPRF", + "BriefDescription": "Level 1 data cache demand access line-fill buffer first hit, recently fetched by prefetch." + }, { "EventCode": "0x8280", "EventName": "L1I_CACHE_PRF", @@ -1807,6 +1867,11 @@ "EventName": "LL_CACHE_REFILL", "BriefDescription": "Last level cache refill" }, + { + "EventCode": "0x828E", + "EventName": "L3D_CACHE_REFILL_PRF", + "BriefDescription": "Level 3 data cache refill, prefetch." + }, { "EventCode": "0x8320", "EventName": "L1D_CACHE_REFILL_PERCYC", @@ -1872,6 +1937,16 @@ "EventName": "FP_FP8_MIN_SPEC", "BriefDescription": "Floating-point operation speculatively_executed, smallest type is 8-bit floating-point." }, + { + "EventCode": "0x8480", + "EventName": "FP_SP_FIXED_MIN_OPS_SPEC", + "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." + }, + { + "EventCode": "0x8482", + "EventName": "FP_HP_FIXED_MIN_OPS_SPEC", + "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." + }, { "EventCode": "0x8483", "EventName": "FP_BF16_FIXED_MIN_OPS_SPEC", @@ -1882,6 +1957,16 @@ "EventName": "FP_FP8_FIXED_MIN_OPS_SPEC", "BriefDescription": "Non-scalable element arithmetic operations speculatively executed, smallest type is 8-bit floating-point." }, + { + "EventCode": "0x8488", + "EventName": "FP_SP_SCALE_MIN_OPS_SPEC", + "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is single-precision floating-point." + }, + { + "EventCode": "0x848A", + "EventName": "FP_HP_SCALE_MIN_OPS_SPEC", + "BriefDescription": "Scalable element arithmetic operations speculatively executed, smallest type is half-precision floating-point." + }, { "EventCode": "0x848B", "EventName": "FP_BF16_SCALE_MIN_OPS_SPEC", diff --git a/tools/perf/pmu-events/arch/arm64/mapfile.csv b/tools/perf/pmu-events/arch/arm64/mapfile.csv index bb3fa8a33496a..7f0eaa7020485 100644 --- a/tools/perf/pmu-events/arch/arm64/mapfile.csv +++ b/tools/perf/pmu-events/arch/arm64/mapfile.csv @@ -46,3 +46,4 @@ 0x00000000500f0000,v1,ampere/emag,core 0x00000000c00fac30,v1,ampere/ampereone,core 0x00000000c00fac40,v1,ampere/ampereonex,core +0x000000004e0f0100,v1,nvidia/t410,core diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json new file mode 100644 index 0000000000000..ef4effc00ec3a --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/branch.json @@ -0,0 +1,45 @@ +[ + { + "ArchStdEvent": "BR_MIS_PRED", + "PublicDescription": "This event counts branches which are speculatively executed and mispredicted." + }, + { + "ArchStdEvent": "BR_PRED", + "PublicDescription": "This event counts all speculatively executed branches." + }, + { + "EventCode": "0x017e", + "EventName": "BR_PRED_BTB_CTX_UPDATE", + "PublicDescription": "Branch context table update." + }, + { + "EventCode": "0x0188", + "EventName": "BR_MIS_PRED_DIR_RESOLVED", + "PublicDescription": "Number of branch misprediction due to direction misprediction." + }, + { + "EventCode": "0x0189", + "EventName": "BR_MIS_PRED_DIR_UNCOND_RESOLVED", + "PublicDescription": "Number of branch misprediction due to direction misprediction for unconditional branches." + }, + { + "EventCode": "0x018a", + "EventName": "BR_MIS_PRED_DIR_UNCOND_DIRECT_RESOLVED", + "PublicDescription": "Number of branch misprediction due to direction misprediction for unconditional direct branches." + }, + { + "EventCode": "0x018b", + "EventName": "BR_PRED_MULTI_RESOLVED", + "PublicDescription": "Number of resolved branch which made prediction by polymorphic indirect predictor." + }, + { + "EventCode": "0x018c", + "EventName": "BR_MIS_PRED_MULTI_RESOLVED", + "PublicDescription": "Number of branch misprediction which made prediction by polymorphic indirect predictor." + }, + { + "EventCode": "0x01e4", + "EventName": "BR_RGN_RECLAIM", + "PublicDescription": "This event counts the Indirect predictor entries flushed by region reclamation." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json new file mode 100644 index 0000000000000..9c315b2d70469 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/brbe.json @@ -0,0 +1,6 @@ +[ + { + "ArchStdEvent": "BRB_FILTRATE", + "PublicDescription": "This event counts each valid branch record captured in the branch record buffer. Branch records that are not captured because they are removed by filtering are not counted." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json new file mode 100644 index 0000000000000..5bb8de617c68b --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/bus.json @@ -0,0 +1,48 @@ +[ + { + "ArchStdEvent": "BUS_ACCESS", + "PublicDescription": "This event counts the number of data-beat accesses between the CPU and the external bus. This count includes accesses due to read, write, and snoop. Each beat of data is counted individually." + }, + { + "ArchStdEvent": "BUS_CYCLES", + "PublicDescription": "This event counts bus cycles in the CPU. Bus cycles represent a clock cycle in which a transaction could be sent or received on the interface from the CPU to the external bus. Since that interface is driven at the same clock speed as the CPU, this event increments at the rate of CPU clock. Regardless of the WFE/WFI state of the PE, this event increments on each processor clock." + }, + { + "ArchStdEvent": "BUS_ACCESS_RD", + "PublicDescription": "This event counts memory Read transactions seen on the external bus. Each beat of data is counted individually." + }, + { + "ArchStdEvent": "BUS_ACCESS_WR", + "PublicDescription": "This event counts memory Write transactions seen on the external bus. Each beat of data is counted individually." + }, + { + "EventCode": "0x0154", + "EventName": "BUS_REQUEST_REQ", + "PublicDescription": "Bus request, request." + }, + { + "EventCode": "0x0155", + "EventName": "BUS_REQUEST_RETRY", + "PublicDescription": "Bus request, retry." + }, + { + "EventCode": "0x0198", + "EventName": "L2_CHI_CBUSY0", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 0." + }, + { + "EventCode": "0x0199", + "EventName": "L2_CHI_CBUSY1", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 1." + }, + { + "EventCode": "0x019a", + "EventName": "L2_CHI_CBUSY2", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 2." + }, + { + "EventCode": "0x019b", + "EventName": "L2_CHI_CBUSY3", + "PublicDescription": "Number of RXDAT or RXRSP response received width CBusy of 3." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json new file mode 100644 index 0000000000000..ecd996c3610be --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/exception.json @@ -0,0 +1,62 @@ +[ + { + "ArchStdEvent": "EXC_TAKEN", + "PublicDescription": "This event counts any taken architecturally visible exceptions such as IRQ, FIQ, SError, and other synchronous exceptions. Exceptions are counted whether or not they are taken locally." + }, + { + "ArchStdEvent": "EXC_RETURN", + "PublicDescription": "This event counts any architecturally executed exception return instructions. For example: AArch64: ERET." + }, + { + "ArchStdEvent": "EXC_UNDEF", + "PublicDescription": "This event counts the number of synchronous exceptions which are taken locally that are due to attempting to execute an instruction that is UNDEFINED.\nAttempting to execute instruction bit patterns that have not been allocated.\nAttempting to execute instructions when they are disabled.\nAttempting to execute instructions at an inappropriate Exception level.\nAttempting to execute an instruction when the value of PSTATE.IL is 1." + }, + { + "ArchStdEvent": "EXC_SVC", + "PublicDescription": "This event counts SVC exceptions taken locally." + }, + { + "ArchStdEvent": "EXC_PABORT", + "PublicDescription": "This event counts synchronous exceptions that are taken locally and caused by Instruction Aborts." + }, + { + "ArchStdEvent": "EXC_DABORT", + "PublicDescription": "This event counts exceptions that are taken locally and are caused by data aborts or SErrors. Conditions that could cause those exceptions are attempting to read or write memory where the MMU generates a fault, attempting to read or write memory with a misaligned address, Interrupts from the nSEI inputs and internally generated SErrors." + }, + { + "ArchStdEvent": "EXC_IRQ", + "PublicDescription": "This event counts IRQ exceptions including the virtual IRQs that are taken locally." + }, + { + "ArchStdEvent": "EXC_FIQ", + "PublicDescription": "This event counts FIQ exceptions including the virtual FIQs that are taken locally." + }, + { + "ArchStdEvent": "EXC_SMC", + "PublicDescription": "This event counts SMC exceptions taken to EL3." + }, + { + "ArchStdEvent": "EXC_HVC", + "PublicDescription": "This event counts HVC exceptions taken to EL2." + }, + { + "ArchStdEvent": "EXC_TRAP_PABORT", + "PublicDescription": "This event counts exceptions which are traps not taken locally and are caused by Instruction Aborts. For example, attempting to execute an instruction with a misaligned PC." + }, + { + "ArchStdEvent": "EXC_TRAP_DABORT", + "PublicDescription": "This event counts exceptions which are traps not taken locally and are caused by Data Aborts or SError Interrupts. Conditions that could cause those exceptions are:\n* Attempting to read or write memory where the MMU generates a fault,\n* Attempting to read or write memory with a misaligned address,\n* Interrupts from the SEI input,\n* Internally generated SErrors." + }, + { + "ArchStdEvent": "EXC_TRAP_OTHER", + "PublicDescription": "This event counts the number of synchronous trap exceptions which are not taken locally and are not SVC, SMC, HVC, Data Aborts, Instruction Aborts, or Interrupts." + }, + { + "ArchStdEvent": "EXC_TRAP_IRQ", + "PublicDescription": "This event counts IRQ exceptions including the virtual IRQs that are not taken locally." + }, + { + "ArchStdEvent": "EXC_TRAP_FIQ", + "PublicDescription": "This event counts FIQs which are not taken locally but taken from EL0, EL1, or EL2 to EL3 (which would be the normal behavior for FIQs when not executing in EL3)." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json new file mode 100644 index 0000000000000..3588e130781db --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/fp_operation.json @@ -0,0 +1,78 @@ +[ + { + "ArchStdEvent": "FP_HP_SPEC", + "PublicDescription": "This event counts speculatively executed half precision floating point operations." + }, + { + "ArchStdEvent": "FP_SP_SPEC", + "PublicDescription": "This event counts speculatively executed single precision floating point operations." + }, + { + "ArchStdEvent": "FP_DP_SPEC", + "PublicDescription": "This event counts speculatively executed double precision floating point operations." + }, + { + "ArchStdEvent": "FP_SCALE_OPS_SPEC", + "PublicDescription": "This event counts speculatively executed scalable single precision floating point operations." + }, + { + "ArchStdEvent": "FP_FIXED_OPS_SPEC", + "PublicDescription": "This event counts speculatively executed non-scalable single precision floating point operations." + }, + { + "ArchStdEvent": "FP_HP_SCALE_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the largest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the counter to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_FIXED_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the largest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or which instruction causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_SCALE_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the largest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_FIXED_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the largest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_DP_SCALE_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the largest type was double-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_DP_FIXED_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the largest type was double-precision floating-point, where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_FIXED_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the smallest type was single-precision floating-point, where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_FIXED_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the smallest type was half-precision floating-point, where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_BF16_FIXED_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the smallest type was BFloat16 floating-point. Where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment. This event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_FP8_FIXED_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed non-scalable element arithmetic operation, due to an instruction where the smallest type was 8-bit floating-point, where v is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_SCALE_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_SP_SCALE_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the smallest type was single-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_HP_SCALE_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the smallest type was half-precision floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_BF16_SCALE_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the smallest type was BFloat16 floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + }, + { + "ArchStdEvent": "FP_FP8_SCALE_MIN_OPS_SPEC", + "PublicDescription": "This event increments by v for each speculatively executed scalable element arithmetic operation, due to an instruction where the smallest type was 8-bit floating-point, where v is a value such that (v*(VL/128)) is the number of arithmetic operations carried out by the operation or instruction which causes the event to increment.\nThis event does not count operations that are counted by FP_FIXED_OPS_SPEC or FP_SCALE2_OPS_SPEC." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json new file mode 100644 index 0000000000000..bd9c248387aae --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/general.json @@ -0,0 +1,15 @@ +[ + { + "ArchStdEvent": "CPU_CYCLES", + "PublicDescription": "This event counts CPU clock cycles when the PE is not in WFE/WFI. The clock measured by this event is defined as the physical clock driving the CPU logic." + }, + { + "ArchStdEvent": "CNT_CYCLES", + "PublicDescription": "This event increments at a constant frequency equal to the rate of increment of the System Counter, CNTPCT_EL0.\nThis event does not increment when the PE is in WFE/WFI." + }, + { + "EventCode": "0x01e1", + "EventName": "CPU_SLOT", + "PublicDescription": "Entitled CPU slots.\nThis event counts the number of slots. When in ST mode, this event shall increment by PMMIR_EL1.SLOTS quantities, and when in SMT partitioned resource mode (regardless of in WFI state or otherwise), this event is incremented by PMMIR_EL1.SLOTS/2 quantities." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json new file mode 100644 index 0000000000000..ed6f764eff242 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1d_cache.json @@ -0,0 +1,122 @@ +[ + { + "ArchStdEvent": "L1D_CACHE_REFILL", + "PublicDescription": "This event counts L1 D-cache refills caused by speculatively executed load or store operations, preload instructions, or hardware cache prefetching that missed in the L1 D-cache. This event only counts one event per cache line.\nSince the caches are Write-back only for this processor, there are no Write-through cache accesses." + }, + { + "ArchStdEvent": "L1D_CACHE", + "PublicDescription": "This event counts L1 D-cache accesses from any load/store operations, software preload, or hardware prefetch operations. Atomic operations that resolve in the CPU's caches (near atomic operations) count as both a write access and read access. Each access to a cache line is counted including the multiple accesses caused by single instructions such as LDM or STM. Each access to other L1 data or unified memory structures, for example refill buffers, write buffers, and write-back buffers, are also counted.\nThis event counts the sum of the following events:\nL1D_CACHE_RD,\nL1D_CACHE_WR,\nL1D_CACHE_PRFM, and\nL1D_CACHE_HWPRF." + }, + { + "ArchStdEvent": "L1D_CACHE_WB", + "PublicDescription": "This event counts write-backs of dirty data from the L1 D-cache to the L2 cache. This occurs when either a dirty cache line is evicted from L1 D-cache and allocated in the L2 cache or dirty data is written to the L2 and possibly to the next level of cache. This event counts both victim cache line evictions and cache write-backs from snoops or cache maintenance operations. The following cache operations are not counted:\n* Invalidations which do not result in data being transferred out of the L1 (such as evictions of clean data),\n* Full line writes which write to L2 without writing L1, such as write streaming mode.\nThis event is the sum of the following events:\nL1D_CACHE_WB_CLEAN and\nL1D_CACHE_WB_VICTIM." + }, + { + "ArchStdEvent": "L1D_CACHE_LMISS_RD", + "PublicDescription": "This event counts cache line refills into the L1 D-cache from any memory Read operations, that incurred additional latency.\nCounts same as L1D_CACHE_REFILL_RD on this CPU." + }, + { + "ArchStdEvent": "L1D_CACHE_RD", + "PublicDescription": "This event counts L1 D-cache accesses from any Load operation. Atomic Load operations that resolve in the CPU's caches count as both a write access and read access." + }, + { + "ArchStdEvent": "L1D_CACHE_WR", + "PublicDescription": "This event counts L1 D-cache accesses generated by Store operations. This event also counts accesses caused by a DC ZVA (D-cache zero, specified by virtual address) instruction. Near atomic operations that resolve in the CPU's caches count as a write access and read access.\nThis event is a subset of the L1D_CACHE event, except this event only counts memory Write operations." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_RD", + "PublicDescription": "This event counts L1 D-cache refills caused by speculatively executed Load instructions where the memory Read operation misses in the L1 D-cache. This event only counts one event per cache line.\nThis event is a subset of the L1D_CACHE_REFILL event, but only counts memory Read operations. This event does not count reads caused by cache maintenance operations or preload instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_WR", + "PublicDescription": "This event counts L1 D-cache refills caused by speculatively executed Store instructions where the memory Write operation misses in the L1 D-cache. This event only counts one event per cache line.\nThis event is a subset of the L1D_CACHE_REFILL event, but only counts memory Write operations." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_INNER", + "PublicDescription": "This event counts L1 D-cache refills (L1D_CACHE_REFILL) where the cache line data came from caches inside the immediate Cluster of the Core (L2 cache)." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_OUTER", + "PublicDescription": "This event counts L1 D-cache refills (L1D_CACHE_REFILL) for which the cache line data came from outside the immediate Cluster of the Core, like an SLC in the system interconnect or DRAM or remote socket." + }, + { + "ArchStdEvent": "L1D_CACHE_WB_VICTIM", + "PublicDescription": "This event counts dirty cache line evictions from the L1 D-cache caused by a new cache line allocation. This event does not count evictions caused by cache maintenance operations.\nThis event is a subset of the L1D_CACHE_WB event, but only counts write-backs that are a result of the line being allocated for an access made by the CPU." + }, + { + "ArchStdEvent": "L1D_CACHE_WB_CLEAN", + "PublicDescription": "This event counts write-backs from the L1 D-cache that are a result of a coherency operation made by another CPU. Event counts include cache maintenance operations.\nThis event is a subset of the L1D_CACHE_WB event." + }, + { + "ArchStdEvent": "L1D_CACHE_INVAL", + "PublicDescription": "This event counts each explicit invalidation of a cache line in the L1 D-cache caused by:\n* Cache Maintenance Operations (CMO) that operate by a virtual address.\n* Broadcast cache coherency operations from another CPU in the system.\nThis event does not count for the following conditions:\n* A cache refill invalidates a cache line.\n* A CMO which is executed on that CPU and invalidates a cache line specified by Set/Way.\nNote that CMOs that operate by Set/Way cannot be broadcast from one CPU to another." + }, + { + "ArchStdEvent": "L1D_CACHE_RW", + "PublicDescription": "This event counts L1 data demand cache accesses from any Load or Store operation. Near atomic operations that resolve in the CPU's caches count as both a write access and read access.\nThis event is implemented as L1D_CACHE_RD + L1D_CACHE_WR" + }, + { + "ArchStdEvent": "L1D_CACHE_PRFM", + "PublicDescription": "This event counts L1 D-cache accesses from software preload or prefetch instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_MISS", + "PublicDescription": "This event counts each demand access counted by L1D_CACHE_RW that misses in the L1 Data or unified cache, causing an access to outside of the L1 caches of this PE." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_PRFM", + "PublicDescription": "This event counts L1 D-cache refills where the cache line access was generated by software preload or prefetch instructions." + }, + { + "ArchStdEvent": "L1D_CACHE_HWPRF", + "PublicDescription": "This event counts L1 D-cache accesses from any Load/Store operations generated by the hardware prefetcher." + }, + { + "ArchStdEvent": "L1D_CACHE_REFILL_HWPRF", + "PublicDescription": "This event counts each hardware prefetch access counted by L1D_CACHE_HWPRF that causes a refill of the L1 D-cache from outside of the L1 D-cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRFM", + "PublicDescription": "This event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched in response to a prefetch instruction. That is, the L1D_CACHE_REFILL_PRFM event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FHWPRF", + "PublicDescription": "This event counts each demand access first hit counted by L1D_CACHE_HIT_RW_FPRF where the cache line was fetched by a hardware prefetcher. That is, the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_CACHE_HIT_RW_FPRF", + "PublicDescription": "This event counts each demand access first hit counted by L1D_CACHE_HIT_RW where the cache line was fetched in response to a prefetch instruction or by a hardware prefetcher. That is, the L1D_CACHE_REFILL_PRF event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FPRFM", + "PublicDescription": "This event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF where the cache line was fetched in response to a prefetch instruction. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_PRFM event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FHWPRF", + "PublicDescription": "This event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW_FPRF, where the cache line was fetched by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_HWPRF Event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1D_LFB_HIT_RW_FPRF", + "PublicDescription": "This event counts each demand access line-fill buffer first hit counted by L1D_LFB_HIT_RW where the cache line was fetched in response to a prefetch instruction or by a hardware prefetcher. That is, the access hits a cache line that is in the process of being loaded into the L1 D-cache, and so does not generate a new refill, but has to wait for the previous refill to complete, and the L1D_CACHE_REFILL_PRF event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x01f5", + "EventName": "L1D_CACHE_REFILL_RW", + "PublicDescription": "L1 D-cache refill, demand Read and Write. This event counts demand Read and Write accesses that causes a refill of the L1 D-cache of this PE, from outside of this cache." + }, + { + "EventCode": "0x0204", + "EventName": "L1D_CACHE_REFILL_OUTER_LLC", + "PublicDescription": "This event counts L1D_CACHE_REFILL from L3 D-cache." + }, + { + "EventCode": "0x0205", + "EventName": "L1D_CACHE_REFILL_OUTER_DRAM", + "PublicDescription": "This event counts L1D_CACHE_REFILL from local memory." + }, + { + "EventCode": "0x0206", + "EventName": "L1D_CACHE_REFILL_OUTER_REMOTE", + "PublicDescription": "This event counts L1D_CACHE_REFILL from a remote memory." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json new file mode 100644 index 0000000000000..952454004d986 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l1i_cache.json @@ -0,0 +1,114 @@ +[ + { + "ArchStdEvent": "L1I_CACHE_REFILL", + "PublicDescription": "This event counts cache line refills in the L1 I-cache caused by a missed instruction fetch (demand, hardware prefetch, and software preload accesses). Instruction fetches may include accessing multiple instructions, but the single cache line allocation is counted once." + }, + { + "ArchStdEvent": "L1I_CACHE", + "PublicDescription": "This event counts instruction fetches (demand, hardware prefetch, and software preload accesses) which access the L1 Instruction Cache. Instruction Cache accesses caused by cache maintenance operations are not counted." + }, + { + "ArchStdEvent": "L1I_CACHE_LMISS", + "PublicDescription": "This event counts cache line refills into the L1 I-cache, that incurred additional latency.\nCounts the same as L1I_CACHE_REFILL in this CPU." + }, + { + "ArchStdEvent": "L1I_CACHE_RD", + "PublicDescription": "This event counts demand instruction fetches which access the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_PRFM", + "PublicDescription": "This event counts instruction fetches generated by software preload or prefetch instructions which access the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HWPRF", + "PublicDescription": "This event counts instruction fetches which access the L1 I-cache generated by the hardware prefetcher." + }, + { + "ArchStdEvent": "L1I_CACHE_REFILL_PRFM", + "PublicDescription": "This event counts cache line refills in the L1 I-cache caused by a missed instruction fetch generated by software preload or prefetch instructions. Instruction fetches may include accessing multiple instructions, but the single cache line allocation is counted once." + }, + { + "ArchStdEvent": "L1I_CACHE_REFILL_HWPRF", + "PublicDescription": "This event counts each hardware prefetch access counted by L1I_CACHE_HWPRF that causes a refill of the Level 1I-cache from outside of the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_RD", + "PublicDescription": "This event counts demand instruction fetches that access the L1 I-cache and hit in the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_RD_FPRF", + "PublicDescription": "This event counts each demand fetch first hit counted by L1I_CACHE_HIT_RD where the cache line was fetched in response to a software preload or by a hardware prefetcher. That is, the L1I_CACHE_REFILL_PRF event was generated when the cache line was fetched into the cache.\nOnly the first hit by a demand access is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT", + "PublicDescription": "This event counts instruction fetches that access the L1 I-cache (demand, hardware prefetch, and software preload accesses) and hit in the L1 I-cache. I-cache accesses caused by cache maintenance operations are not counted." + }, + { + "ArchStdEvent": "L1I_CACHE_HIT_PRFM", + "PublicDescription": "This event counts instruction fetches generated by software preload or prefetch instructions that access the L1 I-cache and hit in the L1 I-cache." + }, + { + "ArchStdEvent": "L1I_LFB_HIT_RD", + "PublicDescription": "This event counts demand instruction fetches that access the L1 I-cache and hit in a line that is in the process of being loaded into the L1 I-cache." + }, + { + "EventCode": "0x0174", + "EventName": "L1I_HWPRF_REQ_DROP", + "PublicDescription": "L1 I-cache hardware prefetch dropped." + }, + { + "EventCode": "0x01e3", + "EventName": "L1I_CACHE_REFILL_RD", + "PublicDescription": "L1 I-cache refill, Read.\nThis event counts demand instruction fetch that causes a refill of the L1 I-cache of this PE, from outside of this cache." + }, + { + "EventCode": "0x01ea", + "EventName": "L1I_CFC_ENTRIES", + "PublicDescription": "This event counts the CFC (Cache Fill Control) entries.\nThe CFC is the fill buffer for I-cache." + }, + { + "EventCode": "0x01ef", + "EventName": "L1I_CACHE_INVAL", + "PublicDescription": "L1 I-cache invalidate.\nThis event counts each explicit invalidation of a cache line in the L1 I-cache caused by:\n* Broadcast cache coherency operations from another CPU in the system.\n* Invalidation dues to capacity eviction in L2 D-cache.\nThis event does not count for the following conditions:\n* A cache refill invalidates a cache line.\n* A CMO which is executed on that CPU Core and invalidates a cache line specified by Set/Way.\n* Cache Maintenance Operations (CMO) that operate by a virtual address.\nNote that\n* CMOs that operate by Set/Way cannot be broadcast from one CPU Core to another.\n* The CMO is treated as No-op for the purposes of L1 I-cache line invalidation, as this Core implements fully coherent I-cache." + }, + { + "EventCode": "0x0212", + "EventName": "L1I_CACHE_HIT_HWPRF", + "PublicDescription": "This event counts each hardware prefetch access that hits an L1 I-cache." + }, + { + "EventCode": "0x0215", + "EventName": "L1I_LFB_HIT", + "PublicDescription": "L1 Line fill buffer hit.\nThis event counts each Demand or software preload or hardware prefetch induced instruction fetch that hits an L1 I-cache line that is in the process of being loaded into the L1 instruction cache, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0216", + "EventName": "L1I_LFB_HIT_PRFM", + "PublicDescription": "This event counts each software prefetch access that hits a cache line that is in the process of being loaded into the L1 instruction cache, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0219", + "EventName": "L1I_LFB_HIT_HWPRF", + "PublicDescription": "This event counts each hardware prefetch access that hits a cache line that is in the process of being loaded into the L1 instruction cache, and so does not generate a new refill, but has to wait for the previous refill to complete." + }, + { + "EventCode": "0x0221", + "EventName": "L1I_PRFM_REQ", + "PublicDescription": "L1 I-cache software prefetch requests." + }, + { + "EventCode": "0x0222", + "EventName": "L1I_HWPRF_REQ", + "PublicDescription": "L1 I-cache hardware prefetch requests." + }, + { + "EventCode": "0x0228", + "EventName": "L1I_CACHE_HIT_PRFM_FPRF", + "PublicDescription": "L1 I-cache software prefetch access first hit, fetched by hardware or software prefetch.\nThis event counts each software preload access first hit where the cache line was fetched in response to a hardware prefetcher or software preload instruction.\nOnly the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x022a", + "EventName": "L1I_CACHE_HIT_HWPRF_FPRF", + "PublicDescription": "L1 I-cache hardware prefetch access first hit, fetched by hardware or software prefetch.\nThis event counts each hardware prefetch access first hit where the cache line was fetched in response to a hardware or prefetch instruction.\nOnly the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json new file mode 100644 index 0000000000000..66f21a94381ed --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/l2d_cache.json @@ -0,0 +1,134 @@ +[ + { + "ArchStdEvent": "L2D_CACHE", + "PublicDescription": "This event counts accesses to the L2 cache due to data accesses. L2 cache is a unified cache for data and instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses. This event also counts write-back of dirty data from L1 D-cache to the L2 cache.\nI-cache accesses are included in this event. This event is the sum of the following events:\nL2D_CACHE_RD,\nL2D_CACHE_WR,\nL2D_CACHE_PRFM, and\nL2D_CACHE_HWPRF." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL", + "PublicDescription": "This event counts cache line refills into the L2 cache. L2 cache is a unified cache for data and instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nI-cache refills are included in this event. This event is the sum of the following events:\nL2D_CACHE_REFILL_RD,\nL2D_CACHE_REFILL_WR,\nL2D_CACHE_REFILL_HWPRF, and\nL2D_CACHE_REFILL_PRFM." + }, + { + "ArchStdEvent": "L2D_CACHE_WB", + "PublicDescription": "This event counts write-backs of data from the L2 cache to outside the CPU. This includes snoops to the L2 (from other CPUs) which return data even if the snoops cause an invalidation. L2 cache line invalidations which do not write data outside the CPU and snoops which return data from an L1 cache are not counted. Data would not be written outside the cache when invalidating a clean cache line.\nThis event is the sum of the following events:\nL2D_CACHE_WB_VICTIM and\nL2D_CACHE_WB_CLEAN." + }, + { + "ArchStdEvent": "L2D_CACHE_RD", + "PublicDescription": "This event counts L2 D-cache accesses due to memory Read operations. L2 cache is a unified cache for data and instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nI-cache accesses are included in this event. This event is a subset of the L2D_CACHE event, but this event only counts memory Read operations." + }, + { + "ArchStdEvent": "L2D_CACHE_WR", + "PublicDescription": "This event counts L2 cache accesses due to memory Write operations. L2 cache is a unified cache for data and instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nThis event is a subset of the L2D_CACHE event, but this event only counts memory Write operations." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_RD", + "PublicDescription": "This event counts refills for memory accesses due to memory Read operation counted by L2D_CACHE_RD. L2 cache is a unified cache for data and instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nThis CPU includes I-cache refills in this counter as an L2I equivalent event was not implemented. This event is a subset of the L2D_CACHE_REFILL event. This event does not count L2 refills caused by stashes into L2.\nThis count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_WR", + "PublicDescription": "This event counts refills for memory accesses due to memory Write operation counted by L2D_CACHE_WR. L2 cache is a unified cache for data and instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nThis count includes demand requests that encounter an L2 prefetch request or an L2 software prefetch request to the same cache line, which is still pending in the L2 LFB." + }, + { + "ArchStdEvent": "L2D_CACHE_WB_VICTIM", + "PublicDescription": "This event counts evictions from the L2 cache because of a line being allocated into the L2 cache.\nThis event is a subset of the L2D_CACHE_WB event." + }, + { + "ArchStdEvent": "L2D_CACHE_WB_CLEAN", + "PublicDescription": "This event counts write-backs from the L2 cache that are a result of any of the following:\n* Cache maintenance operations,\n* Snoop responses, or\n* Direct cache transfers to another CPU due to a forwarding snoop request.\nThis event is a subset of the L2D_CACHE_WB event." + }, + { + "ArchStdEvent": "L2D_CACHE_INVAL", + "PublicDescription": "This event counts each explicit invalidation of a cache line in the L2 cache by cache maintenance operations that operate by a virtual address, or by external coherency operations. This event does not count if either:\n* A cache refill invalidates a cache line, or\n* A cache Maintenance Operation (CMO), which invalidates a cache line specified by Set/Way,\nis executed on that CPU.\nCMOs that operate by Set/Way cannot be broadcast from one CPU to another." + }, + { + "ArchStdEvent": "L2D_CACHE_LMISS_RD", + "PublicDescription": "This event counts cache line refills into the L2 unified cache from any memory Read operations that incurred additional latency.\nCounts the same as L2D_CACHE_REFILL_RD in this CPU" + }, + { + "ArchStdEvent": "L2D_CACHE_RW", + "PublicDescription": "This event counts L2 cache demand accesses from any Load/Store operations. L2 cache is a unified cache for data and instruction accesses, accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nI-cache accesses are included in this event.\nThis event is the sum of the following events:\nL2D_CACHE_RD and\nL2D_CACHE_WR." + }, + { + "ArchStdEvent": "L2D_CACHE_PRFM", + "PublicDescription": "This event counts L2 D-cache accesses generated by software preload or prefetch instructions with target = L1/L2/L3 cache.\nNote that a software preload or prefetch instructions with (target = L1/L2/L3) that hits in L1D will not result in an L2 D-cache access. Therefore, such a software preload or prefetch instructions will not be counted by this event." + }, + { + "ArchStdEvent": "L2D_CACHE_MISS", + "PublicDescription": "This event counts cache line misses in the L2 cache. L2 cache is a unified cache for data and instruction accesses. Accesses are for misses in the L1 D-cache or translation resolutions due to accesses.\nThis event counts the same as L2D_CACHE_REFILL_RD in this CPU." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_PRFM", + "PublicDescription": "This event counts refills due to accesses generated as a result of software preload or prefetch instructions as counted by L2D_CACHE_PRFM. I-cache refills are included in this event." + }, + { + "ArchStdEvent": "L2D_CACHE_HWPRF", + "PublicDescription": "This event counts the L2 D-cache access caused by L1 or L2 hardware prefetcher." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_HWPRF", + "PublicDescription": "This event counts each hardware prefetch access counted by L2D_CACHE_HWPRF that causes a refill of the L2 cache, or any L1 Data, or Instruction cache of this PE, from outside of those caches.\nThis does not include prefetch requests pending waiting for a refill in LFB and a new demand request to the same cache line hitting the LFB entry. All such refills are counted as L2D_LFB_HIT_RWL1PRF_FHWPRF." + }, + { + "ArchStdEvent": "L2D_CACHE_REFILL_PRF", + "PublicDescription": "This event counts each access to L2 Cache due to a prefetch instruction, or hardware prefetch that causes a refill of the L2 or any Level 1, from outside of those caches." + }, + { + "EventCode": "0x0108", + "EventName": "L2D_CACHE_IF_REFILL", + "PublicDescription": "L2 D-cache refill, instruction fetch.\nThis event counts demand instruction fetch that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0109", + "EventName": "L2D_CACHE_TBW_REFILL", + "PublicDescription": "L2 D-cache refill, Page table walk.\nThis event counts demand translation table walk that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x010a", + "EventName": "L2D_CACHE_PF_REFILL", + "PublicDescription": "L2 D-cache refill, prefetch.\nThis event counts L1 or L2 hardware or software prefetch accesses that causes a refill of the L2 cache or L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x010b", + "EventName": "L2D_LFB_HIT_RWL1PRF_FHWPRF", + "PublicDescription": "L2 line fill buffer demand Read, demand Write or L1 prefetch first hit, fetched by hardware prefetch.\nThis event counts each of the following access that hit the line-fill buffer when the same cache line is already being fetched due to an L2 hardware prefetcher.\n* Demand Read or Write\n* L1I-HWPRF\n* L1D-HWPRF\n* L1I PRFM\n* L1D PRFM\nThese accesses hit a cache line that is currently being loaded into the L2 cache as a result of a hardware prefetcher to the same line. Consequently, this access does not initiate a new refill but waits for the completion of the previous refill.\nOnly the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0179", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FHWPRF", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by hardware prefetch. This event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch. Only the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x01b8", + "EventName": "L2D_CACHE_L1PRF", + "PublicDescription": "L2 D-cache access, L1 hardware or software prefetch. This event counts L1 Hardware or software prefetch access to L2 D-cache." + }, + { + "EventCode": "0x01b9", + "EventName": "L2D_CACHE_REFILL_L1PRF", + "PublicDescription": "L2 D-cache refill, L1 hardware or software prefetch.\nThis event counts each access counted by L2D_CACHE_L1PRF that causes a refill of the L2 cache or any L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0201", + "EventName": "L2D_CACHE_BACKSNOOP_L1D_VIRT_ALIASING", + "PublicDescription": "This event counts when the L2 D-cache sends an invalidating back-snoop to the L1 D for an access initiated by the L1 D, where the corresponding line is already present in the L1 D-cache.\nThe L2 D-cache line tags the PE that refilled the line. It also retains specific bits of the VA to identify virtually aliased addresses.\nThe L1 D request requiring a back-snoop can originate either from the same PE that refilled the L2 D line or from a different PE. In either case, this event only counts those back snoop where the requested VA mismatch the VA stored in the L2 D tag.\nThis event is counted only by PE that initiated the original request necessitating a back-snoop.\nNote : The L1 D is VIPT, it identifies this access as a miss. Conversely, as L2 is PIPT, it identifies this as a hit. L2 D utilizes the back-snoop mechanism to refill L1 D with the snooped data." + }, + { + "EventCode": "0x0208", + "EventName": "L2D_CACHE_RWL1PRF", + "PublicDescription": "L2 D-cache access, demand Read, demand Write or L1 hardware or software prefetch.\nThis event counts each access to L2 D-cache due to the following:\n* Demand Read or Write.\n* L1 Hardware or software prefetch." + }, + { + "EventCode": "0x020a", + "EventName": "L2D_CACHE_REFILL_RWL1PRF", + "PublicDescription": "L2 D-cache refill, demand Read, demand Write or L1 hardware or software prefetch.\nThis event counts each access counted by L2D_CACHE_RWL1PRF that causes a refill of the L2 cache, or any L1 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x020c", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRFM", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software prefetch.\nThis event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 software prefetch. Only the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x020e", + "EventName": "L2D_CACHE_HIT_RWL1PRF_FPRF", + "PublicDescription": "L2 D-cache demand Read, demand Write and L1 prefetch hit, fetched by software or hardware prefetch.\nThis event counts each demand Read, demand Write and L1 hardware or software prefetch request that hit an L2 D-cache line that was refilled into L2 D-cache in response to an L2 hardware prefetch or software prefetch. Only the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json new file mode 100644 index 0000000000000..851d0a70de9c0 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/ll_cache.json @@ -0,0 +1,107 @@ +[ + { + "ArchStdEvent": "L3D_CACHE_ALLOCATE", + "PublicDescription": "This event counts each memory Write operation that writes an entire line into the L3 data without fetching data from outside the L3 Data. These are allocations of cache lines in the L3 Data that are not refills counted by\nL3D_CACHE_REFILL. For example:\nA Write-back of an entire cache line from an L2 cache to the L3 D-cache.\n* A Write of an entire cache line from a coalescing Write buffer.\n* An operation such as DC ZVA.\nThis counter does not count writes that write an entire line to beyond level 3. Thus this counter does not count the streaming writes to beyond L3 cache." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL", + "PublicDescription": "This event counts each access counted by L3D_CACHE that causes a refill of the L3 Data, or any L1 Data, instruction or L2 cache of this PE, from outside of those caches. This includes the refill due to hardware prefetch and software prefetch accesses.\nThis event is a sum of L3D_CACHE_MISS, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL_HWPRF event.\nA refill includes any access that causes data to be fetched from outside of the L1 to L3 caches, even if the data is ultimately not allocated into the L3 D-cache." + }, + { + "ArchStdEvent": "L3D_CACHE", + "PublicDescription": "This event counts each memory Read operation or memory Write operation that causes a cache access to the Level 3.\nThis event is a sum of the following Events:\n* L3D_CACHE_RD(0x00a0)\n* L3D_CACHE_ALLOCATE(0x0029)\n* L3D_CACHE_PRFM(0x8151)\n* L3D_CACHE_HWPRF(0x8156)\n* L2D_CACHE_WB(0x0018)" + }, + { + "ArchStdEvent": "LL_CACHE_RD", + "PublicDescription": "This is an alias to the event L3D_CACHE_RD (0x00a0)." + }, + { + "ArchStdEvent": "LL_CACHE_MISS_RD", + "PublicDescription": "This is an alias to the event L3D_CACHE_REFILL_RD (0x00a2)." + }, + { + "ArchStdEvent": "L3D_CACHE_RD", + "PublicDescription": "This event counts each Memory Read operation to L3 D-cache from instruction fetch, Load/Store, and MMU translation table accesses. This does not include hardware prefetcher or PRFM instruction accesses. This include L1 and L2 prefetcher accesses to L3 D-cache." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_RD", + "PublicDescription": "This event counts each access counted by both L3D_CACHE_RD and L3D_CACHE_REFILL. That is, every refill of the L3 cache counted by L3D_CACHE_REFILL that is caused by a Memory Read operation.\nThe L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_LMISS_RD", + "PublicDescription": "This event counts each memory Read operation to the L3 cache counted by L3D_CACHE that incurs additional latency because it returns data from outside of the L1 to L3 caches.\nThe L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_RW", + "PublicDescription": "This event counts each access counted by L3D_CACHE that is due to a demand memory Read operation or demand memory Write operation.\nThis event is a sum of L3D_CACHE_RD(0x00a0), L3D_CACHE_ALLOCATE(0x0029) and L2D_CACHE_WB(0x0018).\nNote that this counter does not count that writes an entire line to beyond level 3. Thus this counter does not count the streaming Writes to beyond L3 cache." + }, + { + "ArchStdEvent": "L3D_CACHE_PRFM", + "PublicDescription": "This event counts each access counted by L3D_CACHE that is due to a prefetch instruction. This includes L3 Data accesses due to the L1, L2, or L3 prefetch instruction." + }, + { + "ArchStdEvent": "L3D_CACHE_MISS", + "PublicDescription": "This event counts each demand Read access counted by L3D_CACHE_RD that misses in the L1 to L3 Data, causing an access to outside of the L3 cache.\nThe L3D_CACHE_MISS(0x8152), L3D_CACHE_REFILL_RD (0x00a2) and L3D_CACHE_LMISS_RD(0x400b) count the same event in the hardware." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_PRFM", + "PublicDescription": "This event counts each access counted by L3D_CACHE_PRFM that causes a refill of the L3 cache, or any L1 or L2 Data, from outside of those caches." + }, + { + "ArchStdEvent": "L3D_CACHE_HWPRF", + "PublicDescription": "This event counts each access to L3 cache that is due to a hardware prefetcher. This includes L3D accesses due to the Level-1 or Level-2 or Level-3 hardware prefetcher." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_HWPRF", + "PublicDescription": "This event counts each hardware prefetch counted by L3D_CACHE_HWPRF that causes a refill of the L3 Data or unified cache, or any L1 or L2 Data, Instruction, or unified cache of this PE, from outside of those caches." + }, + { + "ArchStdEvent": "L3D_CACHE_REFILL_PRF", + "PublicDescription": "This event counts each access to L3 cache due to a prefetch instruction, or hardware prefetch that causes a refill of the L3 Data, or any L1 or L2 Data, from outside of those caches." + }, + { + "EventCode": "0x01e8", + "EventName": "L3D_CACHE_RWL1PRFL2PRF", + "PublicDescription": "L3 cache access, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch.\nThis event counts each access to L3 D-cache due to the following:\n* Demand Read or Write.\n* L1 Hardware or software prefetch.\n* L2 Hardware or software prefetch." + }, + { + "EventCode": "0x01e9", + "EventName": "L3D_CACHE_REFILL_RWL1PRFL2PRF", + "PublicDescription": "L3 cache refill, demand Read, demand Write, L1 hardware or software prefetch or L2 hardware or software prefetch.\nThis event counts each access counted by L3D_CACHE_RWL1PRFL2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x01f6", + "EventName": "L3D_CACHE_REFILL_L2PRF", + "PublicDescription": "This event counts each access counted by L3D_CACHE_L2PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x01f7", + "EventName": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", + "PublicDescription": "L3 cache demand Read, demand Write, L1 prefetch L2 prefetch first hit, fetched by software or hardware prefetch.\nThis event counts each demand Read, demand Write, L1 hardware or software prefetch request and L2 hardware or software prefetch that hit an L3 D-cache line that was refilled into L3 D-cache in response to an L3 hardware prefetch or software prefetch. Only the first hit is counted. After this event is generated for a cache line, the event is not generated again for the same cache line while it remains in the cache." + }, + { + "EventCode": "0x0225", + "EventName": "L3D_CACHE_REFILL_IF", + "PublicDescription": "L3 cache refill, instruction fetch.\nThis event counts demand instruction fetch that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0226", + "EventName": "L3D_CACHE_REFILL_MM", + "PublicDescription": "L3 cache refill, translation table walk access.\nThis event counts demand translation table access that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x0227", + "EventName": "L3D_CACHE_REFILL_L1PRF", + "PublicDescription": "This event counts each access counted by L3D_CACHE_L1PRF that causes a refill of the L3 cache, or any L1 or L2 cache of this PE, from outside of those caches." + }, + { + "EventCode": "0x022c", + "EventName": "L3D_CACHE_L1PRF", + "PublicDescription": "This event counts the L3 D-cache access due to L1 hardware prefetch or software prefetch request.\nThe L1 hardware prefetch or software prefetch requests that miss the L1I, L1D and L2 D-cache are counted by this counter" + }, + { + "EventCode": "0x022d", + "EventName": "L3D_CACHE_L2PRF", + "PublicDescription": "This event counts the L3 D-cache access due to L2 hardware prefetch or software prefetch request.\nThe L2 hardware prefetch or software prefetch requests that miss the L2 D-cache are counted by this counter" + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json new file mode 100644 index 0000000000000..becd2d90bf396 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/memory.json @@ -0,0 +1,46 @@ +[ + { + "ArchStdEvent": "MEM_ACCESS", + "PublicDescription": "This event counts memory accesses issued by the CPU load/store unit, where those accesses are issued due to load or store operations. This event counts memory accesses regardless of whether the data is received from any level of cache hierarchy or external memory. If memory accesses are broken up into smaller transactions than what were specified in the load or store instructions, then the event counts those smaller memory transactions.\nMemory accesses generated by the following instructions or activity are not counted: instruction fetches, cache maintenance instructions, translation table walks or prefetches, memory prefetch operations. This event counts the sum of the following events:\nMEM_ACCESS_RD and\nMEM_ACCESS_WR." + }, + { + "ArchStdEvent": "MEMORY_ERROR", + "PublicDescription": "This event counts any detected correctable or uncorrectable physical memory errors (ECC or parity) in protected CPU RAMs. On the Core, this event counts errors in the caches (including data and tag RAMs). Any detected memory error (from either a speculative and abandoned access, or an architecturally executed access) is counted.\nNote that errors are only detected when the actual protected memory is accessed by an operation." + }, + { + "ArchStdEvent": "REMOTE_ACCESS", + "PublicDescription": "This event counts each external bus read access that causes an access to a remote device. That is, a socket that does not contain the PE." + }, + { + "ArchStdEvent": "MEM_ACCESS_RD", + "PublicDescription": "This event counts memory accesses issued by the CPU due to Load operations. This event counts any memory Load access, no matter whether the data is received from any level of cache hierarchy or external memory. This event also counts atomic Load operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the event counts those smaller transactions.\nThe following instructions are not counted:\n1) Instruction fetches,\n2) Cache maintenance instructions,\n3) Translation table walks or prefetches,\n4) Memory prefetch operations.\nThis event is a subset of the MEM_ACCESS event but the event only counts memory-Read operations." + }, + { + "ArchStdEvent": "MEM_ACCESS_WR", + "PublicDescription": "This event counts memory accesses issued by the CPU due to Store operations. This event counts any memory Store access, no matter whether the data is located in any level of cache or external memory. This event also counts atomic Load and Store operations. If memory accesses are broken up by the Load/Store unit into smaller transactions that are issued by the bus interface, then the event counts those smaller transactions." + }, + { + "ArchStdEvent": "LDST_ALIGN_LAT", + "PublicDescription": "This event counts the number of memory Read and Write accesses in a cycle that incurred additional latency due to the alignment of the address and the size of data being accessed, which results in a store crossing a single cache line.\nThis event is implemented as the sum of the following events on this CPU:\nLD_ALIGN_LAT and\nST_ALIGN_LAT." + }, + { + "ArchStdEvent": "LD_ALIGN_LAT", + "PublicDescription": "This event counts the number of memory Read accesses in a cycle that incurred additional latency due to the alignment of the address and size of data being accessed, which results in a load crossing a single cache line." + }, + { + "ArchStdEvent": "ST_ALIGN_LAT", + "PublicDescription": "This event counts the number of memory Write accesses in a cycle that incurred additional latency due to the alignment of the address and size of data being accessed." + }, + { + "ArchStdEvent": "INST_FETCH_PERCYC", + "PublicDescription": "This event counts number of instruction fetches outstanding per cycle, which will provide an average latency of instruction fetch." + }, + { + "ArchStdEvent": "MEM_ACCESS_RD_PERCYC", + "PublicDescription": "This event counts the number of outstanding Loads or memory Read accesses per cycle." + }, + { + "ArchStdEvent": "INST_FETCH", + "PublicDescription": "This event counts instruction memory accesses that the PE makes." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json new file mode 100644 index 0000000000000..b825ede03f544 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/metrics.json @@ -0,0 +1,722 @@ +[ + { + "MetricName": "backend_bound", + "MetricExpr": "100 * (STALL_SLOT_BACKEND / CPU_SLOT)", + "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the backend of the processor.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "backend_busy_bound", + "MetricExpr": "100 * (STALL_BACKEND_BUSY / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to issue queues being full to accept operations for execution.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_cache_l1d_bound", + "MetricExpr": "100 * (STALL_BACKEND_L1D / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L1 D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_cache_l2d_bound", + "MetricExpr": "100 * (STALL_BACKEND_MEM / (STALL_BACKEND_L1D + STALL_BACKEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by L2 D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_core_bound", + "MetricExpr": "100 * (STALL_BACKEND_CPUBOUND / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints not related to instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_core_rename_bound", + "MetricExpr": "100 * (STALL_BACKEND_RENAME / STALL_BACKEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend as the rename unit registers are unavailable.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_bound", + "MetricExpr": "100 * (STALL_BACKEND_MEMBOUND / STALL_BACKEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to backend Core resource constraints related to memory access latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_cache_bound", + "MetricExpr": "100 * ((STALL_BACKEND_L1D + STALL_BACKEND_MEM) / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory latency issues caused by D-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_store_bound", + "MetricExpr": "100 * (STALL_BACKEND_ST / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory Write pending caused by Stores stalled in the pre-commit stage.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_mem_tlb_bound", + "MetricExpr": "100 * (STALL_BACKEND_TLB / STALL_BACKEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the backend due to memory access latency issues caused by Data TLB misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Backend" + }, + { + "MetricName": "backend_stalled_cycles", + "MetricExpr": "100 * (STALL_BACKEND / CPU_CYCLES)", + "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the backend unit of the processor.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Cycle_Accounting" + }, + { + "MetricName": "bad_speculation", + "MetricExpr": "100 - (frontend_bound + retiring + backend_bound)", + "BriefDescription": "This metric is the percentage of total slots that executed operations and didn't retire due to a pipeline flush. This indicates cycles that were utilized but inefficiently.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "barrier_percentage", + "MetricExpr": "100 * ((ISB_SPEC + DSB_SPEC + DMB_SPEC) / INST_SPEC)", + "BriefDescription": "This metric measures instruction and data barrier operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "branch_direct_ratio", + "MetricExpr": "BR_IMMED_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of direct branches retired to the total number of branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "branch_indirect_ratio", + "MetricExpr": "BR_IND_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of indirect branches retired, including function returns, to the total number of branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "branch_misprediction_ratio", + "MetricExpr": "BR_MIS_PRED_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of branches mispredicted to the total number of branches architecturally executed. This gives an indication of the effectiveness of the branch prediction unit.", + "ScaleUnit": "1per branch", + "MetricGroup": "Miss_Ratio;Branch_Effectiveness" + }, + { + "MetricName": "branch_mpki", + "MetricExpr": "1000 * (BR_MIS_PRED_RETIRED / INST_RETIRED)", + "BriefDescription": "This metric measures the number of branch mispredictions per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;Branch_Effectiveness" + }, + { + "MetricName": "branch_percentage", + "MetricExpr": "100 * ((BR_IMMED_SPEC + BR_INDIRECT_SPEC) / INST_SPEC)", + "BriefDescription": "This metric measures branch operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "branch_return_ratio", + "MetricExpr": "BR_RETURN_RETIRED / BR_RETIRED", + "BriefDescription": "This metric measures the ratio of branches retired that are function returns to the total number of branches architecturally executed.", + "ScaleUnit": "1per branch", + "MetricGroup": "Branch_Effectiveness" + }, + { + "MetricName": "bus_bandwidth", + "MetricExpr": "BUS_ACCESS * 32 / duration_time ", + "BriefDescription": "This metric measures the bus-bandwidth of the data transferred between this PE's L2 with unCore in the system.", + "ScaleUnit": "1Bytes/sec" + }, + { + "MetricName": "cpu_cycles_fraction_in_st_mode", + "MetricExpr": "((CPU_SLOT/CPU_CYCLES) - 5) / 5", + "BriefDescription": "This metric counts fraction of the CPU cycles spent in ST mode during program execution.", + "ScaleUnit": "1fraction of cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "cpu_cycles_in_smt_mode", + "MetricExpr": "(1 - cpu_cycles_fraction_in_st_mode) * CPU_CYCLES", + "BriefDescription": "This metric counts CPU cycles in SMT mode during program execution.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "cpu_cycles_in_st_mode", + "MetricExpr": "cpu_cycles_fraction_in_st_mode * CPU_CYCLES", + "BriefDescription": "This metric counts CPU cycles in ST mode during program execution.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "SMT" + }, + { + "MetricName": "crypto_percentage", + "MetricExpr": "100 * (CRYPTO_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures crypto operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "dtlb_mpki", + "MetricExpr": "1000 * (DTLB_WALK / INST_RETIRED)", + "BriefDescription": "This metric measures the number of Data TLB Walks per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;DTLB_Effectiveness" + }, + { + "MetricName": "dtlb_walk_average_latency", + "MetricExpr": "DTLB_WALK_PERCYC / DTLB_WALK", + "BriefDescription": "This metric measures the average latency of Data TLB walks in CPU cycles.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "dtlb_walk_ratio", + "MetricExpr": "DTLB_WALK / L1D_TLB", + "BriefDescription": "This metric measures the ratio of Data TLB Walks to the total number of Data TLB accesses. This gives an indication of the effectiveness of the Data TLB accesses.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" + }, + { + "MetricName": "fp16_percentage", + "MetricExpr": "100 * (FP_HP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures half-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "fp32_percentage", + "MetricExpr": "100 * (FP_SP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures single-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "fp64_percentage", + "MetricExpr": "100 * (FP_DP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures double-precision floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "FP_Precision_Mix" + }, + { + "MetricName": "fp_ops_per_cycle", + "MetricExpr": "(FP_SCALE_OPS_SPEC + FP_FIXED_OPS_SPEC) / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by any instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "frontend_bound", + "MetricExpr": "100 * (STALL_SLOT_FRONTEND_WITHOUT_MISPRED / CPU_SLOT)", + "BriefDescription": "This metric is the percentage of total slots that were stalled due to resource constraints in the frontend of the processor.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "frontend_cache_l1i_bound", + "MetricExpr": "100 * (STALL_FRONTEND_L1I / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L1 I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_cache_l2i_bound", + "MetricExpr": "100 * (STALL_FRONTEND_MEM / (STALL_FRONTEND_L1I + STALL_FRONTEND_MEM))", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to memory access latency issues caused by L2 I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_bound", + "MetricExpr": "100 * (STALL_FRONTEND_CPUBOUND / STALL_FRONTEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints not related to instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_flow_bound", + "MetricExpr": "100 * (STALL_FRONTEND_FLOW / STALL_FRONTEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the decode unit is awaiting input from the branch prediction unit.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_core_flush_bound", + "MetricExpr": "100 * (STALL_FRONTEND_FLUSH / STALL_FRONTEND_CPUBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend as the processor is recovering from a pipeline flush caused by bad speculation or other machine resteers.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_bound", + "MetricExpr": "100 * (STALL_FRONTEND_MEMBOUND / STALL_FRONTEND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to frontend Core resource constraints related to the instruction fetch latency issues caused by memory access components.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_cache_bound", + "MetricExpr": "100 * ((STALL_FRONTEND_L1I + STALL_FRONTEND_MEM) / STALL_FRONTEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to instruction fetch latency issues caused by I-cache misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_mem_tlb_bound", + "MetricExpr": "100 * (STALL_FRONTEND_TLB / STALL_FRONTEND_MEMBOUND)", + "BriefDescription": "This metric is the percentage of total cycles stalled in the frontend due to instruction fetch latency issues caused by Instruction TLB misses.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Topdown_Frontend" + }, + { + "MetricName": "frontend_stalled_cycles", + "MetricExpr": "100 * (STALL_FRONTEND / CPU_CYCLES)", + "BriefDescription": "This metric is the percentage of cycles that were stalled due to resource constraints in the frontend unit of the processor.", + "ScaleUnit": "1percent of cycles", + "MetricGroup": "Cycle_Accounting" + }, + { + "MetricName": "instruction_fetch_average_latency", + "MetricExpr": "INST_FETCH_PERCYC / INST_FETCH", + "BriefDescription": "This metric measures the average latency of instruction fetches in CPU cycles.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "integer_dp_percentage", + "MetricExpr": "100 * (DP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalar integer operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "ipc", + "MetricExpr": "INST_RETIRED / CPU_CYCLES", + "BriefDescription": "This metric measures the number of instructions retired per cycle.", + "ScaleUnit": "1per cycle", + "MetricGroup": "General" + }, + { + "MetricName": "itlb_mpki", + "MetricExpr": "1000 * (ITLB_WALK / INST_RETIRED)", + "BriefDescription": "This metric measures the number of instruction TLB Walks per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness" + }, + { + "MetricName": "itlb_walk_average_latency", + "MetricExpr": "ITLB_WALK_PERCYC / ITLB_WALK", + "BriefDescription": "This metric measures the average latency of instruction TLB walks in CPU cycles.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "itlb_walk_ratio", + "MetricExpr": "ITLB_WALK / L1I_TLB", + "BriefDescription": "This metric measures the ratio of instruction TLB Walks to the total number of Instruction TLB accesses. This gives an indication of the effectiveness of the Instruction TLB accesses.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" + }, + { + "MetricName": "l1d_cache_miss_ratio", + "MetricExpr": "L1D_CACHE_REFILL / L1D_CACHE", + "BriefDescription": "This metric measures the ratio of L1 D-cache accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L1D_Cache_Effectiveness" + }, + { + "MetricName": "l1d_cache_mpki", + "MetricExpr": "1000 * (L1D_CACHE_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 D-cache accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L1D_Cache_Effectiveness" + }, + { + "MetricName": "l1d_cache_rw_miss_ratio", + "MetricExpr": "l1d_demand_misses / l1d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L1 D-cache Read accesses missed to the total number of L1 D-cache accesses. This gives an indication of the effectiveness of the L1 D-cache for demand Load or Store traffic.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_demand_accesses", + "MetricExpr": "L1D_CACHE_RW", + "BriefDescription": "This metric measures the count of L1 D-cache accesses incurred on Load or Store by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_demand_misses", + "MetricExpr": "L1D_CACHE_REFILL_RW", + "BriefDescription": "This metric measures the count of L1 D-cache misses incurred on a Load or Store by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_prf_accuracy", + "MetricExpr": "100 * (l1d_useful_prf / l1d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the instruction stream.", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_prf_coverage", + "MetricExpr": "100 * (l1d_useful_prf / (l1d_demand_misses + l1d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_refilled_prf", + "MetricExpr": "L1D_CACHE_REFILL_HWPRF + L1D_CACHE_REFILL_PRFM + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 data prefetcher (hardware prefetches or software preload) into L1 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1d_tlb_miss_ratio", + "MetricExpr": "L1D_TLB_REFILL / L1D_TLB", + "BriefDescription": "This metric measures the ratio of L1 Data TLB accesses missed to the total number of L1 Data TLB accesses. This gives an indication of the effectiveness of the L1 Data TLB.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;DTLB_Effectiveness" + }, + { + "MetricName": "l1d_tlb_mpki", + "MetricExpr": "1000 * (L1D_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 Data TLB accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;DTLB_Effectiveness" + }, + { + "MetricName": "l1d_useful_prf", + "MetricExpr": "L1D_CACHE_HIT_RW_FPRF + L1D_LFB_HIT_RW_FHWPRF + L1D_LFB_HIT_RW_FPRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 data prefetcher (hardware prefetches or software preload) into L1 D-cache which are further used by Load or Store from the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1I_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_cache_miss_ratio", + "MetricExpr": "L1I_CACHE_REFILL / L1I_CACHE", + "BriefDescription": "This metric measures the ratio of L1 I-cache accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L1I_Cache_Effectiveness" + }, + { + "MetricName": "l1i_cache_mpki", + "MetricExpr": "1000 * (L1I_CACHE_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 I-cache accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L1I_Cache_Effectiveness" + }, + { + "MetricName": "l1i_cache_rd_miss_ratio", + "MetricExpr": "l1i_demand_misses / l1i_demand_accesses", + "BriefDescription": "This metric measures the ratio of L1 I-cache Read accesses missed to the total number of L1 I-cache accesses. This gives an indication of the effectiveness of the L1 I-cache for demand instruction fetch traffic. Note that cache accesses in this cache are demand instruction fetch.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_demand_accesses", + "MetricExpr": "L1I_CACHE_RD", + "BriefDescription": "This metric measures the count of L1 I-cache accesses caused by an instruction fetch by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_demand_misses", + "MetricExpr": "L1I_CACHE_REFILL_RD", + "BriefDescription": "This metric measures the count of L1 I-cache misses caused by an instruction fetch by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_prf_accuracy", + "MetricExpr": "100 * (l1i_useful_prf / l1i_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the instruction stream.", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_prf_coverage", + "MetricExpr": "100 * (l1i_useful_prf / (l1i_demand_misses + l1i_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_refilled_prf", + "MetricExpr": "L1I_CACHE_REFILL_HWPRF + L1I_CACHE_REFILL_PRFM", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 instruction prefetcher (hardware prefetches or software preload) into L1 I-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l1i_tlb_miss_ratio", + "MetricExpr": "L1I_TLB_REFILL / L1I_TLB", + "BriefDescription": "This metric measures the ratio of L1 Instruction TLB accesses missed to the total number of L1 Instruction TLB accesses. This gives an indication of the effectiveness of the L1 Instruction TLB.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness" + }, + { + "MetricName": "l1i_tlb_mpki", + "MetricExpr": "1000 * (L1I_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L1 Instruction TLB accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness" + }, + { + "MetricName": "l1i_useful_prf", + "MetricExpr": "L1I_CACHE_HIT_RD_FPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L1 instruction prefetcher (hardware prefetches or software preload) into L1 I-cache which are further used by instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L1D_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2_cache_miss_ratio", + "MetricExpr": "L2D_CACHE_REFILL / L2D_CACHE", + "BriefDescription": "This metric measures the ratio of L2 cache accesses missed to the total number of L2 cache accesses. This gives an indication of the effectiveness of the L2 cache, which is a unified cache that stores both data and instruction.\nNote that cache accesses in this cache are either data memory access or instruction fetch as this is a unified cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;L2_Cache_Effectiveness" + }, + { + "MetricName": "l2_cache_mpki", + "MetricExpr": "1000 * (l2d_demand_misses / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L2 unified cache accesses missed per thousand instructions executed.\nNote that cache accesses in this cache are either data memory access or instruction fetch as this is a unified cache.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;L2_Cache_Effectiveness" + }, + { + "MetricName": "l2_tlb_miss_ratio", + "MetricExpr": "L2D_TLB_REFILL / L2D_TLB", + "BriefDescription": "This metric measures the ratio of L2 unified TLB accesses missed to the total number of L2 unified TLB accesses.\nThis gives an indication of the effectiveness of the L2 TLB.", + "ScaleUnit": "1per TLB access", + "MetricGroup": "Miss_Ratio;ITLB_Effectiveness;DTLB_Effectiveness" + }, + { + "MetricName": "l2_tlb_mpki", + "MetricExpr": "1000 * (L2D_TLB_REFILL / INST_RETIRED)", + "BriefDescription": "This metric measures the number of L2 unified TLB accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;ITLB_Effectiveness;DTLB_Effectiveness" + }, + { + "MetricName": "l2d_cache_rwl1prf_miss_ratio", + "MetricExpr": "l2d_demand_misses / l2d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L2 D-cache Read accesses missed to the total number of L2 D-cache accesses.\nThis gives an indication of the effectiveness of the L2 D-cache for demand instruction fetch, Load, Store, or L1 prefetcher accesses traffic.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_demand_accesses", + "MetricExpr": "L2D_CACHE_RD + L2D_CACHE_WR + L2D_CACHE_L1PRF", + "BriefDescription": "This metric measures the count of L2 D-cache accesses incurred on an instruction fetch, Load, Store, or L1 prefetcher accesses by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_demand_misses", + "MetricExpr": "L2D_CACHE_REFILL_RD + L2D_CACHE_REFILL_WR + L2D_CACHE_REFILL_L1PRF", + "BriefDescription": "This metric measures the count of L2 D-cache misses incurred on an instruction fetch, Load, Store, or L1 prefetcher accesses by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_prf_accuracy", + "MetricExpr": "100 * (l2d_useful_prf / l2d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the instruction stream.", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_prf_coverage", + "MetricExpr": "100 * (l2d_useful_prf / (l2d_demand_misses + l2d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_refilled_prf", + "MetricExpr": "(L2D_CACHE_REFILL_PRF - L2D_CACHE_REFILL_L1PRF) + L2D_LFB_HIT_RWL1PRF_FHWPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L2 data prefetcher (hardware prefetches or software preload) into L2 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l2d_useful_prf", + "MetricExpr": "L2D_CACHE_HIT_RWL1PRF_FPRF + L2D_LFB_HIT_RWL1PRF_FHWPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L2 data prefetcher (hardware prefetches or software preload) into L2 D-cache which are further used by instruction fetch, Load, Store, or L1 prefetcher accesses from the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L2_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_cache_rwl1prfl2prf_miss_ratio", + "MetricExpr": "l3d_demand_misses / l3d_demand_accesses", + "BriefDescription": "This metric measures the ratio of L3 D-cache Read accesses missed to the total number of L3 D-cache accesses. This gives an indication of the effectiveness of the L2 D-cache for demand instruction fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses traffic.", + "ScaleUnit": "1per cache access", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_demand_accesses", + "MetricExpr": "L3D_CACHE_RWL1PRFL2PRF", + "BriefDescription": "This metric measures the count of L3 D-cache accesses incurred on an instruction fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_demand_misses", + "MetricExpr": "L3D_CACHE_REFILL_RWL1PRFL2PRF", + "BriefDescription": "This metric measures the count of L3 D-cache misses incurred on an instruction fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses by the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_prf_accuracy", + "MetricExpr": "100 * (l3d_useful_prf / l3d_refilled_prf)", + "BriefDescription": "This metric measures the fraction of prefetched memory addresses that are used by the instruction stream.", + "ScaleUnit": "1percent of prefetch", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_prf_coverage", + "MetricExpr": "100 * (l3d_useful_prf / (l3d_demand_misses + l3d_refilled_prf))", + "BriefDescription": "This metric measures the baseline demand cache misses which the prefetcher brings into the cache.", + "ScaleUnit": "1percent of cache access", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_refilled_prf", + "MetricExpr": "L3D_CACHE_REFILL_HWPRF + L3D_CACHE_REFILL_PRFM - L3D_CACHE_REFILL_L1PRF - L3D_CACHE_REFILL_L2PRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L3 data prefetcher (hardware prefetches or software preload) into L3 D-cache.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "l3d_useful_prf", + "MetricExpr": "L3D_CACHE_HIT_RWL1PRFL2PRF_FPRF", + "BriefDescription": "This metric measures the count of cache lines refilled by L3 data prefetcher (hardware prefetches or software preload) into L3 D-cache which are further used by instruction fetch, Load, Store, L1 prefetcher, or L2 prefetcher accesses from the instruction stream of the program.", + "ScaleUnit": "1count", + "MetricGroup": "L3_Prefetcher_Effectiveness" + }, + { + "MetricName": "ll_cache_read_hit_ratio", + "MetricExpr": "(LL_CACHE_RD - LL_CACHE_MISS_RD) / LL_CACHE_RD", + "BriefDescription": "This metric measures the ratio of last level cache Read accesses hit in the cache to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either data memory access or instruction fetch as this is a system level cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "LL_Cache_Effectiveness" + }, + { + "MetricName": "ll_cache_read_miss_ratio", + "MetricExpr": "LL_CACHE_MISS_RD / LL_CACHE_RD", + "BriefDescription": "This metric measures the ratio of last level cache Read accesses missed to the total number of last level cache accesses. This gives an indication of the effectiveness of the last level cache for Read traffic. Note that cache accesses in this cache are either data memory access or instruction fetch as this is a system level cache.", + "ScaleUnit": "1per cache access", + "MetricGroup": "Miss_Ratio;LL_Cache_Effectiveness" + }, + { + "MetricName": "ll_cache_read_mpki", + "MetricExpr": "1000 * (LL_CACHE_MISS_RD / INST_RETIRED)", + "BriefDescription": "This metric measures the number of last level cache Read accesses missed per thousand instructions executed.", + "ScaleUnit": "1MPKI", + "MetricGroup": "MPKI;LL_Cache_Effectiveness" + }, + { + "MetricName": "load_average_latency", + "MetricExpr": "MEM_ACCESS_RD_PERCYC / MEM_ACCESS", + "BriefDescription": "This metric measures the average latency of Load operations in CPU cycles.", + "ScaleUnit": "1CPU cycles", + "MetricGroup": "Average_Latency" + }, + { + "MetricName": "load_percentage", + "MetricExpr": "100 * (LD_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures Load operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "nonsve_fp_ops_per_cycle", + "MetricExpr": "FP_FIXED_OPS_SPEC / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by an instruction that is not an SVE instruction. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "retiring", + "MetricExpr": "100 * ((OP_RETIRED/OP_SPEC) * (1 - (STALL_SLOT/CPU_SLOT)))", + "BriefDescription": "This metric is the percentage of total slots that retired operations, which indicates cycles that were utilized efficiently.", + "ScaleUnit": "1percent of slots", + "MetricGroup": "TopdownL1" + }, + { + "MetricName": "scalar_fp_percentage", + "MetricExpr": "100 * (VFP_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalar floating point operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "simd_percentage", + "MetricExpr": "100 * (ASE_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures advanced SIMD operations as a percentage of total operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "store_percentage", + "MetricExpr": "100 * (ST_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures Store operations as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "sve_all_percentage", + "MetricExpr": "100 * (SVE_INST_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalable vector operations, including Loads and Stores, as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "Operation_Mix" + }, + { + "MetricName": "sve_fp_ops_per_cycle", + "MetricExpr": "FP_SCALE_OPS_SPEC / CPU_CYCLES", + "BriefDescription": "This metric measures floating point operations per cycle in any precision performed by SVE instructions. Operations are counted by computation and by vector lanes, fused computations such as multiply-add count as twice per vector lane for example.", + "ScaleUnit": "1operations per cycle", + "MetricGroup": "FP_Arithmetic_Intensity" + }, + { + "MetricName": "sve_predicate_empty_percentage", + "MetricExpr": "100 * (SVE_PRED_EMPTY_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with no active predicates as a percentage of SVE predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_full_percentage", + "MetricExpr": "100 * (SVE_PRED_FULL_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with all active predicates as a percentage of SVE predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_partial_percentage", + "MetricExpr": "100 * (SVE_PRED_PARTIAL_SPEC / SVE_PRED_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with at least one active predicates as a percentage of SVE predicated operations speculatively executed.", + "ScaleUnit": "1percent of SVE predicated operations", + "MetricGroup": "SVE_Effectiveness" + }, + { + "MetricName": "sve_predicate_percentage", + "MetricExpr": "100 * (SVE_PRED_SPEC / INST_SPEC)", + "BriefDescription": "This metric measures scalable vector operations with predicates as a percentage of operations speculatively executed.", + "ScaleUnit": "1percent of operations", + "MetricGroup": "SVE_Effectiveness" + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json new file mode 100644 index 0000000000000..8ff87d844e521 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/misc.json @@ -0,0 +1,642 @@ +[ + { + "ArchStdEvent": "SW_INCR", + "PublicDescription": "This event counts software writes to the PMSWINC_EL0 (software PMU increment) register. The PMSWINC_EL0 register is a manually updated counter for use by application software.\nThis event could be used to measure any user program event, such as accesses to a particular data structure (by writing to the PMSWINC_EL0 register each time the data structure is accessed).\nTo use the PMSWINC_EL0 register and event, developers must insert instructions that write to the PMSWINC_EL0 register into the source code.\nSince the SW_INCR event records writes to the PMSWINC_EL0 register, there is no need to do a Read/Increment/Write sequence to the PMSWINC_EL0 register." + }, + { + "ArchStdEvent": "TRB_WRAP", + "PublicDescription": "This event is generated each time the trace buffer current Write pointer is wrapped to the trace buffer base pointer." + }, + { + "ArchStdEvent": "TRCEXTOUT0", + "PublicDescription": "Trace unit external output 0." + }, + { + "ArchStdEvent": "TRCEXTOUT1", + "PublicDescription": "Trace unit external output 1." + }, + { + "ArchStdEvent": "TRCEXTOUT2", + "PublicDescription": "Trace unit external output 2." + }, + { + "ArchStdEvent": "TRCEXTOUT3", + "PublicDescription": "Trace unit external output 3." + }, + { + "ArchStdEvent": "CTI_TRIGOUT4", + "PublicDescription": "Cross-trigger Interface output trigger 4." + }, + { + "ArchStdEvent": "CTI_TRIGOUT5", + "PublicDescription": "Cross-trigger Interface output trigger 5." + }, + { + "ArchStdEvent": "CTI_TRIGOUT6", + "PublicDescription": "Cross-trigger Interface output trigger 6." + }, + { + "ArchStdEvent": "CTI_TRIGOUT7", + "PublicDescription": "Cross-trigger Interface output trigger 7." + }, + { + "EventCode": "0x00e1", + "EventName": "L1I_PRFM_REQ_DROP", + "PublicDescription": "L1 I-cache software prefetch dropped." + }, + { + "EventCode": "0x0100", + "EventName": "L1_PF_REFILL", + "PublicDescription": "L1 prefetch requests, refilled to L1 cache." + }, + { + "EventCode": "0x0120", + "EventName": "FLUSH", + "PublicDescription": "This event counts both the CT flush and BX flush. The BR_MIS_PRED counts the BX flushes. So the FLUSH-BR_MIS_PRED gives the CT flushes." + }, + { + "EventCode": "0x0121", + "EventName": "FLUSH_MEM", + "PublicDescription": "Flushes due to memory hazards. This only includes CT flushes." + }, + { + "EventCode": "0x0122", + "EventName": "FLUSH_BAD_BRANCH", + "PublicDescription": "Flushes due to bad predicted branch. This only includes CT flushes." + }, + { + "EventCode": "0x0123", + "EventName": "FLUSH_STDBYPASS", + "PublicDescription": "Flushes due to bad predecode. This only includes CT flushes." + }, + { + "EventCode": "0x0124", + "EventName": "FLUSH_ISB", + "PublicDescription": "Flushes due to ISB or similar side-effects. This only includes CT flushes." + }, + { + "EventCode": "0x0125", + "EventName": "FLUSH_OTHER", + "PublicDescription": "Flushes due to other hazards. This only includes CT flushes." + }, + { + "EventCode": "0x0126", + "EventName": "STORE_STREAM", + "PublicDescription": "Stored lines in streaming no-Write-allocate mode." + }, + { + "EventCode": "0x0127", + "EventName": "NUKE_RAR", + "PublicDescription": "Load/Store nuke due to Read-after-Read ordering hazard." + }, + { + "EventCode": "0x0128", + "EventName": "NUKE_RAW", + "PublicDescription": "Load/Store nuke due to Read-after-Write ordering hazard." + }, + { + "EventCode": "0x0129", + "EventName": "L1_PF_GEN_PAGE", + "PublicDescription": "Load/Store prefetch to L1 generated, Page mode." + }, + { + "EventCode": "0x012a", + "EventName": "L1_PF_GEN_STRIDE", + "PublicDescription": "Load/Store prefetch to L1 generated, stride mode." + }, + { + "EventCode": "0x012b", + "EventName": "L2_PF_GEN_LD", + "PublicDescription": "Load prefetch to L2 generated." + }, + { + "EventCode": "0x012d", + "EventName": "LS_PF_TRAIN_TABLE_ALLOC", + "PublicDescription": "LS prefetch train table entry allocated." + }, + { + "EventCode": "0x0130", + "EventName": "LS_PF_GEN_TABLE_ALLOC", + "PublicDescription": "This event counts the number of cycles with at least one table allocation, for L2 hardware prefetches (including the software PRFM instructions that are converted into hardware prefetches due to D-TLB miss).\nLS prefetch gen table allocation (for L2 prefetches)." + }, + { + "EventCode": "0x0131", + "EventName": "LS_PF_GEN_TABLE_ALLOC_PF_PEND", + "PublicDescription": "This event counts the number of cycles in which at least one hardware prefetch is dropped due to the inability to identify a victim when the generation table is full. The hardware prefetch considered here includes the software PRFM that is converted into hardware prefetches due to D-TLB miss." + }, + { + "EventCode": "0x0132", + "EventName": "TBW", + "PublicDescription": "Tablewalks." + }, + { + "EventCode": "0x0134", + "EventName": "S1L2_HIT", + "PublicDescription": "Translation cache hit on S1L2 walk cache entry." + }, + { + "EventCode": "0x0135", + "EventName": "S1L1_HIT", + "PublicDescription": "Translation cache hit on S1L1 walk cache entry." + }, + { + "EventCode": "0x0136", + "EventName": "S1L0_HIT", + "PublicDescription": "Translation cache hit on S1L0 walk cache entry." + }, + { + "EventCode": "0x0137", + "EventName": "S2L2_HIT", + "PublicDescription": "Translation cache hit for S2L2 IPA walk cache entry." + }, + { + "EventCode": "0x0138", + "EventName": "IPA_REQ", + "PublicDescription": "Translation cache lookups for IPA to PA entries." + }, + { + "EventCode": "0x0139", + "EventName": "IPA_REFILL", + "PublicDescription": "Translation cache refills for IPA to PA entries." + }, + { + "EventCode": "0x013a", + "EventName": "S1_FLT", + "PublicDescription": "Stage1 tablewalk fault." + }, + { + "EventCode": "0x013b", + "EventName": "S2_FLT", + "PublicDescription": "Stage2 tablewalk fault." + }, + { + "EventCode": "0x013c", + "EventName": "COLT_REFILL", + "PublicDescription": "Aggregated page refill." + }, + { + "EventCode": "0x0145", + "EventName": "L1_PF_HIT", + "PublicDescription": "L1 prefetch requests, hitting in L1 cache." + }, + { + "EventCode": "0x0146", + "EventName": "L1_PF", + "PublicDescription": "L1 prefetch requests." + }, + { + "EventCode": "0x0147", + "EventName": "CACHE_LS_REFILL", + "PublicDescription": "L2 D-cache refill, Load/Store." + }, + { + "EventCode": "0x0148", + "EventName": "CACHE_PF", + "PublicDescription": "L2 prefetch requests." + }, + { + "EventCode": "0x0149", + "EventName": "CACHE_PF_HIT", + "PublicDescription": "L2 prefetch requests, hitting in L2 cache." + }, + { + "EventCode": "0x0150", + "EventName": "UNUSED_PF", + "PublicDescription": "L2 unused prefetch." + }, + { + "EventCode": "0x0151", + "EventName": "PFT_SENT", + "PublicDescription": "L2 prefetch TGT sent.\nNote that PFT_SENT != PFT_USEFUL + PFT_DROP. There may be PFT_SENT for which the accesses resulted in a SLC hit." + }, + { + "EventCode": "0x0152", + "EventName": "PFT_USEFUL", + "PublicDescription": "L2 prefetch TGT useful." + }, + { + "EventCode": "0x0153", + "EventName": "PFT_DROP", + "PublicDescription": "L2 prefetch TGT dropped." + }, + { + "EventCode": "0x0162", + "EventName": "LRQ_FULL", + "PublicDescription": "This event counts the number of cycles the LRQ is full." + }, + { + "EventCode": "0x0163", + "EventName": "FETCH_FQ_EMPTY", + "PublicDescription": "Fetch Queue empty cycles." + }, + { + "EventCode": "0x0164", + "EventName": "FPG2", + "PublicDescription": "Forward progress guarantee. Medium range livelock triggered." + }, + { + "EventCode": "0x0165", + "EventName": "FPG", + "PublicDescription": "Forward progress guarantee. Tofu global livelock buster is triggered." + }, + { + "EventCode": "0x0172", + "EventName": "DEADBLOCK", + "PublicDescription": "Write-back evictions converted to dataless EVICT.\nThe victim line is deemed deadblock if the likeliness of a reuse is low. The Core uses dataless evict to evict a deadblock; and it uses an evict with data to evict an L2 line that is not a deadblock." + }, + { + "EventCode": "0x0173", + "EventName": "PF_PRQ_ALLOC_PF_PEND", + "PublicDescription": "L1 prefetch prq allocation (replacing pending)." + }, + { + "EventCode": "0x0178", + "EventName": "FETCH_ICACHE_INSTR", + "PublicDescription": "Instructions fetched from I-cache." + }, + { + "EventCode": "0x017b", + "EventName": "NEAR_CAS", + "PublicDescription": "Near atomics: compare and swap." + }, + { + "EventCode": "0x017c", + "EventName": "NEAR_CAS_PASS", + "PublicDescription": "Near atomics: compare and swap pass." + }, + { + "EventCode": "0x017d", + "EventName": "FAR_CAS", + "PublicDescription": "Far atomics: compare and swap." + }, + { + "EventCode": "0x0186", + "EventName": "L2_BTB_RELOAD_MAIN_BTB", + "PublicDescription": "Number of completed L1 BTB update initiated by L2 BTB hit which swap branch information between L1 BTB and L2 BTB." + }, + { + "EventCode": "0x018f", + "EventName": "L1_PF_GEN_MCMC", + "PublicDescription": "Load/Store prefetch to L1 generated, MCMC." + }, + { + "EventCode": "0x0190", + "EventName": "PF_MODE_0_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most aggressive mode." + }, + { + "EventCode": "0x0191", + "EventName": "PF_MODE_1_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the more aggressive mode." + }, + { + "EventCode": "0x0192", + "EventName": "PF_MODE_2_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the less aggressive mode." + }, + { + "EventCode": "0x0193", + "EventName": "PF_MODE_3_CYCLES", + "PublicDescription": "Number of cycles in which the hardware prefetcher is in the most conservative mode." + }, + { + "EventCode": "0x0194", + "EventName": "TXREQ_LIMIT_MAX_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is the L2_TQ_SIZE." + }, + { + "EventCode": "0x0195", + "EventName": "TXREQ_LIMIT_3QUARTER_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 3/4 of the L2_TQ_SIZE and the L2_TQ_SIZE-1." + }, + { + "EventCode": "0x0196", + "EventName": "TXREQ_LIMIT_HALF_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/2 of the L2_TQ_SIZE and 3/4 of the L2_TQ_SIZE." + }, + { + "EventCode": "0x0197", + "EventName": "TXREQ_LIMIT_1QUARTER_CYCLES", + "PublicDescription": "Number of cycles in which the dynamic TXREQ limit is between 1/4 of the L2_TQ_SIZE and 1/2 of the L2_TQ_SIZE." + }, + { + "EventCode": "0x019d", + "EventName": "PREFETCH_LATE_CMC", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by CMC prefetch request." + }, + { + "EventCode": "0x019e", + "EventName": "PREFETCH_LATE_BO", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by BO prefetch request." + }, + { + "EventCode": "0x019f", + "EventName": "PREFETCH_LATE_STRIDE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by STRIDE prefetch request." + }, + { + "EventCode": "0x01a0", + "EventName": "PREFETCH_LATE_SPATIAL", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SPATIAL prefetch request." + }, + { + "EventCode": "0x01a2", + "EventName": "PREFETCH_LATE_TBW", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by TBW prefetch request." + }, + { + "EventCode": "0x01a3", + "EventName": "PREFETCH_LATE_PAGE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by PAGE prefetch request." + }, + { + "EventCode": "0x01a4", + "EventName": "PREFETCH_LATE_GSMS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by GSMS prefetch request." + }, + { + "EventCode": "0x01a5", + "EventName": "PREFETCH_LATE_SIP_CONS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit on TQ entry allocated by SIP_CONS prefetch request." + }, + { + "EventCode": "0x01a6", + "EventName": "PREFETCH_REFILL_CMC", + "PublicDescription": "PF/prefetch or PF/readclean request from CMC pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a7", + "EventName": "PREFETCH_REFILL_BO", + "PublicDescription": "PF/prefetch or PF/readclean request from BO pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a8", + "EventName": "PREFETCH_REFILL_STRIDE", + "PublicDescription": "PF/prefetch or PF/readclean request from STRIDE pf engine filled the L2 cache." + }, + { + "EventCode": "0x01a9", + "EventName": "PREFETCH_REFILL_SPATIAL", + "PublicDescription": "PF/prefetch or PF/readclean request from SPATIAL pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ab", + "EventName": "PREFETCH_REFILL_TBW", + "PublicDescription": "PF/prefetch or PF/readclean request from TBW pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ac", + "EventName": "PREFETCH_REFILL_PAGE", + "PublicDescription": "PF/prefetch or PF/readclean request from PAGE pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ad", + "EventName": "PREFETCH_REFILL_GSMS", + "PublicDescription": "PF/prefetch or PF/readclean request from GSMS pf engine filled the L2 cache." + }, + { + "EventCode": "0x01ae", + "EventName": "PREFETCH_REFILL_SIP_CONS", + "PublicDescription": "PF/prefetch or PF/readclean request from SIP_CONS pf engine filled the L2 cache." + }, + { + "EventCode": "0x01af", + "EventName": "CACHE_HIT_LINE_PF_CMC", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by CMC prefetch request." + }, + { + "EventCode": "0x01b0", + "EventName": "CACHE_HIT_LINE_PF_BO", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by BO prefetch request." + }, + { + "EventCode": "0x01b1", + "EventName": "CACHE_HIT_LINE_PF_STRIDE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by STRIDE prefetch request." + }, + { + "EventCode": "0x01b2", + "EventName": "CACHE_HIT_LINE_PF_SPATIAL", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SPATIAL prefetch request." + }, + { + "EventCode": "0x01b4", + "EventName": "CACHE_HIT_LINE_PF_TBW", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by TBW prefetch request." + }, + { + "EventCode": "0x01b5", + "EventName": "CACHE_HIT_LINE_PF_PAGE", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by PAGE prefetch request." + }, + { + "EventCode": "0x01b6", + "EventName": "CACHE_HIT_LINE_PF_GSMS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by GSMS prefetch request." + }, + { + "EventCode": "0x01b7", + "EventName": "CACHE_HIT_LINE_PF_SIP_CONS", + "PublicDescription": "LS/readclean or LS/readunique lookup hit in L2 cache on line filled by SIP_CONS prefetch request." + }, + { + "EventCode": "0x01ba", + "EventName": "PREFETCH_LATE_STORE_ISSUE", + "PublicDescription": "This event counts the number of demand requests that matches a Store-issue prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bb", + "EventName": "PREFETCH_LATE_STORE_STRIDE", + "PublicDescription": "This event counts the number of demand requests that matches a Store-stride prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bc", + "EventName": "PREFETCH_LATE_PC_OFFSET", + "PublicDescription": "This event counts the number of demand requests that matches a PC-offset prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01bd", + "EventName": "PREFETCH_LATE_IFUPF", + "PublicDescription": "This event counts the number of demand requests that matches a IFU prefetcher's pending refill request. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements." + }, + { + "EventCode": "0x01be", + "EventName": "PREFETCH_REFILL_STORE_ISSUE", + "PublicDescription": "This event counts the number of cache refills due to Store-Issue prefetcher." + }, + { + "EventCode": "0x01bf", + "EventName": "PREFETCH_REFILL_STORE_STRIDE", + "PublicDescription": "This event counts the number of cache refills due to Store-stride prefetcher." + }, + { + "EventCode": "0x01c0", + "EventName": "PREFETCH_REFILL_PC_OFFSET", + "PublicDescription": "This event counts the number of cache refills due to PC-offset prefetcher." + }, + { + "EventCode": "0x01c1", + "EventName": "PREFETCH_REFILL_IFUPF", + "PublicDescription": "This event counts the number of cache refills due to IFU prefetcher." + }, + { + "EventCode": "0x01c2", + "EventName": "CACHE_HIT_LINE_PF_STORE_ISSUE", + "PublicDescription": "This event counts the number of first hit to a cache line filled by Store-issue prefetcher." + }, + { + "EventCode": "0x01c3", + "EventName": "CACHE_HIT_LINE_PF_STORE_STRIDE", + "PublicDescription": "This event counts the number of first hit to a cache line filled by Store-stride prefetcher." + }, + { + "EventCode": "0x01c4", + "EventName": "CACHE_HIT_LINE_PF_PC_OFFSET", + "PublicDescription": "This event counts the number of first hit to a cache line filled by PC-offset prefetcher." + }, + { + "EventCode": "0x01c5", + "EventName": "CACHE_HIT_LINE_PF_IFUPF", + "PublicDescription": "This event counts the number of first hit to a cache line filled by IFU prefetcher." + }, + { + "EventCode": "0x01c6", + "EventName": "L2_PF_GEN_ST_ISSUE", + "PublicDescription": "Store-issue prefetch to L2 generated." + }, + { + "EventCode": "0x01c7", + "EventName": "L2_PF_GEN_ST_STRIDE", + "PublicDescription": "Store-stride prefetch to L2 generated" + }, + { + "EventCode": "0x01cb", + "EventName": "L2_TQ_OUTSTANDING", + "PublicDescription": "Outstanding tracker count, per cycle.\nThis event increments by the number of valid entries pertaining to this thread in the L2TQ, in each cycle.\nThis event can be used to calculate the occupancy of L2TQ by dividing this by the CPU_CYCLES event. The L2TQ queue tracks the outstanding Read, Write and Snoop transactions. The Read transaction and the Write transaction entries are attributable to PE, whereas the Snoop transactions are not always attributable to PE." + }, + { + "EventCode": "0x01cc", + "EventName": "TXREQ_LIMIT_COUNT_CYCLES", + "PublicDescription": "This event increments by the dynamic TXREQ value, in each cycle.\nThis is a companion event of TXREQ_LIMIT_MAX_CYCLES, TXREQ_LIMIT_3QUARTER_CYCLES, TXREQ_LIMIT_HALF_CYCLES, and TXREQ_LIMIT_1QUARTER_CYCLES." + }, + { + "EventCode": "0x01ce", + "EventName": "L3DPRFM_TO_L2PRQ_CONVERTED", + "PublicDescription": "This event counts the number of Converted-L3D-PRFMs. These are indeed L3D PRFM and activities around these PRFM are counted by the L3D_CACHE_PRFM, L3D_CACHE_REFILL_PRFM and L3D_CACHE_REFILL Events." + }, + { + "EventCode": "0x01d2", + "EventName": "DVM_TLBI_RCVD", + "PublicDescription": "This event counts the number of TLBI DVM message received over CHI interface, for *this* Core." + }, + { + "EventCode": "0x01d6", + "EventName": "DSB_COMMITING_LOCAL_TLBI", + "PublicDescription": "This event counts the number of DSB that are retired and committed at least one local TLBI instruction. This event increments no more than once (in a cycle) even if the DSB commits multiple local TLBI instruction." + }, + { + "EventCode": "0x01d7", + "EventName": "DSB_COMMITING_BROADCAST_TLBI", + "PublicDescription": "This event counts the number of DSB that are retired and committed at least one broadcast TLBI instruction. This event increments no more than once (in a cycle) even if the DSB commits multiple broadcast TLBI instruction." + }, + { + "EventCode": "0x01eb", + "EventName": "L1DPRFM_L2DPRFM_TO_L2PRQ_CONVERTED", + "PublicDescription": "This event counts the number of Converted-L1D-PRFMs and Converted-L2D-PRFM.\nActivities involving the Converted-L1D-PRFM are counted by the L1D_CACHE_PRFM. However they are *not* counted by the L1D_CACHE_REFILL_PRFM, and L1D_CACHE_REFILL, as these Converted-L1D-PRFM are treated as L2 D hardware prefetches. Activities around the Converted-L1D-PRFMs and Converted-L2D-PRFMs are counted by the L2D_CACHE_PRFM, L2D_CACHE_REFILL_PRFM and L2D_CACHE_REFILL Events." + }, + { + "EventCode": "0x01ec", + "EventName": "PREFETCH_LATE_CONVERTED_PRFM", + "PublicDescription": "This event counts the number of demand requests that matches a Converted-L1D-PRFM or Converted-L2D-PRFM pending refill request at L2 D-cache. These are called late prefetch requests and are still counted as useful prefetcher requests for the sake of accuracy and coverage measurements.\nNote that this event is not counted by the L2D_CACHE_HIT_RWL1PRF_LATE_HWPRF, though the Converted-L1D-PRFM or Converted-L2D-PRFM are replayed by the L2PRQ." + }, + { + "EventCode": "0x01ed", + "EventName": "PREFETCH_REFILL_CONVERTED_PRFM", + "PublicDescription": "This event counts the number of L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM.\nNote : L2D_CACHE_REFILL_PRFM is inclusive of PREFETCH_REFILL_PRFM_CONVERTED, where both the PREFETCH_REFILL_PRFM_CONVERTED and the L2D_CACHE_REFILL_PRFM increment when L2 D-cache refills due to Converted-L1D-PRFM or Converted-L2D-PRFM." + }, + { + "EventCode": "0x01ee", + "EventName": "CACHE_HIT_LINE_PF_CONVERTED_PRFM", + "PublicDescription": "This event counts the number of first hit to a cache line filled by Converted-L1D-PRFM or Converted-L2D-PRFM.\nNote that L2D_CACHE_HIT_RWL1PRF_FPRFM is inclusive of CACHE_HIT_LINE_PF_CONVERTED_PRFM, where both the CACHE_HIT_LINE_PF_CONVERTED_PRFM and the L2D_CACHE_HIT_RWL1PRF_FPRFM increment on a first hit to L2 D-cache filled by Converted-L1D-PRFM or Converted-L2D-PRFM." + }, + { + "EventCode": "0x01f0", + "EventName": "TMS_ST_TO_SMT_LATENCY", + "PublicDescription": "This event counts the number of CPU cycles spent on TMS for ST-to-SMT switch.\nThis event is counted by both the threads - This event in both threads increment during TMS for ST-to-SMT switch." + }, + { + "EventCode": "0x01f1", + "EventName": "TMS_SMT_TO_ST_LATENCY", + "PublicDescription": "This event counts the number of CPU cycles spent on TMS for SMT-to-ST switch. The count also includes the CPU cycles spend due to an aborted SMT-to-ST TMS attempt.\nThis event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x01f2", + "EventName": "TMS_ST_TO_SMT_COUNT", + "PublicDescription": "This event counts the number of completed TMS from ST-to-SMT.\nThis event is counted only by the active thread (the one that is not in WFI).\nNote: When an active thread enters the Debug state in ST-Full resource mode, it is switched to SMT mode. This is because the inactive thread cannot wake up while the other thread remains in the Debug state. To prEvent this issue, threads operating in ST-Full resource mode are transitioned to SMT mode upon entering Debug state. This event count will also reflect such switches from ST to SMT mode.\n(Also see the (NV_CPUACTLR14_EL1.chka_prEvent_st_tx_to_smt_when_tx_in_debug_state bit to disable this behavior.)" + }, + { + "EventCode": "0x01f3", + "EventName": "TMS_SMT_TO_ST_COUNT", + "PublicDescription": "This event counts the number of completed TMS from SMT-to-ST.\nThis event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x01f4", + "EventName": "TMS_SMT_TO_ST_COUNT_ABRT", + "PublicDescription": "This event counts the number of aborted TMS from SMT-to-ST.\nThis event is counted only by the thread that is not in WFI." + }, + { + "EventCode": "0x0202", + "EventName": "L0I_CACHE_RD", + "PublicDescription": "This event counts the number of predict blocks serviced out of L0 I-cache.\nNote: The L0 I-cache performs at most 4 L0 I look-up in a cycle. Two of which are to service PB from L0 I. And the other two to refill L0 I-cache from L1 I. This event count only the L0 I-cache lookup pertaining to servicing the PB from L0 I." + }, + { + "EventCode": "0x0203", + "EventName": "L0I_CACHE_REFILL", + "PublicDescription": "This event counts the number of L0I cache refill from L1 I-cache." + }, + { + "EventCode": "0x0207", + "EventName": "INTR_LATENCY", + "PublicDescription": "This event counts the number of cycles elapsed between when an Interrupt is recognized (after masking) to when a uop associated with the first instruction in the destination exception level is allocated. If there is some other flush condition that pre-empts the Interrupt, then the cycles counted terminates early at the first instruction executed after that flush. In the event of dropped Interrupts (when an Interrupt is deasserted before it is taken), this counter measures the number of cycles that elapse from the moment an Interrupt is recognized (post-masking) until the Interrupt is dropped or deasserted.\nNote that\n* IESB(Implicit Error Synchronization Barrier) is an internal mop, so the latency of an implicit IESB mop executed before the Interrupt taken is included in the Interrupt latency count.\n* Nukes or TMS sequence within the window are also counted by the Interrupt latency Event.\n* A SMT to ST TMS will be aborted on detecting the wake condition for the WFI thread. The Interrupt latency count includes any additional penalty for an aborted TMS." + }, + { + "EventCode": "0x021c", + "EventName": "CWT_ALLOC_ENTRY", + "PublicDescription": "Cache Way Tracker Allocate entry." + }, + { + "EventCode": "0x021d", + "EventName": "CWT_ALLOC_LINE", + "PublicDescription": "Cache Way Tracker Allocate line." + }, + { + "EventCode": "0x021e", + "EventName": "CWT_HIT", + "PublicDescription": "Cache Way Tracker hit." + }, + { + "EventCode": "0x021f", + "EventName": "CWT_HIT_TAG", + "PublicDescription": "Cache Way Tracker hit when ITAG lookup suppressed." + }, + { + "EventCode": "0x0220", + "EventName": "CWT_REPLAY_TAG", + "PublicDescription": "Cache Way Tracker causes ITAG replay due to miss when ITAG lookup suppressed." + }, + { + "EventCode": "0x0250", + "EventName": "GPT_REQ", + "PublicDescription": "GPT lookup." + }, + { + "EventCode": "0x0251", + "EventName": "GPT_WC_HIT", + "PublicDescription": "GPT lookup hit in Walk cache." + }, + { + "EventCode": "0x0252", + "EventName": "GPT_PG_HIT", + "PublicDescription": "GPT lookup hit in TLB." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json new file mode 100644 index 0000000000000..34c7eefa66b05 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/retired.json @@ -0,0 +1,94 @@ +[ + { + "ArchStdEvent": "INST_RETIRED", + "PublicDescription": "This event counts instructions that have been architecturally executed." + }, + { + "ArchStdEvent": "CID_WRITE_RETIRED", + "PublicDescription": "This event counts architecturally executed writes to the CONTEXTIDR_EL1 register, which usually contains the kernel PID and can be output with hardware trace." + }, + { + "ArchStdEvent": "BR_IMMED_RETIRED", + "PublicDescription": "This event counts architecturally executed direct branches." + }, + { + "ArchStdEvent": "BR_RETURN_RETIRED", + "PublicDescription": "This event counts architecturally executed procedure returns." + }, + { + "ArchStdEvent": "TTBR_WRITE_RETIRED", + "PublicDescription": "This event counts architectural writes to TTBR0/1_EL1. If virtualization host extensions are enabled (by setting the HCR_EL2.E2H bit to 1), then accesses to TTBR0/1_EL1 that are redirected to TTBR0/1_EL2, or accesses to TTBR0/1_EL12, are counted. TTBRn registers are typically updated when the kernel is swapping user-space threads or applications." + }, + { + "ArchStdEvent": "BR_RETIRED", + "PublicDescription": "This event counts architecturally executed branches, whether the branch is taken or not. Instructions that explicitly write to the PC are also counted. Note that exception generating instructions, exception return instructions, and context synchronization instructions are not counted." + }, + { + "ArchStdEvent": "BR_MIS_PRED_RETIRED", + "PublicDescription": "This event counts branches counted by BR_RETIRED which were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "OP_RETIRED", + "PublicDescription": "This event counts micro-operations that are architecturally executed. This is a count of number of micro-operations retired from the commit queue in a single cycle." + }, + { + "ArchStdEvent": "BR_INDNR_TAKEN_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches excluding procedure returns that were taken." + }, + { + "ArchStdEvent": "BR_IMMED_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed direct branches that were correctly predicted." + }, + { + "ArchStdEvent": "BR_IMMED_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed direct branches that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_IND_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches including procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_IND_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches including procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_RETURN_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_RETURN_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_INDNR_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches excluding procedure returns that were correctly predicted." + }, + { + "ArchStdEvent": "BR_INDNR_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches excluding procedure returns that were mispredicted and caused a pipeline flush." + }, + { + "ArchStdEvent": "BR_TAKEN_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed branches that were taken and were correctly predicted." + }, + { + "ArchStdEvent": "BR_TAKEN_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed branches that were taken and were mispredicted causing a pipeline flush." + }, + { + "ArchStdEvent": "BR_SKIP_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed branches that were not taken and were correctly predicted." + }, + { + "ArchStdEvent": "BR_SKIP_MIS_PRED_RETIRED", + "PublicDescription": "This event counts architecturally executed branches that were not taken and were mispredicted causing a pipeline flush." + }, + { + "ArchStdEvent": "BR_PRED_RETIRED", + "PublicDescription": "This event counts branch instructions counted by BR_RETIRED which were correctly predicted." + }, + { + "ArchStdEvent": "BR_IND_RETIRED", + "PublicDescription": "This event counts architecturally executed indirect branches including procedure returns." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json new file mode 100644 index 0000000000000..00d0c5051a482 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spe.json @@ -0,0 +1,42 @@ +[ + { + "ArchStdEvent": "SAMPLE_POP", + "PublicDescription": "This event counts statistical profiling sample population, the count of all operations that could be sampled but may or may not be chosen for sampling." + }, + { + "ArchStdEvent": "SAMPLE_FEED", + "PublicDescription": "This event counts statistical profiling samples taken for sampling." + }, + { + "ArchStdEvent": "SAMPLE_FILTRATE", + "PublicDescription": "This event counts statistical profiling samples taken which are not removed by filtering." + }, + { + "ArchStdEvent": "SAMPLE_COLLISION", + "PublicDescription": "This event counts statistical profiling samples that have collided with a previous sample and so therefore not taken." + }, + { + "ArchStdEvent": "SAMPLE_FEED_BR", + "PublicDescription": "This event counts statistical profiling samples taken which are branches." + }, + { + "ArchStdEvent": "SAMPLE_FEED_LD", + "PublicDescription": "This event counts statistical profiling samples taken which are Loads or Load atomic operations." + }, + { + "ArchStdEvent": "SAMPLE_FEED_ST", + "PublicDescription": "This event counts statistical profiling samples taken which are Stores or Store atomic operations." + }, + { + "ArchStdEvent": "SAMPLE_FEED_OP", + "PublicDescription": "This event counts statistical profiling samples taken which are matching any operation type filters supported." + }, + { + "ArchStdEvent": "SAMPLE_FEED_EVENT", + "PublicDescription": "This event counts statistical profiling samples taken which are matching event packet filter constraints." + }, + { + "ArchStdEvent": "SAMPLE_FEED_LAT", + "PublicDescription": "This event counts statistical profiling samples taken which are exceeding minimum latency set by operation latency filter constraints." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json new file mode 100644 index 0000000000000..8bc802f5f3500 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/spec_operation.json @@ -0,0 +1,230 @@ +[ + { + "ArchStdEvent": "INST_SPEC", + "PublicDescription": "This event counts operations that have been speculatively executed." + }, + { + "ArchStdEvent": "OP_SPEC", + "PublicDescription": "This event counts micro-operations speculatively executed. This is the count of the number of micro-operations dispatched in a cycle." + }, + { + "ArchStdEvent": "UNALIGNED_LD_SPEC", + "PublicDescription": "This event counts unaligned memory Read operations issued by the CPU. This event counts unaligned accesses (as defined by the actual instruction), even if they are subsequently issued as multiple aligned accesses.\nThis event does not count preload operations (PLD, PLI).\nThis event is a subset of the UNALIGNED_LDST_SPEC event." + }, + { + "ArchStdEvent": "UNALIGNED_ST_SPEC", + "PublicDescription": "This event counts unaligned memory Write operations issued by the CPU. This event counts unaligned accesses (as defined by the actual instruction), even if they are subsequently issued as multiple aligned accesses.\nThis event is a subset of the UNALIGNED_LDST_SPEC event." + }, + { + "ArchStdEvent": "UNALIGNED_LDST_SPEC", + "PublicDescription": "This event counts unaligned memory operations issued by the CPU. This event counts unaligned accesses (as defined by the actual instruction), even if they are subsequently issued as multiple aligned accesses.\nThis event is the sum of the following events:\nUNALIGNED_ST_SPEC and\nUNALIGNED_LD_SPEC." + }, + { + "ArchStdEvent": "LDREX_SPEC", + "PublicDescription": "This event counts Load-Exclusive operations that have been speculatively executed. For example: LDREX, LDX" + }, + { + "ArchStdEvent": "STREX_PASS_SPEC", + "PublicDescription": "This event counts Store-exclusive operations that have been speculatively executed and have successfully completed the Store operation." + }, + { + "ArchStdEvent": "STREX_FAIL_SPEC", + "PublicDescription": "This event counts Store-exclusive operations that have been speculatively executed and have not successfully completed the Store operation." + }, + { + "ArchStdEvent": "STREX_SPEC", + "PublicDescription": "This event counts Store-exclusive operations that have been speculatively executed.\nThis event is the sum of the following events:\nSTREX_PASS_SPEC and\nSTREX_FAIL_SPEC." + }, + { + "ArchStdEvent": "LD_SPEC", + "PublicDescription": "This event counts speculatively executed Load operations including Single Instruction Multiple Data (SIMD) Load operations." + }, + { + "ArchStdEvent": "ST_SPEC", + "PublicDescription": "This event counts speculatively executed Store operations including Single Instruction Multiple Data (SIMD) Store operations." + }, + { + "ArchStdEvent": "LDST_SPEC", + "PublicDescription": "This event counts Load and Store operations that have been speculatively executed." + }, + { + "ArchStdEvent": "DP_SPEC", + "PublicDescription": "This event counts speculatively executed logical or arithmetic instructions such as MOV/MVN operations." + }, + { + "ArchStdEvent": "ASE_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD operations excluding Load, Store, and Move micro-operations that move data to or from SIMD (vector) registers." + }, + { + "ArchStdEvent": "VFP_SPEC", + "PublicDescription": "This event counts speculatively executed floating point operations. This event does not count operations that move data to or from floating point (vector) registers." + }, + { + "ArchStdEvent": "PC_WRITE_SPEC", + "PublicDescription": "This event counts speculatively executed operations which cause software changes of the PC. Those operations include all taken branch operations." + }, + { + "ArchStdEvent": "CRYPTO_SPEC", + "PublicDescription": "This event counts speculatively executed cryptographic operations except for PMULL and VMULL operations." + }, + { + "ArchStdEvent": "BR_IMMED_SPEC", + "PublicDescription": "This event counts direct branch operations which are speculatively executed." + }, + { + "ArchStdEvent": "BR_RETURN_SPEC", + "PublicDescription": "This event counts procedure return operations (RET, RETAA and RETAB) which are speculatively executed." + }, + { + "ArchStdEvent": "BR_INDIRECT_SPEC", + "PublicDescription": "This event counts indirect branch operations including procedure returns, which are speculatively executed. This includes operations that force a software change of the PC, other than exception-generating operations and direct branch instructions. Some examples of the instructions counted by this event include BR Xn, RET, etc." + }, + { + "ArchStdEvent": "ISB_SPEC", + "PublicDescription": "This event counts ISB operations that are executed." + }, + { + "ArchStdEvent": "DSB_SPEC", + "PublicDescription": "This event counts DSB operations that are speculatively issued to Load/Store unit in the CPU." + }, + { + "ArchStdEvent": "DMB_SPEC", + "PublicDescription": "This event counts DMB operations that are speculatively issued to the Load/Store unit in the CPU. This event does not count implied barriers from Load-acquire/Store-release operations." + }, + { + "ArchStdEvent": "CSDB_SPEC", + "PublicDescription": "This event counts CSDB operations that are speculatively issued to the Load/Store unit in the CPU. This event does not count implied barriers from Load-acquire/Store-release operations." + }, + { + "ArchStdEvent": "RC_LD_SPEC", + "PublicDescription": "This event counts any Load acquire operations that are speculatively executed. For example: LDAR, LDARH, LDARB" + }, + { + "ArchStdEvent": "RC_ST_SPEC", + "PublicDescription": "This event counts any Store release operations that are speculatively executed. For example: STLR, STLRH, STLRB" + }, + { + "ArchStdEvent": "SIMD_INST_SPEC", + "PublicDescription": "This event counts speculatively executed operations that are SIMD or SVE vector operations or Advanced SIMD non-scalar operations." + }, + { + "ArchStdEvent": "ASE_INST_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD operations." + }, + { + "ArchStdEvent": "SVE_INST_SPEC", + "PublicDescription": "This event counts speculatively executed operations that are SVE operations." + }, + { + "ArchStdEvent": "INT_SPEC", + "PublicDescription": "This event counts speculatively executed integer arithmetic operations." + }, + { + "ArchStdEvent": "SVE_PRED_SPEC", + "PublicDescription": "This event counts speculatively executed predicated SVE operations.\nThis counter also counts SVE operation due to instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* events." + }, + { + "ArchStdEvent": "SVE_PRED_EMPTY_SPEC", + "PublicDescription": "This event counts speculatively executed predicated SVE operations with no active predicate elements.\nThis counter also counts SVE operation due to instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* events." + }, + { + "ArchStdEvent": "SVE_PRED_FULL_SPEC", + "PublicDescription": "This event counts speculatively executed predicated SVE operations with all predicate elements active.\nThis counter also counts SVE operation due to instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* events." + }, + { + "ArchStdEvent": "SVE_PRED_PARTIAL_SPEC", + "PublicDescription": "This event counts speculatively executed predicated SVE operations with at least one but not all active predicate elements.\nThis counter also counts SVE operation due to instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* events." + }, + { + "ArchStdEvent": "SVE_PRED_NOT_FULL_SPEC", + "PublicDescription": "This event counts speculatively executed predicated SVE operations with at least one non active predicate elements.\nThis counter also counts SVE operation due to instruction with Governing predicate operand that determines the Active elements that do not write to any SVE Z vector destination register using either zeroing or merging predicate. Thus, the operations due to instructions such as INCP, DECP, UQINCP, UQDECP, SQINCP, SQDECP and PNEXT, are counted by the SVE_PRED_* events." + }, + { + "ArchStdEvent": "PRF_SPEC", + "PublicDescription": "This event counts speculatively executed operations that prefetch memory. For example, Scalar: PRFM, SVE: PRFB, PRFD, PRFH, or PRFW." + }, + { + "ArchStdEvent": "SVE_LDFF_SPEC", + "PublicDescription": "This event counts speculatively executed SVE first fault or non-fault Load operations." + }, + { + "ArchStdEvent": "SVE_LDFF_FAULT_SPEC", + "PublicDescription": "This event counts speculatively executed SVE first fault or non-fault Load operations that clear at least one bit in the FFR." + }, + { + "ArchStdEvent": "ASE_SVE_INT8_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD or SVE integer operations with the largest data type being an 8-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT16_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD or SVE integer operations with the largest data type a 16-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT32_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD or SVE integer operations with the largest data type a 32-bit integer." + }, + { + "ArchStdEvent": "ASE_SVE_INT64_SPEC", + "PublicDescription": "This event counts speculatively executed Advanced SIMD or SVE integer operations with the largest data type a 64-bit integer." + }, + { + "EventCode": "0x011d", + "EventName": "SPEC_RET_STACK_FULL", + "PublicDescription": "This event counts predict pipe stalls due to speculative return address predictor full." + }, + { + "EventCode": "0x011f", + "EventName": "MOPS_SPEC", + "PublicDescription": "Macro-ops speculatively decoded." + }, + { + "EventCode": "0x0180", + "EventName": "BR_SPEC_PRED_TAKEN", + "PublicDescription": "Number of predicted taken from branch predictor." + }, + { + "EventCode": "0x0181", + "EventName": "BR_SPEC_PRED_TAKEN_FROM_L2BTB", + "PublicDescription": "Number of predicted taken branch from L2 BTB." + }, + { + "EventCode": "0x0182", + "EventName": "BR_SPEC_PRED_TAKEN_MULTI", + "PublicDescription": "Number of predicted taken for polymorphic branch." + }, + { + "EventCode": "0x0185", + "EventName": "BR_SPEC_PRED_STATIC", + "PublicDescription": "Number of post fetch prediction." + }, + { + "EventCode": "0x01d0", + "EventName": "TLBI_LOCAL_SPEC", + "PublicDescription": "A non-broadcast TLBI instruction executed (Speculatively or otherwise) on *this* PE." + }, + { + "EventCode": "0x01d1", + "EventName": "TLBI_BROADCAST_SPEC", + "PublicDescription": "A broadcast TLBI instruction executed (Speculatively or otherwise) on *this* PE." + }, + { + "EventCode": "0x01e7", + "EventName": "BR_SPEC_PRED_ALN_REDIR", + "PublicDescription": "BPU predict pipe align redirect (either AL-APQ hit/miss)." + }, + { + "EventCode": "0x0200", + "EventName": "SIMD_CRYPTO_INST_SPEC", + "PublicDescription": "SIMD, SVE, and CRYPTO instructions speculatively decoded." + }, + { + "EventCode": "0x022e", + "EventName": "VPRED_LD_SPEC", + "PublicDescription": "This event counts the number of Speculatively-executed-Load operations with addresses produced by the value-prediction mechanism. The loaded data might be discarded if the predicted address differs from the actual address." + }, + { + "EventCode": "0x022f", + "EventName": "VPRED_LD_SPEC_MISMATCH", + "PublicDescription": "This event counts a subset of VPRED_LD_SPEC where the predicted Load address and the actual address mismatched." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json new file mode 100644 index 0000000000000..92d9e0866c247 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/stall.json @@ -0,0 +1,145 @@ +[ + { + "ArchStdEvent": "STALL_FRONTEND", + "PublicDescription": "This event counts cycles when frontend could not send any micro-operations to the rename stage because of frontend resource stalls caused by fetch memory latency or branch prediction flow stalls. STALL_FRONTEND_SLOTS counts SLOTS during the cycle when this event counts. STALL_SLOT_FRONTEND will count SLOTS when this event is counted on this CPU." + }, + { + "ArchStdEvent": "STALL_BACKEND", + "PublicDescription": "This event counts cycles whenever the rename unit is unable to send any micro-operations to the backend of the pipeline because of backend resource constraints. Backend resource constraints can include issue stage fullness, execution stage fullness, or other internal pipeline resource fullness. All the backend slots were empty during the cycle when this event counts." + }, + { + "ArchStdEvent": "STALL", + "PublicDescription": "This event counts cycles when no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall). This event is the sum of the following events:\nSTALL_FRONTEND and\nSTALL_BACKEND." + }, + { + "ArchStdEvent": "STALL_SLOT_BACKEND", + "PublicDescription": "This event counts slots per cycle in which no operations are sent from the rename unit to the backend due to backend resource constraints. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND counts at least 1. STALL_BACKEND counts during the cycle when STALL_SLOT_BACKEND is SLOTS." + }, + { + "ArchStdEvent": "STALL_SLOT_FRONTEND", + "PublicDescription": "This event counts slots per cycle in which no operations are sent to the rename unit from the frontend due to frontend resource constraints. STALL_FRONTEND counts during the cycle when STALL_SLOT_FRONTEND is SLOTS." + }, + { + "ArchStdEvent": "STALL_SLOT", + "PublicDescription": "This event counts slots per cycle in which no operations are sent to the rename unit from the frontend or from the rename unit to the backend for any reason (either frontend or backend stall).\nSTALL_SLOT is the sum of the following events:\nSTALL_SLOT_FRONTEND and\nSTALL_SLOT_BACKEND." + }, + { + "ArchStdEvent": "STALL_BACKEND_MEM", + "PublicDescription": "This event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the last level Core cache.\nLast level cache in this CPU is Level 2, hence this event counts same as STALL_BACKEND_L2D." + }, + { + "ArchStdEvent": "STALL_FRONTEND_MEMBOUND", + "PublicDescription": "This event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the memory resources." + }, + { + "ArchStdEvent": "STALL_FRONTEND_L1I", + "PublicDescription": "This event counts cycles when the frontend is stalled because there is an instruction fetch request pending in the L1 I-cache." + }, + { + "ArchStdEvent": "STALL_FRONTEND_MEM", + "PublicDescription": "This event counts cycles when the frontend is stalled because there is an instruction fetch request pending in the last level Core cache.\nLast level cache in this CPU is Level 2, hence this event counts rather than STALL_FRONTEND_L2I." + }, + { + "ArchStdEvent": "STALL_FRONTEND_TLB", + "PublicDescription": "This event counts when the frontend is stalled on any TLB misses being handled. This event also counts the TLB accesses made by hardware prefetches." + }, + { + "ArchStdEvent": "STALL_FRONTEND_CPUBOUND", + "PublicDescription": "This event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the CPU resources excluding memory resources." + }, + { + "ArchStdEvent": "STALL_FRONTEND_FLOW", + "PublicDescription": "This event counts cycles when the frontend could not send any micro-operations to the rename stage due to resource constraints in the branch prediction unit." + }, + { + "ArchStdEvent": "STALL_FRONTEND_FLUSH", + "PublicDescription": "This event counts cycles when the frontend could not send any micro-operations to the rename stage as the frontend is recovering from a machine flush or resteer. Example scenarios that cause a flush include branch mispredictions, taken exceptions, microarchitectural flush etc." + }, + { + "ArchStdEvent": "STALL_BACKEND_MEMBOUND", + "PublicDescription": "This event counts cycles when the backend could not accept any micro-operations due to resource constraints in the memory resources." + }, + { + "ArchStdEvent": "STALL_BACKEND_L1D", + "PublicDescription": "This event counts cycles when the backend is stalled because there is a pending demand Load request in progress in the L1 D-cache." + }, + { + "ArchStdEvent": "STALL_BACKEND_TLB", + "PublicDescription": "This event counts cycles when the backend is stalled on any demand TLB misses being handled." + }, + { + "ArchStdEvent": "STALL_BACKEND_ST", + "PublicDescription": "This event counts cycles when the backend is stalled and there is a Store that has not reached the pre-commit stage." + }, + { + "ArchStdEvent": "STALL_BACKEND_CPUBOUND", + "PublicDescription": "This event counts cycles when the backend could not accept any micro-operations due to any resource constraints in the CPU excluding memory resources." + }, + { + "ArchStdEvent": "STALL_BACKEND_BUSY", + "PublicDescription": "This event counts cycles when the backend could not accept any micro-operations because the issue queues are full to take any operations for execution." + }, + { + "ArchStdEvent": "STALL_BACKEND_ILOCK", + "PublicDescription": "This event counts cycles when the backend could not accept any micro-operations due to resource constraints imposed by input dependency." + }, + { + "ArchStdEvent": "STALL_BACKEND_RENAME", + "PublicDescription": "This event counts cycles when backend is stalled even when operations are available from the frontend but at least one is not ready to be sent to the backend because no rename register is available." + }, + { + "EventCode": "0x0158", + "EventName": "FLAG_DISP_STALL", + "PublicDescription": "Rename stalled due to FRF(Flag register file) full." + }, + { + "EventCode": "0x0159", + "EventName": "GEN_DISP_STALL", + "PublicDescription": "Rename stalled due to GRF (General-purpose register file) full." + }, + { + "EventCode": "0x015a", + "EventName": "VEC_DISP_STALL", + "PublicDescription": "Rename stalled due to VRF (Vector register file) full." + }, + { + "EventCode": "0x015c", + "EventName": "SX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, SX." + }, + { + "EventCode": "0x015d", + "EventName": "MX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, MX." + }, + { + "EventCode": "0x015e", + "EventName": "LS_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, LS." + }, + { + "EventCode": "0x015f", + "EventName": "VX_IQ_STALL", + "PublicDescription": "Dispatch stalled due to IQ full, VX." + }, + { + "EventCode": "0x0160", + "EventName": "MCQ_FULL_STALL", + "PublicDescription": "Dispatch stalled due to MCQ full." + }, + { + "EventCode": "0x01cf", + "EventName": "PRD_DISP_STALL", + "PublicDescription": "Rename stalled due to predicate registers (physical) are full." + }, + { + "EventCode": "0x01e0", + "EventName": "CSDB_STALL", + "PublicDescription": "Rename stalled due to CSDB." + }, + { + "EventCode": "0x01e2", + "EventName": "STALL_SLOT_FRONTEND_WITHOUT_MISPRED", + "PublicDescription": "Stall slot frontend during non-mispredicted branch.\nThis event counts the STALL_STOT_FRONTEND Events, except for the 4 cycles following a mispredicted branch Event or 4 cycles following a commit flush&restart Event." + } +] diff --git a/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json b/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json new file mode 100644 index 0000000000000..18ec5c348c873 --- /dev/null +++ b/tools/perf/pmu-events/arch/arm64/nvidia/t410/tlb.json @@ -0,0 +1,158 @@ +[ + { + "ArchStdEvent": "L1I_TLB_REFILL", + "PublicDescription": "This event counts L1 Instruction TLB refills from any instruction fetch (demand, hardware prefetch, and software preload accesses). If there are multiple misses in the TLB that are resolved by the refill, then this event only counts once. This event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL", + "PublicDescription": "This event counts L1 Data TLB accesses that resulted in TLB refills. If there are multiple misses in the TLB that are resolved by the refill, then this event only counts once. This event counts for refills caused by preload instructions or hardware prefetch accesses. This event counts regardless of whether the miss hits in L2 or results in a translation table walk. This event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This event will not count on an access from an AT (Address Translation) instruction.\nThis event counts the sum of the following events:\nL1D_TLB_REFILL_RD and\nL1D_TLB_REFILL_WR." + }, + { + "ArchStdEvent": "L1D_TLB", + "PublicDescription": "This event counts L1 Data TLB accesses caused by any memory Load or Store operation.\nNote that Load or Store instructions can be broken up into multiple memory operations.\nThis event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB", + "PublicDescription": "This event counts L1 instruction TLB accesses (caused by demand or hardware prefetch or software preload accesses), whether the access hits or misses in the TLB. This event counts both demand accesses and prefetch or preload generated accesses.\nThis event is a superset of the L1I_TLB_REFILL event." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL", + "PublicDescription": "This event counts L2 TLB refills caused by memory operations from both data and instruction fetch, except for those caused by TLB maintenance operations and hardware prefetches.\nThis event is the sum of the following events:\nL2D_TLB_REFILL_RD and\nL2D_TLB_REFILL_WR." + }, + { + "ArchStdEvent": "L2D_TLB", + "PublicDescription": "This event counts L2 TLB accesses except those caused by TLB maintenance operations.\nThis event is the sum of the following events:\nL2D_TLB_RD and\nL2D_TLB_WR." + }, + { + "ArchStdEvent": "DTLB_WALK", + "PublicDescription": "This event counts number of demand data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this event does not count walks triggered by TLB maintenance operations.\nThis event does not include prefetches." + }, + { + "ArchStdEvent": "ITLB_WALK", + "PublicDescription": "This event counts number of instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that partial translations that cause a translation table walk are also counted. Also note that this event does not count walks triggered by TLB maintenance operations.\nThis event does not include prefetches." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL_RD", + "PublicDescription": "This event counts L1 Data TLB refills caused by memory Read operations. If there are multiple misses in the TLB that are resolved by the refill, then this event only counts once. This event counts for refills caused by preload instructions or hardware prefetch accesses. This event counts regardless of whether the miss hits in L2 or results in a translation table walk. This event will not count if the translation table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This event will not count on an access from an Address Translation (AT) instruction.\nThis event is a subset of the L1D_TLB_REFILL event." + }, + { + "ArchStdEvent": "L1D_TLB_REFILL_WR", + "PublicDescription": "This event counts L1 Data TLB refills caused by data side memory Write operations. If there are multiple misses in the TLB that are resolved by the refill, then this event only counts once. This event counts for refills caused by preload instructions or hardware prefetch accesses. This event counts regardless of whether the miss hits in L2 or results in a translation table walk. This event will not count if the table walk results in a fault (such as a translation or access fault), since there is no new translation created for the TLB. This event will not count with an access from an Address Translation (AT) instruction.\nThis event is a subset of the L1D_TLB_REFILL event." + }, + { + "ArchStdEvent": "L1D_TLB_RD", + "PublicDescription": "This event counts L1 Data TLB accesses caused by memory Read operations. This event counts whether the access hits or misses in the TLB. This event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1D_TLB_WR", + "PublicDescription": "This event counts any L1 Data side TLB accesses caused by memory Write operations. This event counts whether the access hits or misses in the TLB. This event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL_RD", + "PublicDescription": "This event counts L2 TLB refills caused by memory Read operations from both data and instruction fetch except for those caused by TLB maintenance operations or hardware prefetches.\nThis event is a subset of the L2D_TLB_REFILL event." + }, + { + "ArchStdEvent": "L2D_TLB_REFILL_WR", + "PublicDescription": "This event counts L2 TLB refills caused by memory Write operations from both data and instruction fetch except for those caused by TLB maintenance operations.\nThis event is a subset of the L2D_TLB_REFILL event." + }, + { + "ArchStdEvent": "L2D_TLB_RD", + "PublicDescription": "This event counts L2 TLB accesses caused by memory Read operations from both data and instruction fetch except for those caused by TLB maintenance operations.\nThis event is a subset of the L2D_TLB event." + }, + { + "ArchStdEvent": "L2D_TLB_WR", + "PublicDescription": "This event counts L2 TLB accesses caused by memory Write operations from both data and instruction fetch except for those caused by TLB maintenance operations.\nThis event is a subset of the L2D_TLB event." + }, + { + "ArchStdEvent": "DTLB_WALK_PERCYC", + "PublicDescription": "This event counts the number of data translation table walks in progress per cycle." + }, + { + "ArchStdEvent": "ITLB_WALK_PERCYC", + "PublicDescription": "This event counts the number of instruction translation table walks in progress per cycle." + }, + { + "ArchStdEvent": "L1D_TLB_RW", + "PublicDescription": "This event counts L1 Data TLB demand accesses caused by memory Read or Write operations. This event counts whether the access hits or misses in the TLB. This event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB_RD", + "PublicDescription": "This event counts L1 Instruction TLB demand accesses whether the access hits or misses in the TLB." + }, + { + "ArchStdEvent": "L1D_TLB_PRFM", + "PublicDescription": "This event counts L1 Data TLB accesses generated by software prefetch or preload memory accesses. Load or Store instructions can be broken into multiple memory operations. This event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "L1I_TLB_PRFM", + "PublicDescription": "This event counts L1 Instruction TLB accesses generated by software preload or prefetch instructions. This event counts whether the access hits or misses in the TLB. This event does not count TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_HWUPD", + "PublicDescription": "This event counts number of memory accesses triggered by a data translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD. Note that this event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." + }, + { + "ArchStdEvent": "ITLB_HWUPD", + "PublicDescription": "This event counts number of memory accesses triggered by an instruction translation table walk and performing an update of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." + }, + { + "ArchStdEvent": "DTLB_STEP", + "PublicDescription": "This event counts number of memory accesses triggered by a demand data translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD.\nNote that this event counts accesses triggered by software preloads, but not accesses triggered by hardware prefetchers." + }, + { + "ArchStdEvent": "ITLB_STEP", + "PublicDescription": "This event counts number of memory accesses triggered by an instruction translation table walk and performing a Read of a translation table entry. Memory accesses are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD." + }, + { + "ArchStdEvent": "DTLB_WALK_LARGE", + "PublicDescription": "This event counts number of demand data translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_BLOCK is implemented, then it is an alias for this event in this family.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_LARGE", + "PublicDescription": "This event counts number of instruction translation table walks caused by a miss in the L2 TLB and yielding a large page. The set of large pages is defined as all pages with a final size higher than or equal to 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_BLOCK event.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_SMALL", + "PublicDescription": "This event counts number of data translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. If DTLB_WALK_PAGE event is implemented, then it is an alias for this event in this family. Note that partial translations that cause a translation table walk are also counted.\nAlso note that this event counts walks triggered by software preloads, but not walks triggered by hardware prefetchers, and that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_SMALL", + "PublicDescription": "This event counts number of instruction translation table walks caused by a miss in the L2 TLB and yielding a small page. The set of small pages is defined as all pages with a final size lower than 2MB. Translation table walks that end up taking a translation fault are not counted, as the page size would be undefined in that case. In this family, this is equal to ITLB_WALK_PAGE event.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_RW", + "PublicDescription": "This event counts number of demand data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_RD", + "PublicDescription": "This event counts number of demand instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "DTLB_WALK_PRFM", + "PublicDescription": "This event counts number of software prefetches or preloads generated data translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "ArchStdEvent": "ITLB_WALK_PRFM", + "PublicDescription": "This event counts number of software prefetches or preloads generated instruction translation table walks caused by a miss in the L2 TLB and performing at least one memory access. Translation table walks are counted even if the translation ended up taking a translation fault for reasons different than EPD, E0PD and NFD.\nNote that partial translations that cause a translation table walk are also counted.\nAlso note that this event does not count walks triggered by TLB maintenance operations." + }, + { + "EventCode": "0x010e", + "EventName": "L1D_TLB_REFILL_RD_PF", + "PublicDescription": "L1 Data TLB refill, Read, prefetch." + }, + { + "EventCode": "0x010f", + "EventName": "L2TLB_PF_REFILL", + "PublicDescription": "L2 Data TLB refill, Read, prefetch.\nThis event counts MMU refills due to internal PFStream requests." + }, + { + "EventCode": "0x0223", + "EventName": "L1I_TLB_REFILL_RD", + "PublicDescription": "L1 Instruction TLB refills due to Demand miss." + }, + { + "EventCode": "0x0224", + "EventName": "L1I_TLB_REFILL_PRFM", + "PublicDescription": "L1 Instruction TLB refills due to Software prefetch miss." + } +] From cffe4afd59f3b905bba73d61e357d5bf9824e294 Mon Sep 17 00:00:00 2001 From: Besar Wicaksono Date: Mon, 4 May 2026 17:52:04 +0000 Subject: [PATCH 204/464] NVIDIA: VR: SAUCE: perf/arm_pmu: Skip PMCCNTR_EL0 on NVIDIA Olympus BugLink: https://bugs.launchpad.net/bugs/2149756 PMCCNTR_EL0 may continue to increment on NVIDIA Olympus CPUs while the PE is in WFI/WFE. That does not necessarily match the CPU_CYCLES event counted by a programmable counter, so using PMCCNTR_EL0 for cycles can give results that differ from the programmable counter path. Extend the existing PMCCNTR avoidance decision from the SMT case to also cover Olympus. Store the result in the common arm_pmu state at registration time, so arm_pmuv3 can keep using a single flag when deciding whether CPU_CYCLES may use PMCCNTR_EL0. Signed-off-by: Besar Wicaksono (backported from https://lore.kernel.org/all/20260504175204.3122979-1-bwicaksono@nvidia.com/) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/perf/arm_pmu.c | 7 ++++- drivers/perf/arm_pmuv3.c | 51 +++++++++++++++++++++++++++++++----- include/linux/perf/arm_pmu.h | 2 +- 3 files changed, 51 insertions(+), 9 deletions(-) diff --git a/drivers/perf/arm_pmu.c b/drivers/perf/arm_pmu.c index 939bcbd433aab..aa1dac0b440fa 100644 --- a/drivers/perf/arm_pmu.c +++ b/drivers/perf/arm_pmu.c @@ -931,8 +931,13 @@ int armpmu_register(struct arm_pmu *pmu) /* * By this stage we know our supported CPUs on either DT/ACPI platforms, * detect the SMT implementation. + * On SMT CPUs, the PMCCNTR_EL0 increments from the processor clock rather + * than the PE clock (ARM DDI0487 L.b D13.1.3) which means it'll continue + * counting on a WFI PE if one of its SMT sibling is not idle on a + * multi-threaded implementation. So don't use it on SMT cores. */ - pmu->has_smt = topology_core_has_smt(cpumask_first(&pmu->supported_cpus)); + pmu->avoid_pmccntr |= + topology_core_has_smt(cpumask_first(&pmu->supported_cpus)); if (!pmu->set_event_filter) pmu->pmu.capabilities |= PERF_PMU_CAP_NO_EXCLUDE; diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 8014ff766cff5..1ee4a09d0dccb 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -8,6 +8,7 @@ * This code is based heavily on the ARMv7 perf event code. */ +#include #include #include #include @@ -1002,13 +1003,7 @@ static bool armv8pmu_can_use_pmccntr(struct pmu_hw_events *cpuc, if (has_branch_stack(event)) return false; - /* - * The PMCCNTR_EL0 increments from the processor clock rather than - * the PE clock (ARM DDI0487 L.b D13.1.3) which means it'll continue - * counting on a WFI PE if one of its SMT sibling is not idle on a - * multi-threaded implementation. So don't use it on SMT cores. - */ - if (cpu_pmu->has_smt) + if (cpu_pmu->avoid_pmccntr) return false; return true; @@ -1299,6 +1294,41 @@ static int armv8_vulcan_map_event(struct perf_event *event) &armv8_vulcan_perf_cache_map); } +#ifdef CONFIG_ARM64 +/* + * List of CPUs that should avoid using PMCCNTR_EL0. + */ +static struct midr_range armv8pmu_avoid_pmccntr_cpus[] = { + /* + * The PMCCNTR_EL0 in Olympus CPU may still increment while in WFI/WFE state. + * This is an implementation specific behavior and not an erratum. + * + * From ARM DDI0487 D14.4: + * It is IMPLEMENTATION SPECIFIC whether CPU_CYCLES and PMCCNTR count + * when the PE is in WFI or WFE state, even if the clocks are not stopped. + * + * From ARM DDI0487 D24.5.2: + * All counters are subject to any changes in clock frequency, including + * clock stopping caused by the WFI and WFE instructions. + * This means that it is CONSTRAINED UNPREDICTABLE whether or not + * PMCCNTR_EL0 continues to increment when clocks are stopped by WFI and + * WFE instructions. + */ + MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS), + {} +}; + +static bool armv8pmu_is_in_avoid_pmccntr_cpus(void) +{ + return is_midr_in_range_list(armv8pmu_avoid_pmccntr_cpus); +} +#else +static bool armv8pmu_is_in_avoid_pmccntr_cpus(void) +{ + return false; +} +#endif + struct armv8pmu_probe_info { struct arm_pmu *pmu; bool present; @@ -1348,6 +1378,13 @@ static void __armv8pmu_probe_pmu(void *info) else cpu_pmu->reg_pmmir = 0; + /* + * On some CPUs, PMCCNTR_EL0 does not match the behavior of CPU_CYCLES + * programmable counter, so avoid routing cycles through PMCCNTR_EL0 to + * prevent inconsistency in the results. + */ + cpu_pmu->avoid_pmccntr |= armv8pmu_is_in_avoid_pmccntr_cpus(); + brbe_probe(cpu_pmu); } diff --git a/include/linux/perf/arm_pmu.h b/include/linux/perf/arm_pmu.h index 52b37f7bdbf9e..02d2c7f45b527 100644 --- a/include/linux/perf/arm_pmu.h +++ b/include/linux/perf/arm_pmu.h @@ -119,7 +119,7 @@ struct arm_pmu { /* PMUv3 only */ int pmuver; - bool has_smt; + bool avoid_pmccntr; u64 reg_pmmir; u64 reg_brbidr; #define ARMV8_PMUV3_MAX_COMMON_EVENTS 0x40 From 357a5b3361e6cd8a1ba30e0db733d22fe913a726 Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:52 +0530 Subject: [PATCH 205/464] ACPI: CPPC: Add cppc_get_perf() API to read performance controls BugLink: https://bugs.launchpad.net/bugs/2131705 Add cppc_get_perf() function to read values of performance control registers including desired_perf, min_perf, max_perf, energy_perf, and auto_sel. This provides a read interface to complement the existing cppc_set_perf() write interface for performance control registers. Note that auto_sel is read by cppc_get_perf() but not written by cppc_set_perf() to avoid unintended mode changes during performance updates. It can be updated with existing dedicated cppc_set_auto_sel() API. Use cppc_get_perf() in cppc_cpufreq_get_cpu_data() to initialize perf_ctrls with current hardware register values during cpufreq policy initialization. Signed-off-by: Sumit Gupta Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-2-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 658fa7b1c47a857af484c5c5dff8d0164b7c7bfb) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 80 ++++++++++++++++++++++++++++++++++ drivers/cpufreq/cppc_cpufreq.c | 6 +++ include/acpi/cppc_acpi.h | 5 +++ 3 files changed, 91 insertions(+) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index bcfe2e6b8445f..6095e794be4c5 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -1738,6 +1738,86 @@ int cppc_set_enable(int cpu, bool enable) } EXPORT_SYMBOL_GPL(cppc_set_enable); +/** + * cppc_get_perf - Get a CPU's performance controls. + * @cpu: CPU for which to get performance controls. + * @perf_ctrls: ptr to cppc_perf_ctrls. See cppc_acpi.h + * + * Return: 0 for success with perf_ctrls, -ERRNO otherwise. + */ +int cppc_get_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) +{ + struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpu); + struct cpc_register_resource *desired_perf_reg, + *min_perf_reg, *max_perf_reg, + *energy_perf_reg, *auto_sel_reg; + u64 desired_perf = 0, min = 0, max = 0, energy_perf = 0, auto_sel = 0; + int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpu); + struct cppc_pcc_data *pcc_ss_data = NULL; + int ret = 0, regs_in_pcc = 0; + + if (!cpc_desc) { + pr_debug("No CPC descriptor for CPU:%d\n", cpu); + return -ENODEV; + } + + if (!perf_ctrls) { + pr_debug("Invalid perf_ctrls pointer\n"); + return -EINVAL; + } + + desired_perf_reg = &cpc_desc->cpc_regs[DESIRED_PERF]; + min_perf_reg = &cpc_desc->cpc_regs[MIN_PERF]; + max_perf_reg = &cpc_desc->cpc_regs[MAX_PERF]; + energy_perf_reg = &cpc_desc->cpc_regs[ENERGY_PERF]; + auto_sel_reg = &cpc_desc->cpc_regs[AUTO_SEL_ENABLE]; + + /* Are any of the regs PCC ?*/ + if (CPC_IN_PCC(desired_perf_reg) || CPC_IN_PCC(min_perf_reg) || + CPC_IN_PCC(max_perf_reg) || CPC_IN_PCC(energy_perf_reg) || + CPC_IN_PCC(auto_sel_reg)) { + if (pcc_ss_id < 0) { + pr_debug("Invalid pcc_ss_id for CPU:%d\n", cpu); + return -ENODEV; + } + pcc_ss_data = pcc_data[pcc_ss_id]; + regs_in_pcc = 1; + down_write(&pcc_ss_data->pcc_lock); + /* Ring doorbell once to update PCC subspace */ + if (send_pcc_cmd(pcc_ss_id, CMD_READ) < 0) { + ret = -EIO; + goto out_err; + } + } + + /* Read optional elements if present */ + if (CPC_SUPPORTED(max_perf_reg)) + cpc_read(cpu, max_perf_reg, &max); + perf_ctrls->max_perf = max; + + if (CPC_SUPPORTED(min_perf_reg)) + cpc_read(cpu, min_perf_reg, &min); + perf_ctrls->min_perf = min; + + if (CPC_SUPPORTED(desired_perf_reg)) + cpc_read(cpu, desired_perf_reg, &desired_perf); + perf_ctrls->desired_perf = desired_perf; + + if (CPC_SUPPORTED(energy_perf_reg)) + cpc_read(cpu, energy_perf_reg, &energy_perf); + perf_ctrls->energy_perf = energy_perf; + + if (CPC_SUPPORTED(auto_sel_reg)) + cpc_read(cpu, auto_sel_reg, &auto_sel); + perf_ctrls->auto_sel = (bool)auto_sel; + +out_err: + if (regs_in_pcc) + up_write(&pcc_ss_data->pcc_lock); + return ret; +} +EXPORT_SYMBOL_GPL(cppc_get_perf); + /** * cppc_set_perf - Set a CPU's performance controls. * @cpu: CPU for which to set performance controls. diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 011f35cb47b94..a61a24e0dcaed 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -594,6 +594,12 @@ static struct cppc_cpudata *cppc_cpufreq_get_cpu_data(unsigned int cpu) goto free_mask; } + ret = cppc_get_perf(cpu, &cpu_data->perf_ctrls); + if (ret) { + pr_debug("Err reading CPU%d perf ctrls: ret:%d\n", cpu, ret); + goto free_mask; + } + return cpu_data; free_mask: diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h index 4d644f03098e3..3fc796c0d9022 100644 --- a/include/acpi/cppc_acpi.h +++ b/include/acpi/cppc_acpi.h @@ -151,6 +151,7 @@ extern int cppc_get_desired_perf(int cpunum, u64 *desired_perf); extern int cppc_get_nominal_perf(int cpunum, u64 *nominal_perf); extern int cppc_get_highest_perf(int cpunum, u64 *highest_perf); extern int cppc_get_perf_ctrs(int cpu, struct cppc_perf_fb_ctrs *perf_fb_ctrs); +extern int cppc_get_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls); extern int cppc_set_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls); extern int cppc_set_enable(int cpu, bool enable); extern int cppc_get_perf_caps(int cpu, struct cppc_perf_caps *caps); @@ -193,6 +194,10 @@ static inline int cppc_get_perf_ctrs(int cpu, struct cppc_perf_fb_ctrs *perf_fb_ { return -EOPNOTSUPP; } +static inline int cppc_get_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) +{ + return -EOPNOTSUPP; +} static inline int cppc_set_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) { return -EOPNOTSUPP; From 496afc9284b019659e1fffdb9cf25061042fada1 Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:53 +0530 Subject: [PATCH 206/464] ACPI: CPPC: Warn on missing mandatory DESIRED_PERF register BugLink: https://bugs.launchpad.net/bugs/2131705 Add a warning during CPPC processor probe if the Desired Performance register is not supported when it should be. As per 8.4.6.1.2.3 section of ACPI 6.6 specification, "The Desired Performance Register is optional only when OSPM indicates support for CPPC2 in the platform-wide _OSC capabilities and the Autonomous Selection Enable field is encoded as an Integer with a value of 1." In other words: - In CPPC v1, DESIRED_PERF is mandatory - In CPPC v2, it becomes optional only when AUTO_SEL_ENABLE is supported This helps detect firmware configuration issues early during boot. Link: https://lore.kernel.org/lkml/9fa21599-004a-4af8-acc2-190fd0404e35@nvidia.com/ Suggested-by: Pierre Gondois Signed-off-by: Sumit Gupta Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-3-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit b3e45fb2db9d8a733e94b315f1272e2c4468ed4b) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 6095e794be4c5..1b7c10392566e 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -853,6 +853,16 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr) } per_cpu(cpu_pcc_subspace_idx, pr->id) = pcc_subspace_id; + /* + * In CPPC v1, DESIRED_PERF is mandatory. In CPPC v2, it is optional + * only when AUTO_SEL_ENABLE is supported. + */ + if (!CPC_SUPPORTED(&cpc_ptr->cpc_regs[DESIRED_PERF]) && + (!osc_sb_cppc2_support_acked || + !CPC_SUPPORTED(&cpc_ptr->cpc_regs[AUTO_SEL_ENABLE]))) + pr_warn("Desired perf. register is mandatory if CPPC v2 is not supported " + "or autonomous selection is disabled\n"); + /* * Initialize the remaining cpc_regs as unsupported. * Example: In case FW exposes CPPC v2, the below loop will initialize From 19ef1cab10baac670c9238dd46037135e9ef16de Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:54 +0530 Subject: [PATCH 207/464] ACPI: CPPC: Extend cppc_set_epp_perf() for FFH/SystemMemory BugLink: https://bugs.launchpad.net/bugs/2131705 Extend cppc_set_epp_perf() to write both auto_sel and energy_perf registers when they are in FFH or SystemMemory address space. This keeps the behavior consistent with PCC case where both registers are already updated together, but was missing for FFH/SystemMemory. Signed-off-by: Sumit Gupta Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-4-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 38428a680026c52a1fc64212325d161974c3e4cf) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 1b7c10392566e..3b9bcc4ae18c3 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -1571,6 +1571,8 @@ int cppc_set_epp_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls, bool enable) struct cpc_register_resource *auto_sel_reg; struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpu); struct cppc_pcc_data *pcc_ss_data = NULL; + bool autosel_ffh_sysmem; + bool epp_ffh_sysmem; int ret; if (!cpc_desc) { @@ -1581,6 +1583,11 @@ int cppc_set_epp_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls, bool enable) auto_sel_reg = &cpc_desc->cpc_regs[AUTO_SEL_ENABLE]; epp_set_reg = &cpc_desc->cpc_regs[ENERGY_PERF]; + epp_ffh_sysmem = CPC_SUPPORTED(epp_set_reg) && + (CPC_IN_FFH(epp_set_reg) || CPC_IN_SYSTEM_MEMORY(epp_set_reg)); + autosel_ffh_sysmem = CPC_SUPPORTED(auto_sel_reg) && + (CPC_IN_FFH(auto_sel_reg) || CPC_IN_SYSTEM_MEMORY(auto_sel_reg)); + if (CPC_IN_PCC(epp_set_reg) || CPC_IN_PCC(auto_sel_reg)) { if (pcc_ss_id < 0) { pr_debug("Invalid pcc_ss_id for CPU:%d\n", cpu); @@ -1606,11 +1613,22 @@ int cppc_set_epp_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls, bool enable) ret = send_pcc_cmd(pcc_ss_id, CMD_WRITE); up_write(&pcc_ss_data->pcc_lock); } else if (osc_cpc_flexible_adr_space_confirmed && - CPC_SUPPORTED(epp_set_reg) && CPC_IN_FFH(epp_set_reg)) { - ret = cpc_write(cpu, epp_set_reg, perf_ctrls->energy_perf); + (epp_ffh_sysmem || autosel_ffh_sysmem)) { + if (autosel_ffh_sysmem) { + ret = cpc_write(cpu, auto_sel_reg, enable); + if (ret) + return ret; + } + + if (epp_ffh_sysmem) { + ret = cpc_write(cpu, epp_set_reg, + perf_ctrls->energy_perf); + if (ret) + return ret; + } } else { ret = -ENOTSUPP; - pr_debug("_CPC in PCC and _CPC in FFH are not supported\n"); + pr_debug("_CPC in PCC/FFH/SystemMemory are not supported\n"); } return ret; From 367f9d5c98d3df915cee52f5de32951ba2378628 Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:55 +0530 Subject: [PATCH 208/464] cpufreq: CPPC: Update cached perf_ctrls on sysfs write BugLink: https://bugs.launchpad.net/bugs/2131705 Update the cached perf_ctrls values when writing via sysfs to keep them in sync with hardware registers: - store_auto_select(): update perf_ctrls.auto_sel - store_energy_performance_preference_val(): update perf_ctrls.energy_perf This ensures consistent cached values after sysfs writes, which complements the cppc_get_perf() initialization during policy setup. Signed-off-by: Sumit Gupta Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-5-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 24ad4c6c136bdaa4c92c5c5948856752ce3e9f76) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cppc_cpufreq.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index a61a24e0dcaed..ebb5746df220e 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -855,6 +855,7 @@ static ssize_t show_auto_select(struct cpufreq_policy *policy, char *buf) static ssize_t store_auto_select(struct cpufreq_policy *policy, const char *buf, size_t count) { + struct cppc_cpudata *cpu_data = policy->driver_data; bool val; int ret; @@ -866,6 +867,8 @@ static ssize_t store_auto_select(struct cpufreq_policy *policy, if (ret) return ret; + cpu_data->perf_ctrls.auto_sel = val; + return count; } @@ -916,8 +919,32 @@ static ssize_t store_##_name(struct cpufreq_policy *policy, \ CPPC_CPUFREQ_ATTR_RW_U64(auto_act_window, cppc_get_auto_act_window, cppc_set_auto_act_window) -CPPC_CPUFREQ_ATTR_RW_U64(energy_performance_preference_val, - cppc_get_epp_perf, cppc_set_epp) +static ssize_t +show_energy_performance_preference_val(struct cpufreq_policy *policy, char *buf) +{ + return cppc_cpufreq_sysfs_show_u64(policy->cpu, cppc_get_epp_perf, buf); +} + +static ssize_t +store_energy_performance_preference_val(struct cpufreq_policy *policy, + const char *buf, size_t count) +{ + struct cppc_cpudata *cpu_data = policy->driver_data; + u64 val; + int ret; + + ret = kstrtou64(buf, 0, &val); + if (ret) + return ret; + + ret = cppc_set_epp(policy->cpu, val); + if (ret) + return ret; + + cpu_data->perf_ctrls.energy_perf = val; + + return count; +} cpufreq_freq_attr_ro(freqdomain_cpus); cpufreq_freq_attr_rw(auto_select); From 5d6c182402fbf52aae8fbf4890a34239b18f6f93 Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:56 +0530 Subject: [PATCH 209/464] cpufreq: cppc: Update MIN_PERF/MAX_PERF in target callbacks BugLink: https://bugs.launchpad.net/bugs/2131705 Update MIN_PERF and MAX_PERF registers from policy->min and policy->max in the .target() and .fast_switch() callbacks. This allows controlling performance bounds via standard scaling_min_freq and scaling_max_freq sysfs interfaces. Similar to intel_cpufreq which updates HWP min/max limits in .target(), cppc_cpufreq now programs MIN_PERF/MAX_PERF along with DESIRED_PERF. Since MIN_PERF/MAX_PERF can be updated even when auto_sel is disabled, they are updated unconditionally. Also program MIN_PERF/MAX_PERF in store_auto_select() when enabling autonomous selection so the platform uses correct bounds immediately. Suggested-by: Rafael J. Wysocki Signed-off-by: Sumit Gupta Link: https://patch.msgid.link/20260206142658.72583-6-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit ea3db45ae476889a1ba0ab3617e6afdeeefbda3d) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cppc_cpufreq.c | 41 +++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index ebb5746df220e..8a8cf76828ee2 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -287,6 +287,21 @@ static inline void cppc_freq_invariance_exit(void) } #endif /* CONFIG_ACPI_CPPC_CPUFREQ_FIE */ +static void cppc_cpufreq_update_perf_limits(struct cppc_cpudata *cpu_data, + struct cpufreq_policy *policy) +{ + struct cppc_perf_caps *caps = &cpu_data->perf_caps; + u32 min_perf, max_perf; + + min_perf = cppc_khz_to_perf(caps, policy->min); + max_perf = cppc_khz_to_perf(caps, policy->max); + + cpu_data->perf_ctrls.min_perf = + clamp_t(u32, min_perf, caps->lowest_perf, caps->highest_perf); + cpu_data->perf_ctrls.max_perf = + clamp_t(u32, max_perf, caps->lowest_perf, caps->highest_perf); +} + static int cppc_cpufreq_set_target(struct cpufreq_policy *policy, unsigned int target_freq, unsigned int relation) @@ -298,6 +313,8 @@ static int cppc_cpufreq_set_target(struct cpufreq_policy *policy, cpu_data->perf_ctrls.desired_perf = cppc_khz_to_perf(&cpu_data->perf_caps, target_freq); + cppc_cpufreq_update_perf_limits(cpu_data, policy); + freqs.old = policy->cur; freqs.new = target_freq; @@ -322,8 +339,9 @@ static unsigned int cppc_cpufreq_fast_switch(struct cpufreq_policy *policy, desired_perf = cppc_khz_to_perf(&cpu_data->perf_caps, target_freq); cpu_data->perf_ctrls.desired_perf = desired_perf; - ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); + cppc_cpufreq_update_perf_limits(cpu_data, policy); + ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); if (ret) { pr_debug("Failed to set target on CPU:%d. ret:%d\n", cpu, ret); @@ -869,6 +887,27 @@ static ssize_t store_auto_select(struct cpufreq_policy *policy, cpu_data->perf_ctrls.auto_sel = val; + if (val) { + u32 old_min_perf = cpu_data->perf_ctrls.min_perf; + u32 old_max_perf = cpu_data->perf_ctrls.max_perf; + + /* + * When enabling autonomous selection, program MIN_PERF and + * MAX_PERF from current policy limits so that the platform + * uses the correct performance bounds immediately. + */ + cppc_cpufreq_update_perf_limits(cpu_data, policy); + + ret = cppc_set_perf(policy->cpu, &cpu_data->perf_ctrls); + if (ret) { + cpu_data->perf_ctrls.min_perf = old_min_perf; + cpu_data->perf_ctrls.max_perf = old_max_perf; + cppc_set_auto_sel(policy->cpu, false); + cpu_data->perf_ctrls.auto_sel = false; + return ret; + } + } + return count; } From 8a7dc28bbaaf36c999f1a1283ae2465b09f3942b Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:57 +0530 Subject: [PATCH 210/464] ACPI: CPPC: add APIs and sysfs interface for perf_limited BugLink: https://bugs.launchpad.net/bugs/2131705 Add sysfs interface to read/write the Performance Limited register. The Performance Limited register indicates to the OS that an unpredictable event (like thermal throttling) has limited processor performance. It contains two sticky bits set by the platform: - Bit 0 (Desired_Excursion): Set when delivered performance is constrained below desired performance. Not used when Autonomous Selection is enabled. - Bit 1 (Minimum_Excursion): Set when delivered performance is constrained below minimum performance. These bits remain set until OSPM explicitly clears them. The write operation accepts a bitmask of bits to clear: - Write 0x1 to clear bit 0 - Write 0x2 to clear bit 1 - Write 0x3 to clear both bits This enables users to detect if platform throttling impacted a workload. Users clear the register before execution, run the workload, then check afterward - if set, hardware throttling occurred during that time window. The interface is exposed as: /sys/devices/system/cpu/cpuX/cpufreq/perf_limited Signed-off-by: Sumit Gupta Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-7-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 13c45a26635fa51a68911aa57e6778bdad18b103) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 56 ++++++++++++++++++++++++++++++++++ drivers/cpufreq/cppc_cpufreq.c | 5 +++ include/acpi/cppc_acpi.h | 15 +++++++++ 3 files changed, 76 insertions(+) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 3b9bcc4ae18c3..1306aaa03d2d0 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -1978,6 +1978,62 @@ int cppc_set_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) } EXPORT_SYMBOL_GPL(cppc_set_perf); +/** + * cppc_get_perf_limited - Get the Performance Limited register value. + * @cpu: CPU from which to get Performance Limited register. + * @perf_limited: Pointer to store the Performance Limited value. + * + * The returned value contains sticky status bits indicating platform-imposed + * performance limitations. + * + * Return: 0 for success, -EIO on failure, -EOPNOTSUPP if not supported. + */ +int cppc_get_perf_limited(int cpu, u64 *perf_limited) +{ + return cppc_get_reg_val(cpu, PERF_LIMITED, perf_limited); +} +EXPORT_SYMBOL_GPL(cppc_get_perf_limited); + +/** + * cppc_set_perf_limited() - Clear bits in the Performance Limited register. + * @cpu: CPU on which to write register. + * @bits_to_clear: Bitmask of bits to clear in the perf_limited register. + * + * The Performance Limited register contains two sticky bits set by platform: + * - Bit 0 (Desired_Excursion): Set when delivered performance is constrained + * below desired performance. Not used when Autonomous Selection is enabled. + * - Bit 1 (Minimum_Excursion): Set when delivered performance is constrained + * below minimum performance. + * + * These bits are sticky and remain set until OSPM explicitly clears them. + * This function only allows clearing bits (the platform sets them). + * + * Return: 0 for success, -EINVAL for invalid bits, -EIO on register + * access failure, -EOPNOTSUPP if not supported. + */ +int cppc_set_perf_limited(int cpu, u64 bits_to_clear) +{ + u64 current_val, new_val; + int ret; + + /* Only bits 0 and 1 are valid */ + if (bits_to_clear & ~CPPC_PERF_LIMITED_MASK) + return -EINVAL; + + if (!bits_to_clear) + return 0; + + ret = cppc_get_perf_limited(cpu, ¤t_val); + if (ret) + return ret; + + /* Clear the specified bits */ + new_val = current_val & ~bits_to_clear; + + return cppc_set_reg_val(cpu, PERF_LIMITED, new_val); +} +EXPORT_SYMBOL_GPL(cppc_set_perf_limited); + /** * cppc_get_transition_latency - returns frequency transition latency in ns * @cpu_num: CPU number for per_cpu(). diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 8a8cf76828ee2..94d489a4c90d1 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -985,16 +985,21 @@ store_energy_performance_preference_val(struct cpufreq_policy *policy, return count; } +CPPC_CPUFREQ_ATTR_RW_U64(perf_limited, cppc_get_perf_limited, + cppc_set_perf_limited) + cpufreq_freq_attr_ro(freqdomain_cpus); cpufreq_freq_attr_rw(auto_select); cpufreq_freq_attr_rw(auto_act_window); cpufreq_freq_attr_rw(energy_performance_preference_val); +cpufreq_freq_attr_rw(perf_limited); static struct freq_attr *cppc_cpufreq_attr[] = { &freqdomain_cpus, &auto_select, &auto_act_window, &energy_performance_preference_val, + &perf_limited, NULL, }; diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h index 3fc796c0d9022..f7afa20b8ad9d 100644 --- a/include/acpi/cppc_acpi.h +++ b/include/acpi/cppc_acpi.h @@ -42,6 +42,11 @@ #define CPPC_EPP_PERFORMANCE_PREF 0x00 #define CPPC_EPP_ENERGY_EFFICIENCY_PREF 0xFF +#define CPPC_PERF_LIMITED_DESIRED_EXCURSION BIT(0) +#define CPPC_PERF_LIMITED_MINIMUM_EXCURSION BIT(1) +#define CPPC_PERF_LIMITED_MASK (CPPC_PERF_LIMITED_DESIRED_EXCURSION | \ + CPPC_PERF_LIMITED_MINIMUM_EXCURSION) + /* Each register has the folowing format. */ struct cpc_reg { u8 descriptor; @@ -174,6 +179,8 @@ extern int cppc_get_auto_act_window(int cpu, u64 *auto_act_window); extern int cppc_set_auto_act_window(int cpu, u64 auto_act_window); extern int cppc_get_auto_sel(int cpu, bool *enable); extern int cppc_set_auto_sel(int cpu, bool enable); +extern int cppc_get_perf_limited(int cpu, u64 *perf_limited); +extern int cppc_set_perf_limited(int cpu, u64 bits_to_clear); extern int amd_get_highest_perf(unsigned int cpu, u32 *highest_perf); extern int amd_get_boost_ratio_numerator(unsigned int cpu, u64 *numerator); extern int amd_detect_prefcore(bool *detected); @@ -270,6 +277,14 @@ static inline int cppc_set_auto_sel(int cpu, bool enable) { return -EOPNOTSUPP; } +static inline int cppc_get_perf_limited(int cpu, u64 *perf_limited) +{ + return -EOPNOTSUPP; +} +static inline int cppc_set_perf_limited(int cpu, u64 bits_to_clear) +{ + return -EOPNOTSUPP; +} static inline int amd_get_highest_perf(unsigned int cpu, u32 *highest_perf) { return -ENODEV; From 29a08aaf4389b949c867cbb1b3b45ebde29f476e Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Fri, 6 Feb 2026 19:56:58 +0530 Subject: [PATCH 211/464] cpufreq: CPPC: Add sysfs documentation for perf_limited BugLink: https://bugs.launchpad.net/bugs/2131705 Add ABI documentation for the Performance Limited Register sysfs interface in the cppc_cpufreq driver. Signed-off-by: Sumit Gupta Reviewed-by: Randy Dunlap Reviewed-by: Pierre Gondois Reviewed-by: Lifeng Zheng Link: https://patch.msgid.link/20260206142658.72583-8-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 856250ba2e810e772dc95b3234ebf0d6393a51d9) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- .../ABI/testing/sysfs-devices-system-cpu | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu index 3a05604c21bf8..82d10d556cc89 100644 --- a/Documentation/ABI/testing/sysfs-devices-system-cpu +++ b/Documentation/ABI/testing/sysfs-devices-system-cpu @@ -327,6 +327,24 @@ Description: Energy performance preference This file is only present if the cppc-cpufreq driver is in use. +What: /sys/devices/system/cpu/cpuX/cpufreq/perf_limited +Date: February 2026 +Contact: linux-pm@vger.kernel.org +Description: Performance Limited + + Read to check if platform throttling (thermal/power/current + limits) caused delivered performance to fall below the + requested level. A non-zero value indicates throttling occurred. + + Write the bitmask of bits to clear: + + - 0x1 = clear bit 0 (desired performance excursion) + - 0x2 = clear bit 1 (minimum performance excursion) + - 0x3 = clear both bits + + The platform sets these bits; OSPM can only clear them. + + This file is only present if the cppc-cpufreq driver is in use. What: /sys/devices/system/cpu/cpu*/cache/index3/cache_disable_{0,1} Date: August 2008 From d9501b141f102db8fb7e36762a3b102bc5ca4d94 Mon Sep 17 00:00:00 2001 From: Pengjie Zhang Date: Fri, 13 Feb 2026 18:09:35 +0800 Subject: [PATCH 212/464] ACPI: CPPC: Move reference performance to capabilities BugLink: https://bugs.launchpad.net/bugs/2131705 Currently, the `Reference Performance` register is read every time the CPU frequency is sampled in `cppc_get_perf_ctrs()`. This function is on the hot path of the cppc_cpufreq driver. Reference Performance indicates the performance level that corresponds to the Reference Counter incrementing and is not expected to change dynamically during runtime (unlike the Delivered and Reference counters). Reading this register in the hot path incurs unnecessary overhead, particularly on platforms where CPC registers are located in the PCC (Platform Communication Channel) subspace. This patch moves `reference_perf` from the dynamic feedback counters structure (`cppc_perf_fb_ctrs`) to the static capabilities structure (`cppc_perf_caps`). Signed-off-by: Pengjie Zhang [ rjw: Changelog adjustment ] Link: https://patch.msgid.link/20260213100935.19111-1-zhangpengjie2@huawei.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 8505bfb4e4eca28ef1b20d3369435ec2d6a125c6) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 55 +++++++++++++++------------------- drivers/cpufreq/cppc_cpufreq.c | 21 +++++++------ include/acpi/cppc_acpi.h | 2 +- 3 files changed, 37 insertions(+), 41 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 1306aaa03d2d0..58dc9bdd037d3 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -177,12 +177,12 @@ __ATTR(_name, 0444, show_##_name, NULL) show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, highest_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_perf); +show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, reference_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_nonlinear_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, guaranteed_perf); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_freq); show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_freq); -show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, reference_perf); show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, wraparound_time); /* Check for valid access_width, otherwise, fallback to using bit_width */ @@ -1352,9 +1352,10 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) { struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpunum); struct cpc_register_resource *highest_reg, *lowest_reg, - *lowest_non_linear_reg, *nominal_reg, *guaranteed_reg, - *low_freq_reg = NULL, *nom_freq_reg = NULL; - u64 high, low, guaranteed, nom, min_nonlinear, low_f = 0, nom_f = 0; + *lowest_non_linear_reg, *nominal_reg, *reference_reg, + *guaranteed_reg, *low_freq_reg = NULL, *nom_freq_reg = NULL; + u64 high, low, guaranteed, nom, ref, min_nonlinear, + low_f = 0, nom_f = 0; int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpunum); struct cppc_pcc_data *pcc_ss_data = NULL; int ret = 0, regs_in_pcc = 0; @@ -1368,6 +1369,7 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) lowest_reg = &cpc_desc->cpc_regs[LOWEST_PERF]; lowest_non_linear_reg = &cpc_desc->cpc_regs[LOW_NON_LINEAR_PERF]; nominal_reg = &cpc_desc->cpc_regs[NOMINAL_PERF]; + reference_reg = &cpc_desc->cpc_regs[REFERENCE_PERF]; low_freq_reg = &cpc_desc->cpc_regs[LOWEST_FREQ]; nom_freq_reg = &cpc_desc->cpc_regs[NOMINAL_FREQ]; guaranteed_reg = &cpc_desc->cpc_regs[GUARANTEED_PERF]; @@ -1375,6 +1377,7 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) /* Are any of the regs PCC ?*/ if (CPC_IN_PCC(highest_reg) || CPC_IN_PCC(lowest_reg) || CPC_IN_PCC(lowest_non_linear_reg) || CPC_IN_PCC(nominal_reg) || + (CPC_SUPPORTED(reference_reg) && CPC_IN_PCC(reference_reg)) || CPC_IN_PCC(low_freq_reg) || CPC_IN_PCC(nom_freq_reg) || CPC_IN_PCC(guaranteed_reg)) { if (pcc_ss_id < 0) { @@ -1400,6 +1403,17 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) cpc_read(cpunum, nominal_reg, &nom); perf_caps->nominal_perf = nom; + /* + * If reference perf register is not supported then we should + * use the nominal perf value + */ + if (CPC_SUPPORTED(reference_reg)) { + cpc_read(cpunum, reference_reg, &ref); + perf_caps->reference_perf = ref; + } else { + perf_caps->reference_perf = nom; + } + if (guaranteed_reg->type != ACPI_TYPE_BUFFER || IS_NULL_REG(&guaranteed_reg->cpc_entry.reg)) { perf_caps->guaranteed_perf = 0; @@ -1411,7 +1425,7 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) cpc_read(cpunum, lowest_non_linear_reg, &min_nonlinear); perf_caps->lowest_nonlinear_perf = min_nonlinear; - if (!high || !low || !nom || !min_nonlinear) + if (!high || !low || !nom || !ref || !min_nonlinear) ret = -EFAULT; /* Read optional lowest and nominal frequencies if present */ @@ -1441,20 +1455,10 @@ EXPORT_SYMBOL_GPL(cppc_get_perf_caps); bool cppc_perf_ctrs_in_pcc_cpu(unsigned int cpu) { struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpu); - struct cpc_register_resource *ref_perf_reg; - - /* - * If reference perf register is not supported then we should use the - * nominal perf value - */ - ref_perf_reg = &cpc_desc->cpc_regs[REFERENCE_PERF]; - if (!CPC_SUPPORTED(ref_perf_reg)) - ref_perf_reg = &cpc_desc->cpc_regs[NOMINAL_PERF]; return CPC_IN_PCC(&cpc_desc->cpc_regs[DELIVERED_CTR]) || CPC_IN_PCC(&cpc_desc->cpc_regs[REFERENCE_CTR]) || - CPC_IN_PCC(&cpc_desc->cpc_regs[CTR_WRAP_TIME]) || - CPC_IN_PCC(ref_perf_reg); + CPC_IN_PCC(&cpc_desc->cpc_regs[CTR_WRAP_TIME]); } EXPORT_SYMBOL_GPL(cppc_perf_ctrs_in_pcc_cpu); @@ -1491,10 +1495,10 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) { struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpunum); struct cpc_register_resource *delivered_reg, *reference_reg, - *ref_perf_reg, *ctr_wrap_reg; + *ctr_wrap_reg; int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpunum); struct cppc_pcc_data *pcc_ss_data = NULL; - u64 delivered, reference, ref_perf, ctr_wrap_time; + u64 delivered, reference, ctr_wrap_time; int ret = 0, regs_in_pcc = 0; if (!cpc_desc) { @@ -1504,19 +1508,11 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) delivered_reg = &cpc_desc->cpc_regs[DELIVERED_CTR]; reference_reg = &cpc_desc->cpc_regs[REFERENCE_CTR]; - ref_perf_reg = &cpc_desc->cpc_regs[REFERENCE_PERF]; ctr_wrap_reg = &cpc_desc->cpc_regs[CTR_WRAP_TIME]; - /* - * If reference perf register is not supported then we should - * use the nominal perf value - */ - if (!CPC_SUPPORTED(ref_perf_reg)) - ref_perf_reg = &cpc_desc->cpc_regs[NOMINAL_PERF]; - /* Are any of the regs PCC ?*/ if (CPC_IN_PCC(delivered_reg) || CPC_IN_PCC(reference_reg) || - CPC_IN_PCC(ctr_wrap_reg) || CPC_IN_PCC(ref_perf_reg)) { + CPC_IN_PCC(ctr_wrap_reg)) { if (pcc_ss_id < 0) { pr_debug("Invalid pcc_ss_id\n"); return -ENODEV; @@ -1533,8 +1529,6 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) cpc_read(cpunum, delivered_reg, &delivered); cpc_read(cpunum, reference_reg, &reference); - cpc_read(cpunum, ref_perf_reg, &ref_perf); - /* * Per spec, if ctr_wrap_time optional register is unsupported, then the * performance counters are assumed to never wrap during the lifetime of @@ -1544,14 +1538,13 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) if (CPC_SUPPORTED(ctr_wrap_reg)) cpc_read(cpunum, ctr_wrap_reg, &ctr_wrap_time); - if (!delivered || !reference || !ref_perf) { + if (!delivered || !reference) { ret = -EFAULT; goto out_err; } perf_fb_ctrs->delivered = delivered; perf_fb_ctrs->reference = reference; - perf_fb_ctrs->reference_perf = ref_perf; perf_fb_ctrs->wraparound_time = ctr_wrap_time; out_err: if (regs_in_pcc) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 94d489a4c90d1..5dfb109cf1f4e 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -50,7 +50,8 @@ struct cppc_freq_invariance { static DEFINE_PER_CPU(struct cppc_freq_invariance, cppc_freq_inv); static struct kthread_worker *kworker_fie; -static int cppc_perf_from_fbctrs(struct cppc_perf_fb_ctrs *fb_ctrs_t0, +static int cppc_perf_from_fbctrs(u64 reference_perf, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, struct cppc_perf_fb_ctrs *fb_ctrs_t1); /** @@ -70,7 +71,7 @@ static void __cppc_scale_freq_tick(struct cppc_freq_invariance *cppc_fi) struct cppc_perf_fb_ctrs fb_ctrs = {0}; struct cppc_cpudata *cpu_data; unsigned long local_freq_scale; - u64 perf; + u64 perf, ref_perf; cpu_data = cppc_fi->cpu_data; @@ -79,7 +80,9 @@ static void __cppc_scale_freq_tick(struct cppc_freq_invariance *cppc_fi) return; } - perf = cppc_perf_from_fbctrs(&cppc_fi->prev_perf_fb_ctrs, &fb_ctrs); + ref_perf = cpu_data->perf_caps.reference_perf; + perf = cppc_perf_from_fbctrs(ref_perf, + &cppc_fi->prev_perf_fb_ctrs, &fb_ctrs); if (!perf) return; @@ -747,13 +750,11 @@ static inline u64 get_delta(u64 t1, u64 t0) return (u32)t1 - (u32)t0; } -static int cppc_perf_from_fbctrs(struct cppc_perf_fb_ctrs *fb_ctrs_t0, +static int cppc_perf_from_fbctrs(u64 reference_perf, + struct cppc_perf_fb_ctrs *fb_ctrs_t0, struct cppc_perf_fb_ctrs *fb_ctrs_t1) { u64 delta_reference, delta_delivered; - u64 reference_perf; - - reference_perf = fb_ctrs_t0->reference_perf; delta_reference = get_delta(fb_ctrs_t1->reference, fb_ctrs_t0->reference); @@ -790,7 +791,7 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) struct cpufreq_policy *policy __free(put_cpufreq_policy) = cpufreq_cpu_get(cpu); struct cppc_perf_fb_ctrs fb_ctrs_t0 = {0}, fb_ctrs_t1 = {0}; struct cppc_cpudata *cpu_data; - u64 delivered_perf; + u64 delivered_perf, reference_perf; int ret; if (!policy) @@ -807,7 +808,9 @@ static unsigned int cppc_cpufreq_get_rate(unsigned int cpu) return 0; } - delivered_perf = cppc_perf_from_fbctrs(&fb_ctrs_t0, &fb_ctrs_t1); + reference_perf = cpu_data->perf_caps.reference_perf; + delivered_perf = cppc_perf_from_fbctrs(reference_perf, + &fb_ctrs_t0, &fb_ctrs_t1); if (!delivered_perf) goto out_invalid_counters; diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h index f7afa20b8ad9d..d8e405becdc31 100644 --- a/include/acpi/cppc_acpi.h +++ b/include/acpi/cppc_acpi.h @@ -121,6 +121,7 @@ struct cppc_perf_caps { u32 guaranteed_perf; u32 highest_perf; u32 nominal_perf; + u32 reference_perf; u32 lowest_perf; u32 lowest_nonlinear_perf; u32 lowest_freq; @@ -138,7 +139,6 @@ struct cppc_perf_ctrls { struct cppc_perf_fb_ctrs { u64 reference; u64 delivered; - u64 reference_perf; u64 wraparound_time; }; From fcd10b83ab12660cef95ae0df21b1c79b80f58c4 Mon Sep 17 00:00:00 2001 From: Pengjie Zhang Date: Wed, 11 Mar 2026 15:13:34 +0800 Subject: [PATCH 213/464] ACPI: CPPC: Fix uninitialized ref variable in cppc_get_perf_caps() BugLink: https://bugs.launchpad.net/bugs/2131705 Commit 8505bfb4e4ec ("ACPI: CPPC: Move reference performance to capabilities") introduced a logical error when retrieving the reference performance. On platforms lacking the reference performance register, the fallback logic leaves the local 'ref' variable uninitialized (0). This causes the subsequent sanity check to incorrectly return -EFAULT, breaking amd_pstate initialization. Fix this by assigning 'ref = nom' in the fallback path. Fixes: 8505bfb4e4ec ("ACPI: CPPC: Move reference performance to capabilities") Reported-by: Nathan Chancellor Closes: https://lore.kernel.org/all/20260310003026.GA2639793@ax162/ Tested-by: Nathan Chancellor Signed-off-by: Pengjie Zhang [ rjw: Subject tweak ] Link: https://patch.msgid.link/20260311071334.1494960-1-zhangpengjie2@huawei.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit be473f0591f183990a998edee02161b319047eaa) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index 58dc9bdd037d3..b7011f2f51d76 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -1407,12 +1407,11 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) * If reference perf register is not supported then we should * use the nominal perf value */ - if (CPC_SUPPORTED(reference_reg)) { + if (CPC_SUPPORTED(reference_reg)) cpc_read(cpunum, reference_reg, &ref); - perf_caps->reference_perf = ref; - } else { - perf_caps->reference_perf = nom; - } + else + ref = nom; + perf_caps->reference_perf = ref; if (guaranteed_reg->type != ACPI_TYPE_BUFFER || IS_NULL_REG(&guaranteed_reg->cpc_entry.reg)) { From 0385160bdeff17d241447e6868c8df85fa66eab8 Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Wed, 18 Mar 2026 15:20:05 +0530 Subject: [PATCH 214/464] ACPI: CPPC: Check cpc_read() return values consistently BugLink: https://bugs.launchpad.net/bugs/2131705 Callers of cpc_read() ignore its return value, which can lead to using uninitialized or stale values when the read fails. Fix this by consistently checking cpc_read() return values in cppc_get_perf_caps(), cppc_get_perf_ctrs(), and cppc_get_perf(). Link: https://lore.kernel.org/lkml/48bdf87e-39f1-402f-a7dc-1a0e1e7a819d@nvidia.com/ Suggested-by: Rafael J. Wysocki Signed-off-by: Sumit Gupta Link: https://patch.msgid.link/20260318095005.2437960-1-sumitg@nvidia.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 0cc24977224a6c7d470860265a4990109f0a32ee) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/acpi/cppc_acpi.c | 99 +++++++++++++++++++++++++++++----------- 1 file changed, 72 insertions(+), 27 deletions(-) diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c index b7011f2f51d76..4b09921f30756 100644 --- a/drivers/acpi/cppc_acpi.c +++ b/drivers/acpi/cppc_acpi.c @@ -1394,45 +1394,66 @@ int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps) } } - cpc_read(cpunum, highest_reg, &high); + ret = cpc_read(cpunum, highest_reg, &high); + if (ret) + goto out_err; perf_caps->highest_perf = high; - cpc_read(cpunum, lowest_reg, &low); + ret = cpc_read(cpunum, lowest_reg, &low); + if (ret) + goto out_err; perf_caps->lowest_perf = low; - cpc_read(cpunum, nominal_reg, &nom); + ret = cpc_read(cpunum, nominal_reg, &nom); + if (ret) + goto out_err; perf_caps->nominal_perf = nom; /* * If reference perf register is not supported then we should * use the nominal perf value */ - if (CPC_SUPPORTED(reference_reg)) - cpc_read(cpunum, reference_reg, &ref); - else + if (CPC_SUPPORTED(reference_reg)) { + ret = cpc_read(cpunum, reference_reg, &ref); + if (ret) + goto out_err; + } else { ref = nom; + } perf_caps->reference_perf = ref; if (guaranteed_reg->type != ACPI_TYPE_BUFFER || IS_NULL_REG(&guaranteed_reg->cpc_entry.reg)) { perf_caps->guaranteed_perf = 0; } else { - cpc_read(cpunum, guaranteed_reg, &guaranteed); + ret = cpc_read(cpunum, guaranteed_reg, &guaranteed); + if (ret) + goto out_err; perf_caps->guaranteed_perf = guaranteed; } - cpc_read(cpunum, lowest_non_linear_reg, &min_nonlinear); + ret = cpc_read(cpunum, lowest_non_linear_reg, &min_nonlinear); + if (ret) + goto out_err; perf_caps->lowest_nonlinear_perf = min_nonlinear; - if (!high || !low || !nom || !ref || !min_nonlinear) + if (!high || !low || !nom || !ref || !min_nonlinear) { ret = -EFAULT; + goto out_err; + } /* Read optional lowest and nominal frequencies if present */ - if (CPC_SUPPORTED(low_freq_reg)) - cpc_read(cpunum, low_freq_reg, &low_f); + if (CPC_SUPPORTED(low_freq_reg)) { + ret = cpc_read(cpunum, low_freq_reg, &low_f); + if (ret) + goto out_err; + } - if (CPC_SUPPORTED(nom_freq_reg)) - cpc_read(cpunum, nom_freq_reg, &nom_f); + if (CPC_SUPPORTED(nom_freq_reg)) { + ret = cpc_read(cpunum, nom_freq_reg, &nom_f); + if (ret) + goto out_err; + } perf_caps->lowest_freq = low_f; perf_caps->nominal_freq = nom_f; @@ -1526,16 +1547,25 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs) } } - cpc_read(cpunum, delivered_reg, &delivered); - cpc_read(cpunum, reference_reg, &reference); + ret = cpc_read(cpunum, delivered_reg, &delivered); + if (ret) + goto out_err; + + ret = cpc_read(cpunum, reference_reg, &reference); + if (ret) + goto out_err; + /* * Per spec, if ctr_wrap_time optional register is unsupported, then the * performance counters are assumed to never wrap during the lifetime of * platform */ ctr_wrap_time = (u64)(~((u64)0)); - if (CPC_SUPPORTED(ctr_wrap_reg)) - cpc_read(cpunum, ctr_wrap_reg, &ctr_wrap_time); + if (CPC_SUPPORTED(ctr_wrap_reg)) { + ret = cpc_read(cpunum, ctr_wrap_reg, &ctr_wrap_time); + if (ret) + goto out_err; + } if (!delivered || !reference) { ret = -EFAULT; @@ -1811,24 +1841,39 @@ int cppc_get_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls) } /* Read optional elements if present */ - if (CPC_SUPPORTED(max_perf_reg)) - cpc_read(cpu, max_perf_reg, &max); + if (CPC_SUPPORTED(max_perf_reg)) { + ret = cpc_read(cpu, max_perf_reg, &max); + if (ret) + goto out_err; + } perf_ctrls->max_perf = max; - if (CPC_SUPPORTED(min_perf_reg)) - cpc_read(cpu, min_perf_reg, &min); + if (CPC_SUPPORTED(min_perf_reg)) { + ret = cpc_read(cpu, min_perf_reg, &min); + if (ret) + goto out_err; + } perf_ctrls->min_perf = min; - if (CPC_SUPPORTED(desired_perf_reg)) - cpc_read(cpu, desired_perf_reg, &desired_perf); + if (CPC_SUPPORTED(desired_perf_reg)) { + ret = cpc_read(cpu, desired_perf_reg, &desired_perf); + if (ret) + goto out_err; + } perf_ctrls->desired_perf = desired_perf; - if (CPC_SUPPORTED(energy_perf_reg)) - cpc_read(cpu, energy_perf_reg, &energy_perf); + if (CPC_SUPPORTED(energy_perf_reg)) { + ret = cpc_read(cpu, energy_perf_reg, &energy_perf); + if (ret) + goto out_err; + } perf_ctrls->energy_perf = energy_perf; - if (CPC_SUPPORTED(auto_sel_reg)) - cpc_read(cpu, auto_sel_reg, &auto_sel); + if (CPC_SUPPORTED(auto_sel_reg)) { + ret = cpc_read(cpu, auto_sel_reg, &auto_sel); + if (ret) + goto out_err; + } perf_ctrls->auto_sel = (bool)auto_sel; out_err: From 58d1656bdf31d0a7571521d0b848169473871b02 Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Thu, 26 Mar 2026 21:44:00 +0100 Subject: [PATCH 215/464] cpufreq: Remove max_freq_req update for pre-existing policy BugLink: https://bugs.launchpad.net/bugs/2131705 policy->max_freq_req QoS constraint represents the maximal allowed frequency than can be requested. It is set by: - writing to policyX/scaling_max sysfs file - toggling the cpufreq/boost sysfs file Upon calling freq_qos_update_request(), a successful update of the max_freq_req value triggers cpufreq_notifier_max(), followed by cpufreq_set_policy() which update the requested frequency for the policy. If the new max_freq_req value is not different from the original value, no frequency update is triggered. In a specific sequence of toggling: - cpufreq/boost sysfs file - CPU hot-plugging a CPU could end up with boost enabled but running at the maximal non-boost frequency, cpufreq_notifier_max() not being triggered. The following fixed that: commit 1608f0230510 ("cpufreq: Fix re-boost issue after hotplugging a CPU") The following: commit dd016f379ebc ("cpufreq: Introduce a more generic way to set default per-policy boost flag") also fixed the issue by correctly setting the max_freq_req constraint of a policy that is re-activated. This makes the first fix unnecessary. As the original issue is fixed by another method, this patch reverts: commit 1608f0230510 ("cpufreq: Fix re-boost issue after hotplugging a CPU") Reviewed-by: Lifeng Zheng Signed-off-by: Pierre Gondois Acked-by: Viresh Kumar Link: https://patch.msgid.link/20260326204404.1401849-2-pierre.gondois@arm.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 04aa9d0726cc6a23b348498815a9722b42d27c91) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cpufreq.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 44441ceedb768..17d7a280a499e 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1484,10 +1484,6 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, blocking_notifier_call_chain(&cpufreq_policy_notifier_list, CPUFREQ_CREATE_POLICY, policy); - } else { - ret = freq_qos_update_request(policy->max_freq_req, policy->max); - if (ret < 0) - goto out_destroy_policy; } if (cpufreq_driver->get && has_target()) { From db9cde1f2b84ea3efc3cbafa388b6abf7964fdad Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Thu, 26 Mar 2026 21:44:01 +0100 Subject: [PATCH 216/464] cpufreq: Add boost_freq_req QoS request BugLink: https://bugs.launchpad.net/bugs/2131705 The Power Management Quality of Service (PM QoS) allows to aggregate constraints from multiple entities. It is currently used to manage the min/max frequency of a given policy. Frequency constraints can come for instance from: - Thermal framework: acpi_thermal_cpufreq_init() - Firmware: _PPC objects: acpi_processor_ppc_init() - User: by setting policyX/scaling_[min|max]_freq The minimum of the max frequency constraints is used to compute the resulting maximum allowed frequency. When enabling boost frequencies, the same frequency request object (policy->max_freq_req) as to handle requests from users is used. As a result, when setting: - scaling_max_freq - boost The last sysfs file used overwrites the request from the other sysfs file. To avoid this, create a per-policy boost_freq_req to save the boost constraints instead of overwriting the last scaling_max_freq constraint. policy_set_boost() calls the cpufreq set_boost callback. Update the newly added boost_freq_req request from there: - whenever boost is toggled - to cover all possible paths In the existing .set_boost() callbacks: - Don't update policy->max as this is done through the qos notifier cpufreq_notifier_max() which calls cpufreq_set_policy(). - Remove freq_qos_update_request() calls as the qos request is now done in policy_set_boost() and updates the new boost_freq_req $ ## Init state scaling_max_freq:1000000 cpuinfo_max_freq:1000000 $ echo 700000 > scaling_max_freq scaling_max_freq:700000 cpuinfo_max_freq:1000000 $ echo 1 > ../boost scaling_max_freq:1200000 cpuinfo_max_freq:1200000 $ echo 800000 > scaling_max_freq scaling_max_freq:800000 cpuinfo_max_freq:1200000 $ ## Final step: $ ## Without the patches: $ echo 0 > ../boost scaling_max_freq:1000000 cpuinfo_max_freq:1000000 $ ## With the patches: $ echo 0 > ../boost scaling_max_freq:800000 cpuinfo_max_freq:1000000 Note: cpufreq_frequency_table_cpuinfo() updates policy->min and max from: A. cpufreq_boost_set_sw() \-cpufreq_frequency_table_cpuinfo() B. cpufreq_policy_online() \-cpufreq_table_validate_and_sort() \-cpufreq_frequency_table_cpuinfo() Keep these updates as some drivers expect policy->min and max to be set through B. Reviewed-by: Lifeng Zheng Signed-off-by: Pierre Gondois Acked-by: Viresh Kumar Link: https://patch.msgid.link/20260326204404.1401849-3-pierre.gondois@arm.com Signed-off-by: Rafael J. Wysocki (cherry picked from commit 6e39ba4e5a82aa5469b2ac517b74a71accb0540f) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/amd-pstate.c | 2 -- drivers/cpufreq/cppc_cpufreq.c | 10 ++------ drivers/cpufreq/cpufreq.c | 46 +++++++++++++++++++++++----------- include/linux/cpufreq.h | 1 + 4 files changed, 34 insertions(+), 25 deletions(-) diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index 78b70bdcb008f..c565d5f90b1fa 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -767,8 +767,6 @@ static int amd_pstate_cpu_boost_update(struct cpufreq_policy *policy, bool on) else if (policy->cpuinfo.max_freq > nominal_freq) policy->cpuinfo.max_freq = nominal_freq; - policy->max = policy->cpuinfo.max_freq; - if (cppc_state == AMD_PSTATE_PASSIVE) { ret = freq_qos_update_request(&cpudata->req[1], policy->cpuinfo.max_freq); if (ret < 0) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 5dfb109cf1f4e..7e7f9dfb7a24c 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -834,17 +834,11 @@ static int cppc_cpufreq_set_boost(struct cpufreq_policy *policy, int state) { struct cppc_cpudata *cpu_data = policy->driver_data; struct cppc_perf_caps *caps = &cpu_data->perf_caps; - int ret; if (state) - policy->max = cppc_perf_to_khz(caps, caps->highest_perf); + policy->cpuinfo.max_freq = cppc_perf_to_khz(caps, caps->highest_perf); else - policy->max = cppc_perf_to_khz(caps, caps->nominal_perf); - policy->cpuinfo.max_freq = policy->max; - - ret = freq_qos_update_request(policy->max_freq_req, policy->max); - if (ret < 0) - return ret; + policy->cpuinfo.max_freq = cppc_perf_to_khz(caps, caps->nominal_perf); return 0; } diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 17d7a280a499e..17164f2b240a7 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -609,10 +609,19 @@ static int policy_set_boost(struct cpufreq_policy *policy, bool enable) policy->boost_enabled = enable; ret = cpufreq_driver->set_boost(policy, enable); - if (ret) + if (ret) { policy->boost_enabled = !policy->boost_enabled; + return ret; + } - return ret; + ret = freq_qos_update_request(policy->boost_freq_req, policy->cpuinfo.max_freq); + if (ret < 0) { + policy->boost_enabled = !policy->boost_enabled; + cpufreq_driver->set_boost(policy, policy->boost_enabled); + return ret; + } + + return 0; } static ssize_t store_local_boost(struct cpufreq_policy *policy, @@ -1377,6 +1386,7 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) } freq_qos_remove_request(policy->min_freq_req); + freq_qos_remove_request(policy->boost_freq_req); kfree(policy->min_freq_req); cpufreq_policy_put_kobj(policy); @@ -1442,26 +1452,38 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, cpumask_and(policy->cpus, policy->cpus, cpu_online_mask); if (new_policy) { + unsigned int count; + for_each_cpu(j, policy->related_cpus) { per_cpu(cpufreq_cpu_data, j) = policy; add_cpu_dev_symlink(policy, j, get_cpu_device(j)); } - policy->min_freq_req = kzalloc(2 * sizeof(*policy->min_freq_req), + count = policy->boost_supported ? 3 : 2; + policy->min_freq_req = kzalloc(count * sizeof(*policy->min_freq_req), GFP_KERNEL); if (!policy->min_freq_req) { ret = -ENOMEM; goto out_destroy_policy; } + if (policy->boost_supported) { + policy->boost_freq_req = policy->min_freq_req + 2; + + ret = freq_qos_add_request(&policy->constraints, + policy->boost_freq_req, + FREQ_QOS_MAX, + policy->cpuinfo.max_freq); + if (ret < 0) { + policy->boost_freq_req = NULL; + goto out_destroy_policy; + } + } + ret = freq_qos_add_request(&policy->constraints, policy->min_freq_req, FREQ_QOS_MIN, FREQ_QOS_MIN_DEFAULT_VALUE); if (ret < 0) { - /* - * So we don't call freq_qos_remove_request() for an - * uninitialized request. - */ kfree(policy->min_freq_req); policy->min_freq_req = NULL; goto out_destroy_policy; @@ -2785,16 +2807,10 @@ int cpufreq_boost_set_sw(struct cpufreq_policy *policy, int state) return -ENXIO; ret = cpufreq_frequency_table_cpuinfo(policy); - if (ret) { + if (ret) pr_err("%s: Policy frequency update failed\n", __func__); - return ret; - } - - ret = freq_qos_update_request(policy->max_freq_req, policy->max); - if (ret < 0) - return ret; - return 0; + return ret; } EXPORT_SYMBOL_GPL(cpufreq_boost_set_sw); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 4317c5a312bd1..637d37819a7b9 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -81,6 +81,7 @@ struct cpufreq_policy { struct freq_constraints constraints; struct freq_qos_request *min_freq_req; struct freq_qos_request *max_freq_req; + struct freq_qos_request *boost_freq_req; struct cpufreq_frequency_table *freq_table; enum cpufreq_table_sorting freq_table_sorted; From ee64644cdb93beca0c6a6780724b45d906d0c499 Mon Sep 17 00:00:00 2001 From: Viresh Kumar Date: Tue, 31 Mar 2026 10:33:46 +0530 Subject: [PATCH 217/464] cpufreq: Allocate QoS freq_req objects with policy BugLink: https://bugs.launchpad.net/bugs/2131705 A recent change exposed a bug in the error path: if freq_qos_add_request(boost_freq_req) fails, min_freq_req may remain a valid pointer even though it was never successfully added. During policy teardown, this leads to an unconditional call to freq_qos_remove_request(), triggering a WARN. The current design allocates all three freq_req objects together, making the lifetime rules unclear and error handling fragile. Simplify this by allocating the QoS freq_req objects at policy allocation time. The policy itself is dynamically allocated, and two of the three requests are always needed anyway. This ensures consistent lifetime management and eliminates the inconsistent state in failure paths. Reported-by: Zhongqiu Han Fixes: 6e39ba4e5a82 ("cpufreq: Add boost_freq_req QoS request") Signed-off-by: Viresh Kumar Reviewed-by: Lifeng Zheng Tested-by: Pierre Gondois Reviewed-by: Zhongqiu Han Link: https://patch.msgid.link/a293f29d841b86c51f34699c6e717e01858d8ada.1774933424.git.viresh.kumar@linaro.org Signed-off-by: Rafael J. Wysocki (cherry picked from commit 9266b4da051a410d9e6c5c0b0ef0c877855aa1b8) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cpufreq.c | 53 +++++++++++---------------------------- include/linux/cpufreq.h | 6 ++--- 2 files changed, 17 insertions(+), 42 deletions(-) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 17164f2b240a7..06da81591562d 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -614,7 +614,7 @@ static int policy_set_boost(struct cpufreq_policy *policy, bool enable) return ret; } - ret = freq_qos_update_request(policy->boost_freq_req, policy->cpuinfo.max_freq); + ret = freq_qos_update_request(&policy->boost_freq_req, policy->cpuinfo.max_freq); if (ret < 0) { policy->boost_enabled = !policy->boost_enabled; cpufreq_driver->set_boost(policy, policy->boost_enabled); @@ -769,7 +769,7 @@ static ssize_t store_##file_name \ if (ret) \ return ret; \ \ - ret = freq_qos_update_request(policy->object##_freq_req, val);\ + ret = freq_qos_update_request(&policy->object##_freq_req, val); \ return ret >= 0 ? count : ret; \ } @@ -1374,7 +1374,7 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) /* Cancel any pending policy->update work before freeing the policy. */ cancel_work_sync(&policy->update); - if (policy->max_freq_req) { + if (freq_qos_request_active(&policy->max_freq_req)) { /* * Remove max_freq_req after sending CPUFREQ_REMOVE_POLICY * notification, since CPUFREQ_CREATE_POLICY notification was @@ -1382,12 +1382,13 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) */ blocking_notifier_call_chain(&cpufreq_policy_notifier_list, CPUFREQ_REMOVE_POLICY, policy); - freq_qos_remove_request(policy->max_freq_req); + freq_qos_remove_request(&policy->max_freq_req); } - freq_qos_remove_request(policy->min_freq_req); - freq_qos_remove_request(policy->boost_freq_req); - kfree(policy->min_freq_req); + if (freq_qos_request_active(&policy->min_freq_req)) + freq_qos_remove_request(&policy->min_freq_req); + if (freq_qos_request_active(&policy->boost_freq_req)) + freq_qos_remove_request(&policy->boost_freq_req); cpufreq_policy_put_kobj(policy); free_cpumask_var(policy->real_cpus); @@ -1452,57 +1453,31 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, cpumask_and(policy->cpus, policy->cpus, cpu_online_mask); if (new_policy) { - unsigned int count; - for_each_cpu(j, policy->related_cpus) { per_cpu(cpufreq_cpu_data, j) = policy; add_cpu_dev_symlink(policy, j, get_cpu_device(j)); } - count = policy->boost_supported ? 3 : 2; - policy->min_freq_req = kzalloc(count * sizeof(*policy->min_freq_req), - GFP_KERNEL); - if (!policy->min_freq_req) { - ret = -ENOMEM; - goto out_destroy_policy; - } - if (policy->boost_supported) { - policy->boost_freq_req = policy->min_freq_req + 2; - ret = freq_qos_add_request(&policy->constraints, - policy->boost_freq_req, + &policy->boost_freq_req, FREQ_QOS_MAX, policy->cpuinfo.max_freq); - if (ret < 0) { - policy->boost_freq_req = NULL; + if (ret < 0) goto out_destroy_policy; - } } ret = freq_qos_add_request(&policy->constraints, - policy->min_freq_req, FREQ_QOS_MIN, + &policy->min_freq_req, FREQ_QOS_MIN, FREQ_QOS_MIN_DEFAULT_VALUE); - if (ret < 0) { - kfree(policy->min_freq_req); - policy->min_freq_req = NULL; + if (ret < 0) goto out_destroy_policy; - } - - /* - * This must be initialized right here to avoid calling - * freq_qos_remove_request() on uninitialized request in case - * of errors. - */ - policy->max_freq_req = policy->min_freq_req + 1; ret = freq_qos_add_request(&policy->constraints, - policy->max_freq_req, FREQ_QOS_MAX, + &policy->max_freq_req, FREQ_QOS_MAX, FREQ_QOS_MAX_DEFAULT_VALUE); - if (ret < 0) { - policy->max_freq_req = NULL; + if (ret < 0) goto out_destroy_policy; - } blocking_notifier_call_chain(&cpufreq_policy_notifier_list, CPUFREQ_CREATE_POLICY, policy); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 637d37819a7b9..fa6370d73b0fc 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -79,9 +79,9 @@ struct cpufreq_policy { * called, but you're in IRQ context */ struct freq_constraints constraints; - struct freq_qos_request *min_freq_req; - struct freq_qos_request *max_freq_req; - struct freq_qos_request *boost_freq_req; + struct freq_qos_request min_freq_req; + struct freq_qos_request max_freq_req; + struct freq_qos_request boost_freq_req; struct cpufreq_frequency_table *freq_table; enum cpufreq_table_sorting freq_table_sorted; From 9eda570e9b8c730c6200eccec8f36020c4d59f91 Mon Sep 17 00:00:00 2001 From: "Mario Limonciello (AMD)" Date: Thu, 26 Mar 2026 14:36:20 -0500 Subject: [PATCH 218/464] cpufreq/amd-pstate: Cache the max frequency in cpudata BugLink: https://bugs.launchpad.net/bugs/2131705 The value of maximum frequency is fixed and never changes. Doing calculations every time based off of perf is unnecessary. Reviewed-by: Gautham R. Shenoy Link: https://lore.kernel.org/r/20260326193620.649441-1-mario.limonciello@amd.com Signed-off-by: Mario Limonciello (AMD) (backported from commit 8cdc494013dfcd48f31eafe19b18fd67c224dd8a) [jamien: minor context-line drift in amd-pstate.c hunks from 3-way auto-merge; +/- content is byte-identical to upstream.] Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/amd-pstate.c | 27 +++++++++------------------ drivers/cpufreq/amd-pstate.h | 2 ++ 2 files changed, 11 insertions(+), 18 deletions(-) diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index c565d5f90b1fa..b2bb91b945886 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -755,15 +755,13 @@ static void amd_pstate_adjust_perf(struct cpufreq_policy *policy, static int amd_pstate_cpu_boost_update(struct cpufreq_policy *policy, bool on) { struct amd_cpudata *cpudata = policy->driver_data; - union perf_cached perf = READ_ONCE(cpudata->perf); - u32 nominal_freq, max_freq; + u32 nominal_freq; int ret = 0; nominal_freq = READ_ONCE(cpudata->nominal_freq); - max_freq = perf_to_freq(perf, cpudata->nominal_freq, perf.highest_perf); if (on) - policy->cpuinfo.max_freq = max_freq; + policy->cpuinfo.max_freq = cpudata->max_freq; else if (policy->cpuinfo.max_freq > nominal_freq) policy->cpuinfo.max_freq = nominal_freq; @@ -948,13 +946,15 @@ static int amd_pstate_init_freq(struct amd_cpudata *cpudata) WRITE_ONCE(cpudata->nominal_freq, nominal_freq); + /* max_freq is calculated according to (nominal_freq * highest_perf)/nominal_perf */ max_freq = perf_to_freq(perf, nominal_freq, perf.highest_perf); + WRITE_ONCE(cpudata->max_freq, max_freq); + lowest_nonlinear_freq = perf_to_freq(perf, nominal_freq, perf.lowest_nonlinear_perf); WRITE_ONCE(cpudata->lowest_nonlinear_freq, lowest_nonlinear_freq); /** * Below values need to be initialized correctly, otherwise driver will fail to load - * max_freq is calculated according to (nominal_freq * highest_perf)/nominal_perf * lowest_nonlinear_freq is a value between [min_freq, nominal_freq] * Check _CPC in ACPI table objects if any values are incorrect */ @@ -1017,9 +1017,7 @@ static int amd_pstate_cpu_init(struct cpufreq_policy *policy) policy->cpuinfo.min_freq = policy->min = perf_to_freq(perf, cpudata->nominal_freq, perf.lowest_perf); - policy->cpuinfo.max_freq = policy->max = perf_to_freq(perf, - cpudata->nominal_freq, - perf.highest_perf); + policy->cpuinfo.max_freq = policy->max = cpudata->max_freq; ret = amd_pstate_cppc_enable(policy); if (ret) @@ -1086,14 +1084,9 @@ static void amd_pstate_cpu_exit(struct cpufreq_policy *policy) static ssize_t show_amd_pstate_max_freq(struct cpufreq_policy *policy, char *buf) { - struct amd_cpudata *cpudata; - union perf_cached perf; - - cpudata = policy->driver_data; - perf = READ_ONCE(cpudata->perf); + struct amd_cpudata *cpudata = policy->driver_data; - return sysfs_emit(buf, "%u\n", - perf_to_freq(perf, cpudata->nominal_freq, perf.highest_perf)); + return sysfs_emit(buf, "%u\n", cpudata->max_freq); } static ssize_t show_amd_pstate_lowest_nonlinear_freq(struct cpufreq_policy *policy, @@ -1499,9 +1492,7 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) policy->cpuinfo.min_freq = policy->min = perf_to_freq(perf, cpudata->nominal_freq, perf.lowest_perf); - policy->cpuinfo.max_freq = policy->max = perf_to_freq(perf, - cpudata->nominal_freq, - perf.highest_perf); + policy->cpuinfo.max_freq = policy->max = cpudata->max_freq; policy->driver_data = cpudata; ret = amd_pstate_cppc_enable(policy); diff --git a/drivers/cpufreq/amd-pstate.h b/drivers/cpufreq/amd-pstate.h index cb45fdca27a6c..0d26e56b7938b 100644 --- a/drivers/cpufreq/amd-pstate.h +++ b/drivers/cpufreq/amd-pstate.h @@ -68,6 +68,7 @@ struct amd_aperf_mperf { * @min_limit_freq: Cached value of policy->min (in khz) * @max_limit_freq: Cached value of policy->max (in khz) * @nominal_freq: the frequency (in khz) that mapped to nominal_perf + * @max_freq: in ideal conditions the maximum frequency (in khz) possible frequency * @lowest_nonlinear_freq: the frequency (in khz) that mapped to lowest_nonlinear_perf * @cur: Difference of Aperf/Mperf/tsc count between last and current sample * @prev: Last Aperf/Mperf/tsc count value read from register @@ -94,6 +95,7 @@ struct amd_cpudata { u32 min_limit_freq; u32 max_limit_freq; u32 nominal_freq; + u32 max_freq; u32 lowest_nonlinear_freq; struct amd_aperf_mperf cur; From 2e5400874f8acc73bf266cd7e811718bd66f09c2 Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Mon, 11 May 2026 15:55:28 +0200 Subject: [PATCH 219/464] NVIDIA: SAUCE: cpufreq: Extract cpufreq_policy_init_qos() function BugLink: https://bugs.launchpad.net/bugs/2131705 Extract the QoS related logic from cpufreq_policy_online() to make the function shorter/simpler. The logic is placed in cpufreq_policy_init_qos() and is now executed right after the following calls: - cpufreq_driver->init() - cpufreq_table_validate_and_sort() This helps preparing following patches that will, in cpufreq_policy_init_qos(): - treat the policy->min/max values set by drivers as QoS requests. - set a default policy->min/max value to all policies. No functional change. Signed-off-by: Pierre Gondois (backported from https://lore.kernel.org/lkml/20260511135538.522653-1-pierre.gondois@arm.com/) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cpufreq.c | 53 +++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 06da81591562d..906d8820c5778 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1397,6 +1397,32 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) kfree(policy); } +static int cpufreq_policy_init_qos(struct cpufreq_policy *policy) +{ + int ret; + + if (policy->boost_supported) { + ret = freq_qos_add_request(&policy->constraints, + &policy->boost_freq_req, + FREQ_QOS_MAX, + policy->cpuinfo.max_freq); + if (ret < 0) + return ret; + } + + ret = freq_qos_add_request(&policy->constraints, &policy->min_freq_req, + FREQ_QOS_MIN, FREQ_QOS_MIN_DEFAULT_VALUE); + if (ret < 0) + return ret; + + ret = freq_qos_add_request(&policy->constraints, &policy->max_freq_req, + FREQ_QOS_MAX, FREQ_QOS_MAX_DEFAULT_VALUE); + if (ret < 0) + return ret; + + return ret; +} + static int cpufreq_policy_online(struct cpufreq_policy *policy, unsigned int cpu, bool new_policy) { @@ -1442,6 +1468,12 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, if (ret) goto out_offline_policy; + if (new_policy) { + ret = cpufreq_policy_init_qos(policy); + if (ret < 0) + goto out_offline_policy; + } + /* related_cpus should at least include policy->cpus. */ cpumask_copy(policy->related_cpus, policy->cpus); } @@ -1458,27 +1490,6 @@ static int cpufreq_policy_online(struct cpufreq_policy *policy, add_cpu_dev_symlink(policy, j, get_cpu_device(j)); } - if (policy->boost_supported) { - ret = freq_qos_add_request(&policy->constraints, - &policy->boost_freq_req, - FREQ_QOS_MAX, - policy->cpuinfo.max_freq); - if (ret < 0) - goto out_destroy_policy; - } - - ret = freq_qos_add_request(&policy->constraints, - &policy->min_freq_req, FREQ_QOS_MIN, - FREQ_QOS_MIN_DEFAULT_VALUE); - if (ret < 0) - goto out_destroy_policy; - - ret = freq_qos_add_request(&policy->constraints, - &policy->max_freq_req, FREQ_QOS_MAX, - FREQ_QOS_MAX_DEFAULT_VALUE); - if (ret < 0) - goto out_destroy_policy; - blocking_notifier_call_chain(&cpufreq_policy_notifier_list, CPUFREQ_CREATE_POLICY, policy); } From 7c9c4e411a0ddbe5851b273fd33d780a6ed1efde Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Mon, 11 May 2026 15:55:29 +0200 Subject: [PATCH 220/464] NVIDIA: SAUCE: cpufreq: Set default policy->min/max values for all drivers BugLink: https://bugs.launchpad.net/bugs/2131705 Some drivers set policy->min/max in their .init() callback. cpufreq_set_policy() will ultimately override them through: cpufreq_policy_online() \-cpufreq_init_policy() \-cpufreq_set_policy() \-/* Set policy->min/max */ Thus the policy min/max values provided are only temporary. There is an exception if CPUFREQ_NEED_INITIAL_FREQ_CHECK is set and: cpufreq_policy_online() \-__cpufreq_driver_target() \-cpufreq_driver->target() To prepare for a following patch that will remove all policy->min/max initialization in the driver .init() callback if the min/max value is equal to the cpuinfo.min/max_freq, set a default policy->min/max value for all drivers. Signed-off-by: Pierre Gondois (backported from https://lore.kernel.org/lkml/20260511135538.522653-1-pierre.gondois@arm.com/) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/cpufreq.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 906d8820c5778..494187930fe4b 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1401,6 +1401,13 @@ static int cpufreq_policy_init_qos(struct cpufreq_policy *policy) { int ret; + /* + * If the driver didn't set policy->min/max, set them as + * they are used to clamp frequency requests. + */ + policy->min = policy->min ? policy->min : policy->cpuinfo.min_freq; + policy->max = policy->max ? policy->max : policy->cpuinfo.max_freq; + if (policy->boost_supported) { ret = freq_qos_add_request(&policy->constraints, &policy->boost_freq_req, From 6ebae5778537ea3512a6f06a533d00f725a55496 Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Mon, 11 May 2026 15:55:30 +0200 Subject: [PATCH 221/464] NVIDIA: SAUCE: cpufreq: Remove driver default policy->min/max init BugLink: https://bugs.launchpad.net/bugs/2131705 Prior to [1], drivers were setting policy->min/max and the value was used as a QoS constraint. After that change, the values were only temporarily used: cpufreq_set_policy() ultimately overriding them through: cpufreq_policy_online() \-cpufreq_init_policy() \-cpufreq_set_policy() \-/* Set policy->min/max */ This patch reinstate the initial behaviour. This will allow drivers to request min/max QoS frequencies if desired. For instance, the cppc driver advertises a lowest non-linear frequency, which should be used as a min QoS value. To avoid having drivers setting policy->min/max to default values which are considered as QoS values (i.e. the reason why [1] was introduced), remove the initialization of policy->min/max in .init() callbacks wherever the policy->min/max values are identical to the policy->cpuinfo.min/max_freq. Indeed, the previous patch ("cpufreq: Set default policy->min/max values for all drivers") makes this initialization redundant. The only drivers where these values are different are: - gx-suspmod.c (min) - cppc-cpufreq.c (min) - longrun.c [1] commit 521223d8b3ec ("cpufreq: Fix initialization of min and max frequency QoS requests") Signed-off-by: Pierre Gondois (backported from https://lore.kernel.org/lkml/20260511135538.522653-1-pierre.gondois@arm.com/) [jamien: 3-way auto-merge resolved context drift in amd-pstate.c and intel_pstate.c against this tree; +/- content is byte-identical to v2 3/4.] Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/cpufreq/amd-pstate.c | 14 ++++++-------- drivers/cpufreq/cppc_cpufreq.c | 5 ++--- drivers/cpufreq/cpufreq-nforce2.c | 4 ++-- drivers/cpufreq/freq_table.c | 7 +++---- drivers/cpufreq/gx-suspmod.c | 2 +- drivers/cpufreq/intel_pstate.c | 3 --- drivers/cpufreq/pcc-cpufreq.c | 10 ++++------ drivers/cpufreq/pxa3xx-cpufreq.c | 5 ++--- drivers/cpufreq/sh-cpufreq.c | 6 ++---- drivers/cpufreq/virtual-cpufreq.c | 5 +---- 10 files changed, 23 insertions(+), 38 deletions(-) diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index b2bb91b945886..aa0de6f201cb5 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -1014,10 +1014,9 @@ static int amd_pstate_cpu_init(struct cpufreq_policy *policy) perf = READ_ONCE(cpudata->perf); - policy->cpuinfo.min_freq = policy->min = perf_to_freq(perf, - cpudata->nominal_freq, - perf.lowest_perf); - policy->cpuinfo.max_freq = policy->max = cpudata->max_freq; + policy->cpuinfo.min_freq = perf_to_freq(perf, cpudata->nominal_freq, + perf.lowest_perf); + policy->cpuinfo.max_freq = cpudata->max_freq; ret = amd_pstate_cppc_enable(policy); if (ret) @@ -1489,10 +1488,9 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) perf = READ_ONCE(cpudata->perf); - policy->cpuinfo.min_freq = policy->min = perf_to_freq(perf, - cpudata->nominal_freq, - perf.lowest_perf); - policy->cpuinfo.max_freq = policy->max = cpudata->max_freq; + policy->cpuinfo.min_freq = perf_to_freq(perf, cpudata->nominal_freq, + perf.lowest_perf); + policy->cpuinfo.max_freq = cpudata->max_freq; policy->driver_data = cpudata; ret = amd_pstate_cppc_enable(policy); diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 7e7f9dfb7a24c..5abac50df7508 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -660,8 +660,6 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) * Section 8.4.7.1.1.5 of ACPI 6.1 spec) */ policy->min = cppc_perf_to_khz(caps, caps->lowest_nonlinear_perf); - policy->max = cppc_perf_to_khz(caps, policy->boost_enabled ? - caps->highest_perf : caps->nominal_perf); /* * Set cpuinfo.min_freq to Lowest to make the full range of performance @@ -669,7 +667,8 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) * nonlinear perf */ policy->cpuinfo.min_freq = cppc_perf_to_khz(caps, caps->lowest_perf); - policy->cpuinfo.max_freq = policy->max; + policy->cpuinfo.max_freq = cppc_perf_to_khz(caps, policy->boost_enabled ? + caps->highest_perf : caps->nominal_perf); policy->transition_delay_us = cppc_cpufreq_get_transition_delay_us(cpu); policy->shared_type = cpu_data->shared_type; diff --git a/drivers/cpufreq/cpufreq-nforce2.c b/drivers/cpufreq/cpufreq-nforce2.c index fbbbe501cf2dc..831102522ad64 100644 --- a/drivers/cpufreq/cpufreq-nforce2.c +++ b/drivers/cpufreq/cpufreq-nforce2.c @@ -355,8 +355,8 @@ static int nforce2_cpu_init(struct cpufreq_policy *policy) min_fsb = NFORCE2_MIN_FSB; /* cpuinfo and default policy values */ - policy->min = policy->cpuinfo.min_freq = min_fsb * fid * 100; - policy->max = policy->cpuinfo.max_freq = max_fsb * fid * 100; + policy->cpuinfo.min_freq = min_fsb * fid * 100; + policy->cpuinfo.max_freq = max_fsb * fid * 100; return 0; } diff --git a/drivers/cpufreq/freq_table.c b/drivers/cpufreq/freq_table.c index 5b364d8da4f92..ea994647abc88 100644 --- a/drivers/cpufreq/freq_table.c +++ b/drivers/cpufreq/freq_table.c @@ -49,16 +49,15 @@ int cpufreq_frequency_table_cpuinfo(struct cpufreq_policy *policy) max_freq = freq; } - policy->min = policy->cpuinfo.min_freq = min_freq; - policy->max = max_freq; + policy->cpuinfo.min_freq = min_freq; /* * If the driver has set its own cpuinfo.max_freq above max_freq, leave * it as is. */ if (policy->cpuinfo.max_freq < max_freq) - policy->max = policy->cpuinfo.max_freq = max_freq; + policy->cpuinfo.max_freq = max_freq; - if (policy->min == ~0) + if (min_freq == ~0) return -EINVAL; else return 0; diff --git a/drivers/cpufreq/gx-suspmod.c b/drivers/cpufreq/gx-suspmod.c index d269a4f26f98e..d40c9e0bbb740 100644 --- a/drivers/cpufreq/gx-suspmod.c +++ b/drivers/cpufreq/gx-suspmod.c @@ -421,7 +421,7 @@ static int cpufreq_gx_cpu_init(struct cpufreq_policy *policy) policy->min = maxfreq / max_duration; else policy->min = maxfreq / POLICY_MIN_DIV; - policy->max = maxfreq; + policy->cpuinfo.min_freq = maxfreq / max_duration; policy->cpuinfo.max_freq = maxfreq; diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index 8ef15e1db0cd6..886e0aca65403 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -3049,9 +3049,6 @@ static int __intel_pstate_cpu_init(struct cpufreq_policy *policy) policy->cpuinfo.max_freq = READ_ONCE(global.no_turbo) ? cpu->pstate.max_freq : cpu->pstate.turbo_freq; - policy->min = policy->cpuinfo.min_freq; - policy->max = policy->cpuinfo.max_freq; - intel_pstate_init_acpi_perf_limits(policy); policy->fast_switch_possible = true; diff --git a/drivers/cpufreq/pcc-cpufreq.c b/drivers/cpufreq/pcc-cpufreq.c index ac2e90a65f0c4..0f185a13577f8 100644 --- a/drivers/cpufreq/pcc-cpufreq.c +++ b/drivers/cpufreq/pcc-cpufreq.c @@ -551,13 +551,11 @@ static int pcc_cpufreq_cpu_init(struct cpufreq_policy *policy) goto out; } - policy->max = policy->cpuinfo.max_freq = - ioread32(&pcch_hdr->nominal) * 1000; - policy->min = policy->cpuinfo.min_freq = - ioread32(&pcch_hdr->minimum_frequency) * 1000; + policy->cpuinfo.max_freq = ioread32(&pcch_hdr->nominal) * 1000; + policy->cpuinfo.min_freq = ioread32(&pcch_hdr->minimum_frequency) * 1000; - pr_debug("init: policy->max is %d, policy->min is %d\n", - policy->max, policy->min); + pr_debug("init: max_freq is %d, min_freq is %d\n", + policy->cpuinfo.max_freq, policy->cpuinfo.min_freq); out: return result; } diff --git a/drivers/cpufreq/pxa3xx-cpufreq.c b/drivers/cpufreq/pxa3xx-cpufreq.c index 50ff3b6a69000..06b27cbc59d6a 100644 --- a/drivers/cpufreq/pxa3xx-cpufreq.c +++ b/drivers/cpufreq/pxa3xx-cpufreq.c @@ -185,9 +185,8 @@ static int pxa3xx_cpufreq_init(struct cpufreq_policy *policy) int ret = -EINVAL; /* set default policy and cpuinfo */ - policy->min = policy->cpuinfo.min_freq = 104000; - policy->max = policy->cpuinfo.max_freq = - (cpu_is_pxa320()) ? 806000 : 624000; + policy->cpuinfo.min_freq = 104000; + policy->cpuinfo.max_freq = (cpu_is_pxa320()) ? 806000 : 624000; policy->cpuinfo.transition_latency = 1000; /* FIXME: 1 ms, assumed */ if (cpu_is_pxa300() || cpu_is_pxa310()) diff --git a/drivers/cpufreq/sh-cpufreq.c b/drivers/cpufreq/sh-cpufreq.c index 642ddb9ea217e..3c99d7009cbe2 100644 --- a/drivers/cpufreq/sh-cpufreq.c +++ b/drivers/cpufreq/sh-cpufreq.c @@ -124,10 +124,8 @@ static int sh_cpufreq_cpu_init(struct cpufreq_policy *policy) dev_notice(dev, "no frequency table found, falling back " "to rate rounding.\n"); - policy->min = policy->cpuinfo.min_freq = - (clk_round_rate(cpuclk, 1) + 500) / 1000; - policy->max = policy->cpuinfo.max_freq = - (clk_round_rate(cpuclk, ~0UL) + 500) / 1000; + policy->cpuinfo.min_freq = (clk_round_rate(cpuclk, 1) + 500) / 1000; + policy->cpuinfo.max_freq = (clk_round_rate(cpuclk, ~0UL) + 500) / 1000; } return 0; diff --git a/drivers/cpufreq/virtual-cpufreq.c b/drivers/cpufreq/virtual-cpufreq.c index 4159f31349b16..dc78b74409af4 100644 --- a/drivers/cpufreq/virtual-cpufreq.c +++ b/drivers/cpufreq/virtual-cpufreq.c @@ -164,10 +164,7 @@ static int virt_cpufreq_get_freq_info(struct cpufreq_policy *policy) policy->cpuinfo.min_freq = 1; policy->cpuinfo.max_freq = virt_cpufreq_get_perftbl_entry(policy->cpu, 0); - policy->min = policy->cpuinfo.min_freq; - policy->max = policy->cpuinfo.max_freq; - - policy->cur = policy->max; + policy->cur = policy->cpuinfo.max_freq; return 0; } From e1c620418f5f28d5ff43cf80e676b923d8a96ad6 Mon Sep 17 00:00:00 2001 From: Pierre Gondois Date: Mon, 11 May 2026 15:55:31 +0200 Subject: [PATCH 222/464] NVIDIA: SAUCE: cpufreq: Use policy->min/max init as QoS request BugLink: https://bugs.launchpad.net/bugs/2131705 Consider policy->min/max being set in the driver .init() callback as a QoS request. Impacted driver are: - gx-suspmod.c (min) - cppc-cpufreq.c (min) - longrun.c (min/max) Update the documentation accordingly. Signed-off-by: Pierre Gondois (backported from https://lore.kernel.org/lkml/20260511135538.522653-1-pierre.gondois@arm.com/) Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- Documentation/cpu-freq/cpu-drivers.rst | 10 ++++++++-- drivers/cpufreq/cpufreq.c | 12 ++++++++++-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/Documentation/cpu-freq/cpu-drivers.rst b/Documentation/cpu-freq/cpu-drivers.rst index c5635ac3de547..ab4f3c0f3a89b 100644 --- a/Documentation/cpu-freq/cpu-drivers.rst +++ b/Documentation/cpu-freq/cpu-drivers.rst @@ -114,8 +114,14 @@ Then, the driver must fill in the following values: |policy->cur | The current operating frequency of | | | this CPU (if appropriate) | +-----------------------------------+--------------------------------------+ -|policy->min, | | -|policy->max, | | +|policy->min | If set by the driver in ->init(), | +| | used as initial minimum frequency | +| | QoS request. | ++-----------------------------------+--------------------------------------+ +|policy->max | If set by the driver in ->init(), | +| | used as initial maximum frequency | +| | QoS request. | ++-----------------------------------+--------------------------------------+ |policy->policy and, if necessary, | | |policy->governor | must contain the "default policy" for| | | this CPU. A few moments later, | diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 494187930fe4b..04dc562214b2b 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1399,8 +1399,16 @@ static void cpufreq_policy_free(struct cpufreq_policy *policy) static int cpufreq_policy_init_qos(struct cpufreq_policy *policy) { + unsigned int min_freq, max_freq; int ret; + /* Use policy->min/max set by the driver as QoS requests. */ + min_freq = max(FREQ_QOS_MIN_DEFAULT_VALUE, policy->min); + if (policy->max) + max_freq = min(FREQ_QOS_MAX_DEFAULT_VALUE, policy->max); + else + max_freq = FREQ_QOS_MAX_DEFAULT_VALUE; + /* * If the driver didn't set policy->min/max, set them as * they are used to clamp frequency requests. @@ -1418,12 +1426,12 @@ static int cpufreq_policy_init_qos(struct cpufreq_policy *policy) } ret = freq_qos_add_request(&policy->constraints, &policy->min_freq_req, - FREQ_QOS_MIN, FREQ_QOS_MIN_DEFAULT_VALUE); + FREQ_QOS_MIN, min_freq); if (ret < 0) return ret; ret = freq_qos_add_request(&policy->constraints, &policy->max_freq_req, - FREQ_QOS_MAX, FREQ_QOS_MAX_DEFAULT_VALUE); + FREQ_QOS_MAX, max_freq); if (ret < 0) return ret; From d8b70bf2393797796972a18f35c04af3f154253e Mon Sep 17 00:00:00 2001 From: Sumit Gupta Date: Sat, 25 Apr 2026 01:48:14 +0530 Subject: [PATCH 223/464] NVIDIA: SAUCE: cpufreq: CPPC: add autonomous mode boot parameter support BugLink: https://bugs.launchpad.net/bugs/2131705 Add a kernel boot parameter 'cppc_cpufreq.auto_sel_mode' to enable CPPC autonomous performance selection on all CPUs at system startup. When autonomous mode is enabled, the hardware automatically adjusts CPU performance based on workload demands using Energy Performance Preference (EPP) hints. When auto_sel_mode=1: - Configure all CPUs for autonomous operation on first init - Set EPP to performance preference (0x0) - Use HW min/max_perf when available; otherwise initialize from caps - Clamp desired_perf to bounds before enabling autonomous mode - Hardware controls frequency instead of the OS governor The boot parameter is applied only during first policy initialization. Skip applying it on CPU hotplug to preserve runtime sysfs configuration. This patch depends on patch [2] ("cpufreq: Set policy->min and max as real QoS constraints") so that the policy->min/max set in cppc_cpufreq_cpu_init() are not overridden by cpufreq_set_policy() during init. Reviewed-by: Randy Dunlap Signed-off-by: Sumit Gupta (backported from https://lore.kernel.org/lkml/20260424201814.230071-1-sumitg@nvidia.com/) [jamien: hunk #2 (cppc_set_enable() insertion in cppc_cpufreq_cpu_init) rebased onto Pierre's v2 series, which replaced the local min/max vars with direct policy->min assignment; insertion point and code are unchanged.] Signed-off-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- .../admin-guide/kernel-parameters.txt | 13 +++ drivers/cpufreq/cppc_cpufreq.c | 89 +++++++++++++++++-- 2 files changed, 97 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a030ce253b4b7..2915345847fde 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1052,6 +1052,19 @@ Kernel parameters policy to use. This governor must be registered in the kernel before the cpufreq driver probes. + cppc_cpufreq.auto_sel_mode= + [CPU_FREQ] Enable ACPI CPPC autonomous performance + selection. When enabled, hardware automatically adjusts + CPU frequency on all CPUs based on workload demands. + In Autonomous mode, Energy Performance Preference (EPP) + hints guide hardware toward performance (0x0) or energy + efficiency (0xff). + Requires ACPI CPPC autonomous selection register support. + Format: + Default: 0 (disabled) + 0: use cpufreq governors + 1: enable if supported by hardware + cpu_init_udelay=N [X86,EARLY] Delay for N microsec between assert and de-assert of APIC INIT to start processors. This delay occurs diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 5abac50df7508..be28d7e6cf63e 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -28,6 +28,9 @@ static struct cpufreq_driver cppc_cpufreq_driver; +/* Autonomous Selection boot parameter */ +static bool auto_sel_mode; + #ifdef CONFIG_ACPI_CPPC_CPUFREQ_FIE static enum { FIE_UNSET = -1, @@ -655,6 +658,14 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) caps = &cpu_data->perf_caps; policy->driver_data = cpu_data; + /* + * Enable CPPC for both OS-driven and autonomous modes. + * The Enable register is optional - some platforms may not support it + */ + ret = cppc_set_enable(cpu, true); + if (ret && ret != -EOPNOTSUPP) + pr_warn("Failed to enable CPPC for CPU%d (%d)\n", cpu, ret); + /* * Set min to lowest nonlinear perf to avoid any efficiency penalty (see * Section 8.4.7.1.1.5 of ACPI 6.1 spec) @@ -707,11 +718,71 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) policy->cur = cppc_perf_to_khz(caps, caps->highest_perf); cpu_data->perf_ctrls.desired_perf = caps->highest_perf; - ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); - if (ret) { - pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", - caps->highest_perf, cpu, ret); - goto out; + /* + * Enable autonomous mode on first init if boot param is set. + * Check last_governor to detect first init and skip if auto_sel + * is already enabled. + */ + if (auto_sel_mode && policy->last_governor[0] == '\0' && + !cpu_data->perf_ctrls.auto_sel) { + /* Init min/max_perf from caps if not already set by HW. */ + if (!cpu_data->perf_ctrls.min_perf) + cpu_data->perf_ctrls.min_perf = caps->lowest_nonlinear_perf; + if (!cpu_data->perf_ctrls.max_perf) + cpu_data->perf_ctrls.max_perf = policy->boost_enabled ? + caps->highest_perf : caps->nominal_perf; + + cpu_data->perf_ctrls.desired_perf = + clamp_t(u32, cpu_data->perf_ctrls.desired_perf, + cpu_data->perf_ctrls.min_perf, + cpu_data->perf_ctrls.max_perf); + + policy->cur = cppc_perf_to_khz(caps, + cpu_data->perf_ctrls.desired_perf); + + /* EPP is optional - some platforms may not support it */ + ret = cppc_set_epp(cpu, CPPC_EPP_PERFORMANCE_PREF); + if (ret && ret != -EOPNOTSUPP) + pr_warn("Failed to set EPP for CPU%d (%d)\n", cpu, ret); + else if (!ret) + cpu_data->perf_ctrls.energy_perf = CPPC_EPP_PERFORMANCE_PREF; + + /* Program min/max/desired into CPPC regs before enabling auto_sel. */ + ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); + if (ret) { + pr_debug("Err setting perf for autonomous mode CPU:%d ret:%d\n", + cpu, ret); + goto out; + } + + ret = cppc_set_auto_sel(cpu, true); + if (ret && ret != -EOPNOTSUPP) { + pr_warn("Failed autonomous config for CPU%d (%d)\n", + cpu, ret); + goto out; + } + if (!ret) + cpu_data->perf_ctrls.auto_sel = true; + } + + if (cpu_data->perf_ctrls.auto_sel) { + /* Sync policy limits from HW when autonomous mode is active */ + policy->min = cppc_perf_to_khz(caps, + cpu_data->perf_ctrls.min_perf ?: + caps->lowest_nonlinear_perf); + policy->max = cppc_perf_to_khz(caps, + cpu_data->perf_ctrls.max_perf ?: + (policy->boost_enabled ? + caps->highest_perf : + caps->nominal_perf)); + } else { + /* Normal mode: governors control frequency */ + ret = cppc_set_perf(cpu, &cpu_data->perf_ctrls); + if (ret) { + pr_debug("Err setting perf value:%d on CPU:%d. ret:%d\n", + caps->highest_perf, cpu, ret); + goto out; + } } cppc_cpufreq_cpu_fie_init(policy); @@ -1031,10 +1102,18 @@ static int __init cppc_cpufreq_init(void) static void __exit cppc_cpufreq_exit(void) { + unsigned int cpu; + + for_each_present_cpu(cpu) + cppc_set_auto_sel(cpu, false); + cpufreq_unregister_driver(&cppc_cpufreq_driver); cppc_freq_invariance_exit(); } +module_param(auto_sel_mode, bool, 0444); +MODULE_PARM_DESC(auto_sel_mode, "Enable CPPC autonomous performance selection at boot"); + module_exit(cppc_cpufreq_exit); MODULE_AUTHOR("Ashwin Chaugule"); MODULE_DESCRIPTION("CPUFreq driver based on the ACPI CPPC v5.0+ spec"); From 5fef67abb3c3559a55fb1831f4f8d3408ed91860 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 30 Apr 2026 02:40:45 -0700 Subject: [PATCH 224/464] Revert "NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on" BugLink: https://bugs.launchpad.net/bugs/2150727 This reverts commit 11d6b22546c50c28e16c79e3c24be23839a765c1. Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 75 +++------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 - 2 files changed, 11 insertions(+), 65 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 654ddd68e40cc..f4fa1d84de4e5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1489,7 +1489,7 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) if (!arm_smmu_cdtab_allocated(&master->cd_table)) return; cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (!cdptr) + if (WARN_ON(!cdptr)) return; arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } @@ -1503,22 +1503,6 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; cd_table->s1cdmax = master->ssid_bits; - - /* - * When a device doesn't support PASID (non default SSID), ssid_bits is - * set to 0. This also sets S1CDMAX to 0, which disables the substreams - * and ignores the S1DSS field. - * - * On the other hand, if a device demands ATS to be always on even when - * its default substream is IOMMU bypassed, it has to use EATS that is - * only effective with an STE (CFG=S1translate, S1DSS=Bypass). For such - * use cases, S1CDMAX has to be !0, in order to make use of S1DSS/EATS. - * - * Set S1CDMAX no lower than 1. This would add a dummy substream in the - * CD table but it should never be used by an actual CD. - */ - if (master->ats_always_on) - cd_table->s1cdmax = max_t(u8, cd_table->s1cdmax, 1); max_contexts = 1 << cd_table->s1cdmax; if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_CDTAB) || @@ -3273,8 +3257,7 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain, * When the last user of the CD table goes away downgrade the STE back * to a non-cd_table one, by re-attaching its sid_domain. */ - if (!master->ats_always_on && - !arm_smmu_ssids_in_use(&master->cd_table)) { + if (!arm_smmu_ssids_in_use(&master->cd_table)) { struct iommu_domain *sid_domain = iommu_driver_get_domain_for_dev(master->dev); @@ -3290,7 +3273,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, struct iommu_domain *old_domain, struct device *dev, struct arm_smmu_ste *ste, - unsigned int s1dss, bool ats_always_on) + unsigned int s1dss) { struct arm_smmu_master *master = dev_iommu_priv_get(dev); struct arm_smmu_attach_state state = { @@ -3309,7 +3292,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, * If the CD table is not in use we can use the provided STE, otherwise * we use a cdtable STE with the provided S1DSS. */ - if (ats_always_on || arm_smmu_ssids_in_use(&master->cd_table)) { + if (arm_smmu_ssids_in_use(&master->cd_table)) { /* * If a CD table has to be present then we need to run with ATS * on because we have to assume a PASID is using ATS. For @@ -3345,8 +3328,7 @@ static int arm_smmu_attach_dev_identity(struct iommu_domain *domain, arm_smmu_master_clear_vmaster(master); arm_smmu_make_bypass_ste(master->smmu, &ste); arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste, - STRTAB_STE_1_S1DSS_BYPASS, - master->ats_always_on); + STRTAB_STE_1_S1DSS_BYPASS); return 0; } @@ -3369,8 +3351,7 @@ static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain, arm_smmu_master_clear_vmaster(master); arm_smmu_make_abort_ste(&ste); arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste, - STRTAB_STE_1_S1DSS_TERMINATE, - false); + STRTAB_STE_1_S1DSS_TERMINATE); return 0; } @@ -3607,40 +3588,6 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master) kfree(master->streams); } -static int arm_smmu_master_prepare_ats(struct arm_smmu_master *master) -{ - bool s1p = master->smmu->features & ARM_SMMU_FEAT_TRANS_S1; - unsigned int stu = __ffs(master->smmu->pgsize_bitmap); - struct pci_dev *pdev = to_pci_dev(master->dev); - int ret; - - if (!arm_smmu_ats_supported(master)) - return 0; - - if (!pci_ats_always_on(pdev)) - goto out_prepare; - - /* - * S1DSS is required for ATS to be always on for identity domain cases. - * However, the S1DSS field is ignored if !IDR0_S1P or !IDR1_SSIDSIZE. - */ - if (!s1p || !master->smmu->ssid_bits) { - dev_info_once(master->dev, - "SMMU doesn't support ATS to be always on\n"); - goto out_prepare; - } - - master->ats_always_on = true; - - ret = arm_smmu_alloc_cd_tables(master); - if (ret) - return ret; - -out_prepare: - pci_prepare_ats(pdev, stu); - return 0; -} - static struct iommu_device *arm_smmu_probe_device(struct device *dev) { int ret; @@ -3689,14 +3636,14 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; - ret = arm_smmu_master_prepare_ats(master); - if (ret) - goto err_disable_pasid; + if (dev_is_pci(dev)) { + unsigned int stu = __ffs(smmu->pgsize_bitmap); + + pci_prepare_ats(to_pci_dev(dev), stu); + } return &smmu->iommu; -err_disable_pasid: - arm_smmu_disable_pasid(master); err_free_master: kfree(master); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index f966d474b61fd..3c6d65d36164f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -848,7 +848,6 @@ struct arm_smmu_master { bool ats_enabled : 1; bool ste_ats_enabled : 1; bool stall_enabled; - bool ats_always_on; unsigned int ssid_bits; unsigned int iopf_refcount; }; From f1e61487f6ca82a23247a4cac4cff0da29c1ffbd Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 30 Apr 2026 02:40:45 -0700 Subject: [PATCH 225/464] Revert "NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for non-CXL NVIDIA GPUs" BugLink: https://bugs.launchpad.net/bugs/2150727 This reverts commit 208b48c1e520001d08e8abe6ed49e0f7e2e5aa24. Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/pci/ats.c | 3 +-- drivers/pci/pci.h | 9 --------- drivers/pci/quirks.c | 23 ----------------------- 3 files changed, 1 insertion(+), 34 deletions(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 6db45ae2cc8e3..1795131f0697f 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -245,8 +245,7 @@ bool pci_ats_always_on(struct pci_dev *pdev) if (pdev->is_virtfn) pdev = pci_physfn(pdev); - return pci_cxl_ats_always_on(pdev) || - pci_dev_specific_ats_always_on(pdev); + return pci_cxl_ats_always_on(pdev); } EXPORT_SYMBOL_GPL(pci_ats_always_on); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 13fa71f965900..13d998fbacce6 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -1150,15 +1150,6 @@ static inline int pci_dev_specific_reset(struct pci_dev *dev, bool probe) } #endif -#if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_PCI_ATS) -bool pci_dev_specific_ats_always_on(struct pci_dev *dev); -#else -static inline bool pci_dev_specific_ats_always_on(struct pci_dev *dev) -{ - return false; -} -#endif - #if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_ARM64) int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment, struct resource *res); diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 6009eb1122c9b..cdb6271397f6a 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -5729,29 +5729,6 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1457, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1459, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145a, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145c, quirk_intel_e2000_no_ats); - -static const struct pci_dev_ats_always_on { - u16 vendor; - u16 device; -} pci_dev_ats_always_on[] = { - { PCI_VENDOR_ID_NVIDIA, 0x2e12, }, - { PCI_VENDOR_ID_NVIDIA, 0x2e2a, }, - { PCI_VENDOR_ID_NVIDIA, 0x2e2b, }, - { 0 } -}; - -/* Some non-CXL devices support ATS on RID when it is IOMMU-bypassed */ -bool pci_dev_specific_ats_always_on(struct pci_dev *pdev) -{ - const struct pci_dev_ats_always_on *i; - - for (i = pci_dev_ats_always_on; i->vendor; i++) { - if (i->vendor == pdev->vendor && i->device == pdev->device) - return true; - } - - return false; -} #endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ From e81415c8294e6419a3c56d49b7a675c388858408 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Thu, 30 Apr 2026 02:40:45 -0700 Subject: [PATCH 226/464] Revert "NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable devices" BugLink: https://bugs.launchpad.net/bugs/2150727 This reverts commit 967f9fe2d7279bb9f61f993c2f788bcbcdfc69bf. Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/pci/ats.c | 44 ----------------------------------- include/linux/pci-ats.h | 3 --- include/uapi/linux/pci_regs.h | 5 ---- 3 files changed, 52 deletions(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 1795131f0697f..ec6c8dbdc5e9c 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -205,50 +205,6 @@ int pci_ats_page_aligned(struct pci_dev *pdev) return 0; } -/* - * CXL r4.0, sec 3.2.5.13 Memory Type on CXL.cache notes: to source requests on - * CXL.cache, devices need to get the Host Physical Address (HPA) from the Host - * by means of an ATS request on CXL.io. - * - * In other world, CXL.cache devices cannot access physical memory without ATS. - */ -static bool pci_cxl_ats_always_on(struct pci_dev *pdev) -{ - int offset; - u16 cap; - - offset = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, - CXL_DVSEC_PCIE_DEVICE); - if (!offset) - return false; - - pci_read_config_word(pdev, offset + CXL_DVSEC_CAP_OFFSET, &cap); - if (cap & CXL_DVSEC_CACHE_CAPABLE) - return true; - - return false; -} - -/** - * pci_ats_always_on - Whether the PCI device requires ATS to be always enabled - * @pdev: the PCI device - * - * Returns true, if the PCI device requires non-PASID ATS function on an IOMMU - * bypassed configuration. - */ -bool pci_ats_always_on(struct pci_dev *pdev) -{ - if (pci_ats_disabled() || !pci_ats_supported(pdev)) - return false; - - /* A VF inherits its PF's requirement for ATS function */ - if (pdev->is_virtfn) - pdev = pci_physfn(pdev); - - return pci_cxl_ats_always_on(pdev); -} -EXPORT_SYMBOL_GPL(pci_ats_always_on); - #ifdef CONFIG_PCI_PRI void pci_pri_init(struct pci_dev *pdev) { diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index d14ba727d38b3..75c6c86cf09dc 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -12,7 +12,6 @@ int pci_prepare_ats(struct pci_dev *dev, int ps); void pci_disable_ats(struct pci_dev *dev); int pci_ats_queue_depth(struct pci_dev *dev); int pci_ats_page_aligned(struct pci_dev *dev); -bool pci_ats_always_on(struct pci_dev *dev); #else /* CONFIG_PCI_ATS */ static inline bool pci_ats_supported(struct pci_dev *d) { return false; } @@ -25,8 +24,6 @@ static inline int pci_ats_queue_depth(struct pci_dev *d) { return -ENODEV; } static inline int pci_ats_page_aligned(struct pci_dev *dev) { return 0; } -static inline bool pci_ats_always_on(struct pci_dev *dev) -{ return false; } #endif /* CONFIG_PCI_ATS */ #ifdef CONFIG_PCI_PRI diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index ea7c124490fdb..543275ff9ed62 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1412,9 +1412,4 @@ #define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_ID __GENMASK(15, 8) #define PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW __GENMASK(31, 16) -/* CXL 2.0 8.1.3: PCIe DVSEC for CXL Device */ -#define CXL_DVSEC_PCIE_DEVICE 0 -#define CXL_DVSEC_CAP_OFFSET 0xA -#define CXL_DVSEC_CACHE_CAPABLE BIT(0) - #endif /* LINUX_PCI_REGS_H */ From 37ec8232e77a2f3e9dc75c730df6cc6107381f9d Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Sun, 26 Apr 2026 22:54:00 -0700 Subject: [PATCH 227/464] NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable devices BugLink: https://bugs.launchpad.net/bugs/2150727 Controlled by the IOMMU driver, ATS is usually enabled "on demand" when a given PASID on a device is attached to an I/O page table. This is working even when a device has no translation on its RID (i.e., the RID is IOMMU bypassed). However, certain PCIe devices require non-PASID ATS on their RID even when the RID is IOMMU bypassed. Call this "always on". For example, CXL spec r4.0 notes in sec 3.2.5.13 Memory Type on CXL.cache: "To source requests on CXL.cache, devices need to get the Host Physical Address (HPA) from the Host by means of an ATS request on CXL.io." In other words, the CXL.cache capability requires ATS; otherwise, it can't access host physical memory. Introduce a new pci_ats_always_on() helper for the IOMMU driver to scan a PCI device and shift ATS policies between "on demand" and "always on". Add the support for CXL.cache devices first. Pre-CXL devices will be added in quirks.c file. Note that pci_ats_always_on() validates against pci_ats_supported(), so we ensure that untrusted devices (e.g. external ports) will not be always on. This maintains the existing ATS security policy regarding potential side- channel attacks via ATS. Cc: linux-cxl@vger.kernel.org Suggested-by: Vikram Sethi Suggested-by: Jason Gunthorpe Reviewed-by: Jonathan Cameron Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian Tested-by: Nirmoy Das Acked-by: Nirmoy Das Signed-off-by: Nicolin Chen Reviewed-by: Dave Jiang (backported from https://lore.kernel.org/r/f6734b9dad0050138676f11ecd14e9db1cf6b697.1777269009.git.nicolinc@nvidia.com) [Nirmoy: Adapt to already existing PCI_DVSEC_CXL_CACHE_CAPABLE.] Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/pci/ats.c | 43 +++++++++++++++++++++++++++++++++++++++++ include/linux/pci-ats.h | 3 +++ 2 files changed, 46 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index ec6c8dbdc5e9c..fc871858b65bc 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -205,6 +205,49 @@ int pci_ats_page_aligned(struct pci_dev *pdev) return 0; } +/* + * CXL r4.0, sec 3.2.5.13 Memory Type on CXL.cache notes: to source requests on + * CXL.cache, devices need to get the Host Physical Address (HPA) from the Host + * by means of an ATS request on CXL.io. + * + * In other words, CXL.cache devices cannot access host physical memory without + * ATS. + */ +static bool pci_cxl_ats_always_on(struct pci_dev *pdev) +{ + int offset; + u16 cap; + + offset = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!offset) + return false; + + if (pci_read_config_word(pdev, offset + PCI_DVSEC_CXL_CAP, &cap)) + return false; + + return cap & PCI_DVSEC_CXL_CACHE_CAPABLE; +} + +/** + * pci_ats_always_on - Whether the PCI device requires ATS to be always enabled + * @pdev: the PCI device + * + * Returns true, if the PCI device requires ATS for basic functional operation. + */ +bool pci_ats_always_on(struct pci_dev *pdev) +{ + if (pci_ats_disabled() || !pci_ats_supported(pdev)) + return false; + + /* A VF inherits its PF's requirement for ATS function */ + if (pdev->is_virtfn) + pdev = pci_physfn(pdev); + + return pci_cxl_ats_always_on(pdev); +} +EXPORT_SYMBOL_GPL(pci_ats_always_on); + #ifdef CONFIG_PCI_PRI void pci_pri_init(struct pci_dev *pdev) { diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index 75c6c86cf09dc..d14ba727d38b3 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -12,6 +12,7 @@ int pci_prepare_ats(struct pci_dev *dev, int ps); void pci_disable_ats(struct pci_dev *dev); int pci_ats_queue_depth(struct pci_dev *dev); int pci_ats_page_aligned(struct pci_dev *dev); +bool pci_ats_always_on(struct pci_dev *dev); #else /* CONFIG_PCI_ATS */ static inline bool pci_ats_supported(struct pci_dev *d) { return false; } @@ -24,6 +25,8 @@ static inline int pci_ats_queue_depth(struct pci_dev *d) { return -ENODEV; } static inline int pci_ats_page_aligned(struct pci_dev *dev) { return 0; } +static inline bool pci_ats_always_on(struct pci_dev *dev) +{ return false; } #endif /* CONFIG_PCI_ATS */ #ifdef CONFIG_PCI_PRI From 1e219e60fd2fac80a1a3bdcf319dd03cd5aa4534 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Sun, 26 Apr 2026 22:54:01 -0700 Subject: [PATCH 228/464] NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for pre-CXL devices BugLink: https://bugs.launchpad.net/bugs/2150727 Some NVIDIA GPU/NIC devices, though they don't implement CXL config space, have many CXL-like properties. Call this kind "pre-CXL". Similar to CXL.cache capability, these pre-CXL devices also require the ATS function even when their RIDs are IOMMU bypassed, i.e. keep ATS "always on" v.s. "on demand" when a non-zero PASID line gets enabled in SVA use cases. Introduce pci_dev_specific_ats_always_on() quirk function to scan a list of IDs for these devices. Then, include it in pci_ats_always_on(). Suggested-by: Jason Gunthorpe Reviewed-by: Nirmoy Das Tested-by: Nirmoy Das Reviewed-by: Jonathan Cameron Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian Signed-off-by: Nicolin Chen Reviewed-by: Dave Jiang (backported from https://lore.kernel.org/r/1a8cf5e88051ab5c10417edb94df598ecbc810cf.1777269009.git.nicolinc@nvidia.com) [Nirmoy: Apply after reverting older ATS always-on PCI quirk support.] Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/pci/ats.c | 3 ++- drivers/pci/pci.h | 9 +++++++++ drivers/pci/quirks.c | 38 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index fc871858b65bc..3846447ea322f 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -244,7 +244,8 @@ bool pci_ats_always_on(struct pci_dev *pdev) if (pdev->is_virtfn) pdev = pci_physfn(pdev); - return pci_cxl_ats_always_on(pdev); + return pci_cxl_ats_always_on(pdev) || + pci_dev_specific_ats_always_on(pdev); } EXPORT_SYMBOL_GPL(pci_ats_always_on); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 13d998fbacce6..13fa71f965900 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -1150,6 +1150,15 @@ static inline int pci_dev_specific_reset(struct pci_dev *dev, bool probe) } #endif +#if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_PCI_ATS) +bool pci_dev_specific_ats_always_on(struct pci_dev *dev); +#else +static inline bool pci_dev_specific_ats_always_on(struct pci_dev *dev) +{ + return false; +} +#endif + #if defined(CONFIG_PCI_QUIRKS) && defined(CONFIG_ARM64) int acpi_get_rc_resources(struct device *dev, const char *hid, u16 segment, struct resource *res); diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index cdb6271397f6a..b9d6c37272ddd 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -5729,6 +5729,44 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1457, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x1459, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145a, quirk_intel_e2000_no_ats); DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x145c, quirk_intel_e2000_no_ats); + +static bool quirk_nvidia_gpu_ats_always_on(struct pci_dev *pdev) +{ + switch (pdev->device) { + case 0x2e00 ... 0x2e3f: /* GB20B */ + return true; + } + return false; +} + +static const struct pci_dev_ats_always_on { + u16 vendor; + u16 device; + bool (*ats_always_on)(struct pci_dev *dev); +} pci_dev_ats_always_on[] = { + /* NVIDIA GPUs */ + { PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, quirk_nvidia_gpu_ats_always_on }, + /* NVIDIA CX10 Family NVlink-C2C */ + { PCI_VENDOR_ID_MELLANOX, 0x2101, NULL }, + { 0 } +}; + +/* Some pre-CXL devices require ATS when it is IOMMU-bypassed */ +bool pci_dev_specific_ats_always_on(struct pci_dev *pdev) +{ + const struct pci_dev_ats_always_on *i; + + for (i = pci_dev_ats_always_on; i->vendor; i++) { + if (i->vendor != pdev->vendor) + continue; + if (i->ats_always_on && i->ats_always_on(pdev)) + return true; + if (!i->ats_always_on && i->device == pdev->device) + return true; + } + + return false; +} #endif /* CONFIG_PCI_ATS */ /* Freescale PCIe doesn't support MSI in RC mode */ From 992e44e47425a145cd1e347dd23f557200484032 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Sun, 26 Apr 2026 22:54:02 -0700 Subject: [PATCH 229/464] NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on BugLink: https://bugs.launchpad.net/bugs/2150727 When a device's default substream attaches to an identity domain, the SMMU driver currently sets the device's STE between two modes: Mode 1: Cfg=Translate, S1DSS=Bypass, EATS=1 Mode 2: Cfg=bypass (EATS is ignored by HW) When there is an active PASID (non-default substream), mode 1 is used. And when there is no PASID support or no active PASID, mode 2 is used. The driver will also downgrade an STE from mode 1 to mode 2, when the last active substream becomes inactive. However, there are PCIe devices that demand ATS to be always on. For these devices, their STEs have to use the mode 1 as HW ignores EATS with mode 2. Change the driver accordingly: - always use the mode 1 - never downgrade to mode 2 - allocate and retain a CD table (see note below) Note that these devices might not support PASID, i.e. doing non-PASID ATS. In such a case, the ssid_bits is set to 0. However, s1cdmax must be set to a !0 value in order to keep the S1DSS field effective. Thus, when a master requires ats_always_on, set its s1cdmax to at least 1, meaning that the CD table will have a dummy entry (SSID=1) that will never be used. Now for these devices, arm_smmu_cdtab_allocated() will always return true, v.s. false prior to this change. When its default substream is attached to an IDENTITY domain, its first CD is NULL in the table, which is a totally valid case. Thus, add "!master->ats_always_on" to the condition. Reviewed-by: Jonathan Cameron Tested-by: Nirmoy Das Acked-by: Nirmoy Das Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian Signed-off-by: Nicolin Chen Reviewed-by: Dave Jiang (backported from https://lore.kernel.org/r/7403163ebf59380f88c7503b3adf0dae07428df8.1777269009.git.nicolinc@nvidia.com) [Nirmoy: Apply after reverting older ATS always-on arm-smmu-v3 support.] Signed-off-by: Nirmoy Das Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 75 ++++++++++++++++++--- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 68 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f4fa1d84de4e5..f033288d91116 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1489,8 +1489,11 @@ void arm_smmu_clear_cd(struct arm_smmu_master *master, ioasid_t ssid) if (!arm_smmu_cdtab_allocated(&master->cd_table)) return; cdptr = arm_smmu_get_cd_ptr(master, ssid); - if (WARN_ON(!cdptr)) + if (!cdptr) { + /* Only ats_always_on allows a NULL CD on default substream */ + WARN_ON(!master->ats_always_on || ssid); return; + } arm_smmu_write_cd_entry(master, ssid, cdptr, &target); } @@ -1503,6 +1506,22 @@ static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master) struct arm_smmu_ctx_desc_cfg *cd_table = &master->cd_table; cd_table->s1cdmax = master->ssid_bits; + + /* + * When a device doesn't support PASID (non default SSID), ssid_bits is + * set to 0. This also sets S1CDMAX to 0, which disables the substreams + * and ignores the S1DSS field. + * + * On the other hand, if a device demands ATS to be always on even when + * its default substream is IOMMU bypassed, it has to use EATS that is + * only effective with an STE (CFG=S1translate, S1DSS=Bypass). For such + * use cases, S1CDMAX has to be !0, in order to make use of S1DSS/EATS. + * + * Set S1CDMAX no lower than 1. This would add a dummy substream in the + * CD table but it should never be used by an actual CD. + */ + if (master->ats_always_on) + cd_table->s1cdmax = max_t(u8, cd_table->s1cdmax, 1); max_contexts = 1 << cd_table->s1cdmax; if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_CDTAB) || @@ -3257,7 +3276,8 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain, * When the last user of the CD table goes away downgrade the STE back * to a non-cd_table one, by re-attaching its sid_domain. */ - if (!arm_smmu_ssids_in_use(&master->cd_table)) { + if (!master->ats_always_on && + !arm_smmu_ssids_in_use(&master->cd_table)) { struct iommu_domain *sid_domain = iommu_driver_get_domain_for_dev(master->dev); @@ -3281,6 +3301,8 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, .old_domain = old_domain, .ssid = IOMMU_NO_PASID, }; + bool ats_always_on = master->ats_always_on && + s1dss != STRTAB_STE_1_S1DSS_TERMINATE; /* * Do not allow any ASID to be changed while are working on the STE, @@ -3292,7 +3314,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain, * If the CD table is not in use we can use the provided STE, otherwise * we use a cdtable STE with the provided S1DSS. */ - if (arm_smmu_ssids_in_use(&master->cd_table)) { + if (ats_always_on || arm_smmu_ssids_in_use(&master->cd_table)) { /* * If a CD table has to be present then we need to run with ATS * on because we have to assume a PASID is using ATS. For @@ -3588,6 +3610,42 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master) kfree(master->streams); } +static int arm_smmu_master_prepare_ats(struct arm_smmu_master *master) +{ + bool s1p = master->smmu->features & ARM_SMMU_FEAT_TRANS_S1; + unsigned int stu = __ffs(master->smmu->pgsize_bitmap); + struct pci_dev *pdev; + int ret; + + if (!arm_smmu_ats_supported(master)) + return 0; + + pdev = to_pci_dev(master->dev); + + if (!pci_ats_always_on(pdev)) + goto out_prepare; + + /* + * S1DSS is required for ATS to be always on for identity domain cases. + * However, the S1DSS field is ignored if !IDR0_S1P or !IDR1_SSIDSIZE. + */ + if (!s1p || !master->smmu->ssid_bits) { + dev_info_once(master->dev, + "SMMU doesn't support ATS to be always on\n"); + goto out_prepare; + } + + master->ats_always_on = true; + + ret = arm_smmu_alloc_cd_tables(master); + if (ret) + return ret; + +out_prepare: + pci_prepare_ats(pdev, stu); + return 0; +} + static struct iommu_device *arm_smmu_probe_device(struct device *dev) { int ret; @@ -3636,14 +3694,15 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; - if (dev_is_pci(dev)) { - unsigned int stu = __ffs(smmu->pgsize_bitmap); - - pci_prepare_ats(to_pci_dev(dev), stu); - } + ret = arm_smmu_master_prepare_ats(master); + if (ret) + goto err_disable_pasid; return &smmu->iommu; +err_disable_pasid: + arm_smmu_disable_pasid(master); + arm_smmu_remove_master(master); err_free_master: kfree(master); return ERR_PTR(ret); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 3c6d65d36164f..f966d474b61fd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -848,6 +848,7 @@ struct arm_smmu_master { bool ats_enabled : 1; bool ste_ats_enabled : 1; bool stall_enabled; + bool ats_always_on; unsigned int ssid_bits; unsigned int iopf_refcount; }; From 16175337cdc0607d8006dc1d0be2a8c9f81ae255 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Fri, 6 Mar 2026 16:47:38 +0000 Subject: [PATCH 230/464] cxl: support Type2 when initializing cxl_dev_state BugLink: https://bugs.launchpad.net/bugs/2153819 In preparation for type2 drivers add function and macro for differentiating CXL memory expanders (type 3) from CXL device accelerators (type 2) helping drivers built from public headers to embed struct cxl_dev_state inside a private struct. Update type3 driver for using this same initialization. Signed-off-by: Alejandro Lucero Reviewed-by: Dave Jiang Reviewed-by: Alison Schofield Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Link: https://patch.msgid.link/20260306164741.3796372-2-alejandro.lucero-palau@amd.com Signed-off-by: Dave Jiang (cherry picked from commit 9a775c07bb04384f7c03a35dd04818ed818c1f71) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/mbox.c | 12 +++++------- drivers/cxl/core/memdev.c | 24 ++++++++++++++++++++++++ drivers/cxl/cxlmem.h | 34 +++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 14 +++++++------- tools/testing/cxl/test/mem.c | 3 +-- 5 files changed, 70 insertions(+), 17 deletions(-) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index 12386d9127054..aa3724f51ce93 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1521,23 +1521,21 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host) } EXPORT_SYMBOL_NS_GPL(cxl_mailbox_init, "CXL"); -struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev) +struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev, u64 serial, + u16 dvsec) { struct cxl_memdev_state *mds; int rc; - mds = devm_kzalloc(dev, sizeof(*mds), GFP_KERNEL); + mds = devm_cxl_dev_state_create(dev, CXL_DEVTYPE_CLASSMEM, serial, + dvsec, struct cxl_memdev_state, cxlds, + true); if (!mds) { dev_err(dev, "No memory available\n"); return ERR_PTR(-ENOMEM); } mutex_init(&mds->event.log_lock); - mds->cxlds.dev = dev; - mds->cxlds.reg_map.host = dev; - mds->cxlds.cxl_mbox.host = dev; - mds->cxlds.reg_map.resource = CXL_RESOURCE_NONE; - mds->cxlds.type = CXL_DEVTYPE_CLASSMEM; rc = devm_cxl_register_mce_notifier(dev, &mds->mce_notifier); if (rc == -EOPNOTSUPP) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 273c22118d3d8..99e422594885a 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -656,6 +656,30 @@ static void detach_memdev(struct work_struct *work) static struct lock_class_key cxl_memdev_key; +struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, + enum cxl_devtype type, + u64 serial, u16 dvsec, + size_t size, bool has_mbox) +{ + struct cxl_dev_state *cxlds = devm_kzalloc(dev, size, GFP_KERNEL); + + if (!cxlds) + return NULL; + + cxlds->dev = dev; + cxlds->type = type; + cxlds->serial = serial; + cxlds->cxl_dvsec = dvsec; + cxlds->reg_map.host = dev; + cxlds->reg_map.resource = CXL_RESOURCE_NONE; + + if (has_mbox) + cxlds->cxl_mbox.host = dev; + + return cxlds; +} +EXPORT_SYMBOL_NS_GPL(_devm_cxl_dev_state_create, "CXL"); + static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds, const struct file_operations *fops, const struct cxl_memdev_attach *attach) diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index e21d744d639bd..71367cb5178ca 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -523,6 +523,37 @@ to_cxl_memdev_state(struct cxl_dev_state *cxlds) return container_of(cxlds, struct cxl_memdev_state, cxlds); } +struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, + enum cxl_devtype type, + u64 serial, u16 dvsec, + size_t size, bool has_mbox); + +/** + * cxl_dev_state_create - safely create and cast a cxl dev state embedded in a + * driver specific struct. + * + * @parent: device behind the request + * @type: CXL device type + * @serial: device identification + * @dvsec: dvsec capability offset + * @drv_struct: driver struct embedding a cxl_dev_state struct + * @member: name of the struct cxl_dev_state member in drv_struct + * @mbox: true if mailbox supported + * + * Returns a pointer to the drv_struct allocated and embedding a cxl_dev_state + * struct initialized. + * + * Introduced for Type2 driver support. + */ +#define devm_cxl_dev_state_create(parent, type, serial, dvsec, drv_struct, member, mbox) \ + ({ \ + static_assert(__same_type(struct cxl_dev_state, \ + ((drv_struct *)NULL)->member)); \ + static_assert(offsetof(drv_struct, member) == 0); \ + (drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec, \ + sizeof(drv_struct), mbox); \ + }) + enum cxl_opcode { CXL_MBOX_OP_INVALID = 0x0000, CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID, @@ -858,7 +889,8 @@ int cxl_dev_state_identify(struct cxl_memdev_state *mds); int cxl_await_media_ready(struct cxl_dev_state *cxlds); int cxl_enumerate_cmds(struct cxl_memdev_state *mds); int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info); -struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev); +struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev, u64 serial, + u16 dvsec); void set_exclusive_cxl_commands(struct cxl_memdev_state *mds, unsigned long *cmds); void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds, diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index a5922116db2a8..cfd6a0553e0fc 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -865,25 +865,25 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) int rc, pmu_count; unsigned int i; bool irq_avail; + u16 dvsec; rc = pcim_enable_device(pdev); if (rc) return rc; pci_set_master(pdev); - mds = cxl_memdev_state_create(&pdev->dev); + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + pci_warn(pdev, "Device DVSEC not present, skip CXL.mem init\n"); + + mds = cxl_memdev_state_create(&pdev->dev, pci_get_dsn(pdev), dvsec); if (IS_ERR(mds)) return PTR_ERR(mds); cxlds = &mds->cxlds; pci_set_drvdata(pdev, cxlds); cxlds->rcd = is_cxl_restricted(pdev); - cxlds->serial = pci_get_dsn(pdev); - cxlds->cxl_dvsec = pci_find_dvsec_capability( - pdev, PCI_VENDOR_ID_CXL, PCI_DVSEC_CXL_DEVICE); - if (!cxlds->cxl_dvsec) - dev_warn(&pdev->dev, - "Device DVSEC not present, skip CXL.mem init\n"); rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map); if (rc) diff --git a/tools/testing/cxl/test/mem.c b/tools/testing/cxl/test/mem.c index cb87e8c0e63c0..79f42f4474d47 100644 --- a/tools/testing/cxl/test/mem.c +++ b/tools/testing/cxl/test/mem.c @@ -1716,7 +1716,7 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) if (rc) return rc; - mds = cxl_memdev_state_create(dev); + mds = cxl_memdev_state_create(dev, pdev->id + 1, 0); if (IS_ERR(mds)) return PTR_ERR(mds); @@ -1732,7 +1732,6 @@ static int cxl_mock_mem_probe(struct platform_device *pdev) mds->event.buf = (struct cxl_get_event_payload *) mdata->event_buf; INIT_DELAYED_WORK(&mds->security.poll_dwork, cxl_mockmem_sanitize_work); - cxlds->serial = pdev->id + 1; if (is_rcd(pdev)) cxlds->rcd = true; From 72fa35948dfada0d7f4e4e8832afe39226e9aeab Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Fri, 6 Mar 2026 16:47:39 +0000 Subject: [PATCH 231/464] cxl: export internal structs for external Type2 drivers BugLink: https://bugs.launchpad.net/bugs/2153819 In preparation for type2 support, move structs and functions a type2 driver will need to access to into a new shared header file. Differentiate between public and private data to be preserved by type2 drivers. Signed-off-by: Alejandro Lucero Reviewed-by: Dave Jiang Tested-by: Alison Schofield Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Link: https://patch.msgid.link/20260306164741.3796372-3-alejandro.lucero-palau@amd.com Signed-off-by: Dave Jiang (cherry picked from commit 005869886d1d370afb6c10cd40709d956960e9c2) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/cxl.h | 97 +------------------ drivers/cxl/cxlmem.h | 114 ---------------------- include/cxl/cxl.h | 226 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 227 insertions(+), 210 deletions(-) create mode 100644 include/cxl/cxl.h diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 9b947286eb9b0..1d94217729f76 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -12,6 +12,7 @@ #include #include #include +#include extern const struct nvdimm_security_ops *cxl_security_ops; @@ -201,97 +202,6 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw) #define CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48) #define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20 -/* - * Using struct_group() allows for per register-block-type helper routines, - * without requiring block-type agnostic code to include the prefix. - */ -struct cxl_regs { - /* - * Common set of CXL Component register block base pointers - * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure - * @ras: CXL 2.0 8.2.5.9 CXL RAS Capability Structure - */ - struct_group_tagged(cxl_component_regs, component, - void __iomem *hdm_decoder; - void __iomem *ras; - ); - /* - * Common set of CXL Device register block base pointers - * @status: CXL 2.0 8.2.8.3 Device Status Registers - * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers - * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers - */ - struct_group_tagged(cxl_device_regs, device_regs, - void __iomem *status, *mbox, *memdev; - ); - - struct_group_tagged(cxl_pmu_regs, pmu_regs, - void __iomem *pmu; - ); - - /* - * RCH downstream port specific RAS register - * @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB - */ - struct_group_tagged(cxl_rch_regs, rch_regs, - void __iomem *dport_aer; - ); - - /* - * RCD upstream port specific PCIe cap register - * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB - */ - struct_group_tagged(cxl_rcd_regs, rcd_regs, - void __iomem *rcd_pcie_cap; - ); -}; - -struct cxl_reg_map { - bool valid; - int id; - unsigned long offset; - unsigned long size; -}; - -struct cxl_component_reg_map { - struct cxl_reg_map hdm_decoder; - struct cxl_reg_map ras; -}; - -struct cxl_device_reg_map { - struct cxl_reg_map status; - struct cxl_reg_map mbox; - struct cxl_reg_map memdev; -}; - -struct cxl_pmu_reg_map { - struct cxl_reg_map pmu; -}; - -/** - * struct cxl_register_map - DVSEC harvested register block mapping parameters - * @host: device for devm operations and logging - * @base: virtual base of the register-block-BAR + @block_offset - * @resource: physical resource base of the register block - * @max_size: maximum mapping size to perform register search - * @reg_type: see enum cxl_regloc_type - * @component_map: cxl_reg_map for component registers - * @device_map: cxl_reg_maps for device registers - * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units - */ -struct cxl_register_map { - struct device *host; - void __iomem *base; - resource_size_t resource; - resource_size_t max_size; - u8 reg_type; - union { - struct cxl_component_reg_map component_map; - struct cxl_device_reg_map device_map; - struct cxl_pmu_reg_map pmu_map; - }; -}; - void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map); void cxl_probe_device_regs(struct device *dev, void __iomem *base, @@ -497,11 +407,6 @@ struct cxl_region_params { resource_size_t cache_size; }; -enum cxl_partition_mode { - CXL_PARTMODE_RAM, - CXL_PARTMODE_PMEM, -}; - /* * Indicate whether this region has been assembled by autodetection or * userspace assembly. Prevent endpoint decoders outside of automatic diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 71367cb5178ca..281546de426e4 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -113,8 +113,6 @@ int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled, resource_size_t base, resource_size_t len, resource_size_t skipped); -#define CXL_NR_PARTITIONS_MAX 2 - struct cxl_dpa_info { u64 size; struct cxl_dpa_part_info { @@ -373,87 +371,6 @@ struct cxl_security_state { struct kernfs_node *sanitize_node; }; -/* - * enum cxl_devtype - delineate type-2 from a generic type-3 device - * @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or - * HDM-DB, no requirement that this device implements a - * mailbox, or other memory-device-standard manageability - * flows. - * @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with - * HDM-H and class-mandatory memory device registers - */ -enum cxl_devtype { - CXL_DEVTYPE_DEVMEM, - CXL_DEVTYPE_CLASSMEM, -}; - -/** - * struct cxl_dpa_perf - DPA performance property entry - * @dpa_range: range for DPA address - * @coord: QoS performance data (i.e. latency, bandwidth) - * @cdat_coord: raw QoS performance data from CDAT - * @qos_class: QoS Class cookies - */ -struct cxl_dpa_perf { - struct range dpa_range; - struct access_coordinate coord[ACCESS_COORDINATE_MAX]; - struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX]; - int qos_class; -}; - -/** - * struct cxl_dpa_partition - DPA partition descriptor - * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res) - * @perf: performance attributes of the partition from CDAT - * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic... - */ -struct cxl_dpa_partition { - struct resource res; - struct cxl_dpa_perf perf; - enum cxl_partition_mode mode; -}; - -/** - * struct cxl_dev_state - The driver device state - * - * cxl_dev_state represents the CXL driver/device state. It provides an - * interface to mailbox commands as well as some cached data about the device. - * Currently only memory devices are represented. - * - * @dev: The device associated with this CXL state - * @cxlmd: The device representing the CXL.mem capabilities of @dev - * @reg_map: component and ras register mapping parameters - * @regs: Class device "Device" registers - * @cxl_dvsec: Offset to the PCIe device DVSEC - * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH) - * @media_ready: Indicate whether the device media is usable - * @dpa_res: Overall DPA resource tree for the device - * @part: DPA partition array - * @nr_partitions: Number of DPA partitions - * @serial: PCIe Device Serial Number - * @type: Generic Memory Class device or Vendor Specific Memory device - * @cxl_mbox: CXL mailbox context - * @cxlfs: CXL features context - */ -struct cxl_dev_state { - struct device *dev; - struct cxl_memdev *cxlmd; - struct cxl_register_map reg_map; - struct cxl_device_regs regs; - int cxl_dvsec; - bool rcd; - bool media_ready; - struct resource dpa_res; - struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX]; - unsigned int nr_partitions; - u64 serial; - enum cxl_devtype type; - struct cxl_mailbox cxl_mbox; -#ifdef CONFIG_CXL_FEATURES - struct cxl_features_state *cxlfs; -#endif -}; - static inline resource_size_t cxl_pmem_size(struct cxl_dev_state *cxlds) { /* @@ -523,37 +440,6 @@ to_cxl_memdev_state(struct cxl_dev_state *cxlds) return container_of(cxlds, struct cxl_memdev_state, cxlds); } -struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, - enum cxl_devtype type, - u64 serial, u16 dvsec, - size_t size, bool has_mbox); - -/** - * cxl_dev_state_create - safely create and cast a cxl dev state embedded in a - * driver specific struct. - * - * @parent: device behind the request - * @type: CXL device type - * @serial: device identification - * @dvsec: dvsec capability offset - * @drv_struct: driver struct embedding a cxl_dev_state struct - * @member: name of the struct cxl_dev_state member in drv_struct - * @mbox: true if mailbox supported - * - * Returns a pointer to the drv_struct allocated and embedding a cxl_dev_state - * struct initialized. - * - * Introduced for Type2 driver support. - */ -#define devm_cxl_dev_state_create(parent, type, serial, dvsec, drv_struct, member, mbox) \ - ({ \ - static_assert(__same_type(struct cxl_dev_state, \ - ((drv_struct *)NULL)->member)); \ - static_assert(offsetof(drv_struct, member) == 0); \ - (drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec, \ - sizeof(drv_struct), mbox); \ - }) - enum cxl_opcode { CXL_MBOX_OP_INVALID = 0x0000, CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID, diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h new file mode 100644 index 0000000000000..fa72691546205 --- /dev/null +++ b/include/cxl/cxl.h @@ -0,0 +1,226 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2020 Intel Corporation. */ +/* Copyright(c) 2026 Advanced Micro Devices, Inc. */ + +#ifndef __CXL_CXL_H__ +#define __CXL_CXL_H__ + +#include +#include +#include + +/** + * enum cxl_devtype - delineate type-2 from a generic type-3 device + * @CXL_DEVTYPE_DEVMEM: Vendor specific CXL Type-2 device implementing HDM-D or + * HDM-DB, no requirement that this device implements a + * mailbox, or other memory-device-standard manageability + * flows. + * @CXL_DEVTYPE_CLASSMEM: Common class definition of a CXL Type-3 device with + * HDM-H and class-mandatory memory device registers + */ +enum cxl_devtype { + CXL_DEVTYPE_DEVMEM, + CXL_DEVTYPE_CLASSMEM, +}; + +struct device; + +/* + * Using struct_group() allows for per register-block-type helper routines, + * without requiring block-type agnostic code to include the prefix. + */ +struct cxl_regs { + /* + * Common set of CXL Component register block base pointers + * @hdm_decoder: CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure + * @ras: CXL 2.0 8.2.5.9 CXL RAS Capability Structure + */ + struct_group_tagged(cxl_component_regs, component, + void __iomem *hdm_decoder; + void __iomem *ras; + ); + /* + * Common set of CXL Device register block base pointers + * @status: CXL 2.0 8.2.8.3 Device Status Registers + * @mbox: CXL 2.0 8.2.8.4 Mailbox Registers + * @memdev: CXL 2.0 8.2.8.5 Memory Device Registers + */ + struct_group_tagged(cxl_device_regs, device_regs, + void __iomem *status, *mbox, *memdev; + ); + + struct_group_tagged(cxl_pmu_regs, pmu_regs, + void __iomem *pmu; + ); + + /* + * RCH downstream port specific RAS register + * @aer: CXL 3.0 8.2.1.1 RCH Downstream Port RCRB + */ + struct_group_tagged(cxl_rch_regs, rch_regs, + void __iomem *dport_aer; + ); + + /* + * RCD upstream port specific PCIe cap register + * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB + */ + struct_group_tagged(cxl_rcd_regs, rcd_regs, + void __iomem *rcd_pcie_cap; + ); +}; + +struct cxl_reg_map { + bool valid; + int id; + unsigned long offset; + unsigned long size; +}; + +struct cxl_component_reg_map { + struct cxl_reg_map hdm_decoder; + struct cxl_reg_map ras; +}; + +struct cxl_device_reg_map { + struct cxl_reg_map status; + struct cxl_reg_map mbox; + struct cxl_reg_map memdev; +}; + +struct cxl_pmu_reg_map { + struct cxl_reg_map pmu; +}; + +/** + * struct cxl_register_map - DVSEC harvested register block mapping parameters + * @host: device for devm operations and logging + * @base: virtual base of the register-block-BAR + @block_offset + * @resource: physical resource base of the register block + * @max_size: maximum mapping size to perform register search + * @reg_type: see enum cxl_regloc_type + * @component_map: cxl_reg_map for component registers + * @device_map: cxl_reg_maps for device registers + * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units + */ +struct cxl_register_map { + struct device *host; + void __iomem *base; + resource_size_t resource; + resource_size_t max_size; + u8 reg_type; + union { + struct cxl_component_reg_map component_map; + struct cxl_device_reg_map device_map; + struct cxl_pmu_reg_map pmu_map; + }; +}; + +/** + * struct cxl_dpa_perf - DPA performance property entry + * @dpa_range: range for DPA address + * @coord: QoS performance data (i.e. latency, bandwidth) + * @cdat_coord: raw QoS performance data from CDAT + * @qos_class: QoS Class cookies + */ +struct cxl_dpa_perf { + struct range dpa_range; + struct access_coordinate coord[ACCESS_COORDINATE_MAX]; + struct access_coordinate cdat_coord[ACCESS_COORDINATE_MAX]; + int qos_class; +}; + +enum cxl_partition_mode { + CXL_PARTMODE_RAM, + CXL_PARTMODE_PMEM, +}; + +/** + * struct cxl_dpa_partition - DPA partition descriptor + * @res: shortcut to the partition in the DPA resource tree (cxlds->dpa_res) + * @perf: performance attributes of the partition from CDAT + * @mode: operation mode for the DPA capacity, e.g. ram, pmem, dynamic... + */ +struct cxl_dpa_partition { + struct resource res; + struct cxl_dpa_perf perf; + enum cxl_partition_mode mode; +}; + +#define CXL_NR_PARTITIONS_MAX 2 + +/** + * struct cxl_dev_state - The driver device state + * + * cxl_dev_state represents the CXL driver/device state. It provides an + * interface to mailbox commands as well as some cached data about the device. + * Currently only memory devices are represented. + * + * @dev: The device associated with this CXL state + * @cxlmd: The device representing the CXL.mem capabilities of @dev + * @reg_map: component and ras register mapping parameters + * @regs: Parsed register blocks + * @cxl_dvsec: Offset to the PCIe device DVSEC + * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH) + * @media_ready: Indicate whether the device media is usable + * @dpa_res: Overall DPA resource tree for the device + * @part: DPA partition array + * @nr_partitions: Number of DPA partitions + * @serial: PCIe Device Serial Number + * @type: Generic Memory Class device or Vendor Specific Memory device + * @cxl_mbox: CXL mailbox context + * @cxlfs: CXL features context + */ +struct cxl_dev_state { + /* public for Type2 drivers */ + struct device *dev; + struct cxl_memdev *cxlmd; + + /* private for Type2 drivers */ + struct cxl_register_map reg_map; + struct cxl_device_regs regs; + int cxl_dvsec; + bool rcd; + bool media_ready; + struct resource dpa_res; + struct cxl_dpa_partition part[CXL_NR_PARTITIONS_MAX]; + unsigned int nr_partitions; + u64 serial; + enum cxl_devtype type; + struct cxl_mailbox cxl_mbox; +#ifdef CONFIG_CXL_FEATURES + struct cxl_features_state *cxlfs; +#endif +}; + +struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, + enum cxl_devtype type, + u64 serial, u16 dvsec, + size_t size, bool has_mbox); + +/** + * cxl_dev_state_create - safely create and cast a cxl dev state embedded in a + * driver specific struct. + * + * @parent: device behind the request + * @type: CXL device type + * @serial: device identification + * @dvsec: dvsec capability offset + * @drv_struct: driver struct embedding a cxl_dev_state struct + * @member: name of the struct cxl_dev_state member in drv_struct + * @mbox: true if mailbox supported + * + * Returns a pointer to the drv_struct allocated and embedding a cxl_dev_state + * struct initialized. + * + * Introduced for Type2 driver support. + */ +#define devm_cxl_dev_state_create(parent, type, serial, dvsec, drv_struct, member, mbox) \ + ({ \ + static_assert(__same_type(struct cxl_dev_state, \ + ((drv_struct *)NULL)->member)); \ + static_assert(offsetof(drv_struct, member) == 0); \ + (drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec, \ + sizeof(drv_struct), mbox); \ + }) +#endif /* __CXL_CXL_H__ */ From f285bd6757001ff1a0f6d6610ac60efd9f717766 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Fri, 6 Mar 2026 16:47:40 +0000 Subject: [PATCH 232/464] cxl: Move pci generic code from cxl_pci to core/cxl_pci BugLink: https://bugs.launchpad.net/bugs/2153819 Inside cxl/core/pci.c there are helpers for CXL PCIe initialization meanwhile cxl/pci_drv.c implements the functionality for a Type3 device initialization. In preparation for type2 support, move helper functions from cxl/pci.c to cxl/core/pci.c in order to be exported and used by type2 drivers. [ dj: Clarified subject. ] Signed-off-by: Alejandro Lucero Reviewed-by: Dave Jiang Reviewed-by: Gregory Price Reviewed-by: Jonathan Cameron Signed-off-by: Gregory Price Link: https://patch.msgid.link/20260306164741.3796372-4-alejandro.lucero-palau@amd.com Signed-off-by: Dave Jiang (cherry picked from commit 58f28930c7fb0e24cdf2972a9c3b7c91aeef4539) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/core.h | 2 ++ drivers/cxl/core/pci.c | 62 ++++++++++++++++++++++++++++++++++++ drivers/cxl/core/regs.c | 1 - drivers/cxl/cxl.h | 2 -- drivers/cxl/cxlpci.h | 13 ++++++++ drivers/cxl/pci.c | 70 ----------------------------------------- 6 files changed, 77 insertions(+), 73 deletions(-) diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 5b0570df0fd9c..5539e941782f6 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -224,4 +224,6 @@ int cxl_set_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid, u16 *return_code); #endif +resource_size_t cxl_rcd_component_reg_phys(struct device *dev, + struct cxl_dport *dport); #endif /* __CXL_CORE_H__ */ diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index f96ce884a2130..c32cc62c501de 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -696,6 +696,68 @@ bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port) } EXPORT_SYMBOL_NS_GPL(cxl_endpoint_decoder_reset_detected, "CXL"); +static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev, + struct cxl_register_map *map, + struct cxl_dport *dport) +{ + resource_size_t component_reg_phys; + + *map = (struct cxl_register_map) { + .host = &pdev->dev, + .resource = CXL_RESOURCE_NONE, + }; + + struct cxl_port *port __free(put_cxl_port) = + cxl_pci_find_port(pdev, &dport); + if (!port) + return -EPROBE_DEFER; + + component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport); + if (component_reg_phys == CXL_RESOURCE_NONE) + return -ENXIO; + + map->resource = component_reg_phys; + map->reg_type = CXL_REGLOC_RBI_COMPONENT; + map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE; + + return 0; +} + +int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, + struct cxl_register_map *map) +{ + int rc; + + rc = cxl_find_regblock(pdev, type, map); + + /* + * If the Register Locator DVSEC does not exist, check if it + * is an RCH and try to extract the Component Registers from + * an RCRB. + */ + if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) { + struct cxl_dport *dport; + struct cxl_port *port __free(put_cxl_port) = + cxl_pci_find_port(pdev, &dport); + if (!port) + return -EPROBE_DEFER; + + rc = cxl_rcrb_get_comp_regs(pdev, map, dport); + if (rc) + return rc; + + rc = cxl_dport_map_rcd_linkcap(pdev, dport); + if (rc) + return rc; + + } else if (rc) { + return rc; + } + + return cxl_setup_regs(map); +} +EXPORT_SYMBOL_NS_GPL(cxl_pci_setup_regs, "CXL"); + int cxl_pci_get_bandwidth(struct pci_dev *pdev, struct access_coordinate *c) { int speed, bw; diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index a010b32143422..93710cf4f0a69 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -641,4 +641,3 @@ resource_size_t cxl_rcd_component_reg_phys(struct device *dev, return CXL_RESOURCE_NONE; return __rcrb_to_component(dev, &dport->rcrb, CXL_RCRB_UPSTREAM); } -EXPORT_SYMBOL_NS_GPL(cxl_rcd_component_reg_phys, "CXL"); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 1d94217729f76..8194447f75d30 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -222,8 +222,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); int cxl_setup_regs(struct cxl_register_map *map); struct cxl_dport; -resource_size_t cxl_rcd_component_reg_phys(struct device *dev, - struct cxl_dport *dport); int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport); #define CXL_RESOURCE_NONE ((resource_size_t) -1) diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 0cf64218aa16e..b826eb53cf7ba 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -74,6 +74,17 @@ static inline bool cxl_pci_flit_256(struct pci_dev *pdev) return lnksta2 & PCI_EXP_LNKSTA2_FLIT; } +/* + * Assume that the caller has already validated that @pdev has CXL + * capabilities, any RCiEP with CXL capabilities is treated as a + * Restricted CXL Device (RCD) and finds upstream port and endpoint + * registers in a Root Complex Register Block (RCRB). + */ +static inline bool is_cxl_restricted(struct pci_dev *pdev) +{ + return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; +} + struct cxl_dev_state; void read_cdat_data(struct cxl_port *port); @@ -101,4 +112,6 @@ static inline void devm_cxl_port_ras_setup(struct cxl_port *port) } #endif +int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, + struct cxl_register_map *map); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index cfd6a0553e0fc..a878d9fb3e58e 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -465,76 +465,6 @@ static int cxl_pci_setup_mailbox(struct cxl_memdev_state *mds, bool irq_avail) return 0; } -/* - * Assume that any RCIEP that emits the CXL memory expander class code - * is an RCD - */ -static bool is_cxl_restricted(struct pci_dev *pdev) -{ - return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; -} - -static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev, - struct cxl_register_map *map, - struct cxl_dport *dport) -{ - resource_size_t component_reg_phys; - - *map = (struct cxl_register_map) { - .host = &pdev->dev, - .resource = CXL_RESOURCE_NONE, - }; - - struct cxl_port *port __free(put_cxl_port) = - cxl_pci_find_port(pdev, &dport); - if (!port) - return -EPROBE_DEFER; - - component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport); - if (component_reg_phys == CXL_RESOURCE_NONE) - return -ENXIO; - - map->resource = component_reg_phys; - map->reg_type = CXL_REGLOC_RBI_COMPONENT; - map->max_size = CXL_COMPONENT_REG_BLOCK_SIZE; - - return 0; -} - -static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, - struct cxl_register_map *map) -{ - int rc; - - rc = cxl_find_regblock(pdev, type, map); - - /* - * If the Register Locator DVSEC does not exist, check if it - * is an RCH and try to extract the Component Registers from - * an RCRB. - */ - if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) { - struct cxl_dport *dport; - struct cxl_port *port __free(put_cxl_port) = - cxl_pci_find_port(pdev, &dport); - if (!port) - return -EPROBE_DEFER; - - rc = cxl_rcrb_get_comp_regs(pdev, map, dport); - if (rc) - return rc; - - rc = cxl_dport_map_rcd_linkcap(pdev, dport); - if (rc) - return rc; - - } else if (rc) { - return rc; - } - - return cxl_setup_regs(map); -} - static void free_event_buf(void *buf) { kvfree(buf); From 07cb7e64ac2a9fa272877525c47f754c1a41f029 Mon Sep 17 00:00:00 2001 From: Gregory Price Date: Fri, 6 Mar 2026 16:47:41 +0000 Subject: [PATCH 233/464] cxl/pci: Remove redundant cxl_pci_find_port() call BugLink: https://bugs.launchpad.net/bugs/2153819 Remove the redundant port lookup from cxl_rcrb_get_comp_regs() and use the dport parameter directly. The caller has already validated the port is non-NULL before invoking this function, and dport is given as a param. This is simpler than getting dport in the callee and return the pointer to the caller what would require more changes. Signed-off-by: Gregory Price Reviewed-by: Alejandro Lucero Reviewed-by: Jonathan Cameron Reviewed-by: Davidlohr Bueso Link: https://patch.msgid.link/20260306164741.3796372-5-alejandro.lucero-palau@amd.com Signed-off-by: Dave Jiang (cherry picked from commit d537d953c47866bafc89feb66d8ef34baf17659a) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/pci.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index c32cc62c501de..d1f487b3d809a 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -707,11 +707,6 @@ static int cxl_rcrb_get_comp_regs(struct pci_dev *pdev, .resource = CXL_RESOURCE_NONE, }; - struct cxl_port *port __free(put_cxl_port) = - cxl_pci_find_port(pdev, &dport); - if (!port) - return -EPROBE_DEFER; - component_reg_phys = cxl_rcd_component_reg_phys(&pdev->dev, dport); if (component_reg_phys == CXL_RESOURCE_NONE) return -ENXIO; From 056139ddc0d13c8a6ab341b72d0cd730e037bde0 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:21 +0100 Subject: [PATCH 234/464] NVIDIA: VR: SAUCE: sfc: add cxl support BugLink: https://bugs.launchpad.net/bugs/2153819 Add CXL initialization based on new CXL API for accel drivers and make it dependent on kernel CXL configuration. Signed-off-by: Alejandro Lucero Reviewed-by: Jonathan Cameron Acked-by: Edward Cree Reviewed-by: Alison Schofield Reviewed-by: Dan Williams Reviewed-by: Dave Jiang Reviewed-by: Ben Cheatham (cherry picked from https://lore.kernel.org/r/20260423180528.17166-2-alejandro.lucero-palau@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/sfc/Kconfig | 9 +++++ drivers/net/ethernet/sfc/Makefile | 1 + drivers/net/ethernet/sfc/efx.c | 14 +++++++- drivers/net/ethernet/sfc/efx_cxl.c | 52 +++++++++++++++++++++++++++ drivers/net/ethernet/sfc/efx_cxl.h | 29 +++++++++++++++ drivers/net/ethernet/sfc/net_driver.h | 10 ++++++ 6 files changed, 114 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/sfc/efx_cxl.c create mode 100644 drivers/net/ethernet/sfc/efx_cxl.h diff --git a/drivers/net/ethernet/sfc/Kconfig b/drivers/net/ethernet/sfc/Kconfig index c4c43434f3143..979f2801e2a8e 100644 --- a/drivers/net/ethernet/sfc/Kconfig +++ b/drivers/net/ethernet/sfc/Kconfig @@ -66,6 +66,15 @@ config SFC_MCDI_LOGGING Driver-Interface) commands and responses, allowing debugging of driver/firmware interaction. The tracing is actually enabled by a sysfs file 'mcdi_logging' under the PCI device. +config SFC_CXL + bool "Solarflare SFC9100-family CXL support" + depends on SFC && CXL_BUS >= SFC + default SFC + help + This enables SFC CXL support if the kernel is configuring CXL for + using CTPIO with CXL.mem. The SFC device with CXL support and + with a CXL-aware firmware can be used for minimizing latencies + when sending through CTPIO. source "drivers/net/ethernet/sfc/falcon/Kconfig" source "drivers/net/ethernet/sfc/siena/Kconfig" diff --git a/drivers/net/ethernet/sfc/Makefile b/drivers/net/ethernet/sfc/Makefile index d99039ec468d6..bb0f1891cde65 100644 --- a/drivers/net/ethernet/sfc/Makefile +++ b/drivers/net/ethernet/sfc/Makefile @@ -13,6 +13,7 @@ sfc-$(CONFIG_SFC_SRIOV) += sriov.o ef10_sriov.o ef100_sriov.o ef100_rep.o \ mae.o tc.o tc_bindings.o tc_counters.o \ tc_encap_actions.o tc_conntrack.o +sfc-$(CONFIG_SFC_CXL) += efx_cxl.o obj-$(CONFIG_SFC) += sfc.o obj-$(CONFIG_SFC_FALCON) += falcon/ diff --git a/drivers/net/ethernet/sfc/efx.c b/drivers/net/ethernet/sfc/efx.c index 8f136a11d3968..90ccbe3103860 100644 --- a/drivers/net/ethernet/sfc/efx.c +++ b/drivers/net/ethernet/sfc/efx.c @@ -34,6 +34,7 @@ #include "selftest.h" #include "sriov.h" #include "efx_devlink.h" +#include "efx_cxl.h" #include "mcdi_port_common.h" #include "mcdi_pcol.h" @@ -981,12 +982,14 @@ static void efx_pci_remove(struct pci_dev *pci_dev) efx_pci_remove_main(efx); efx_fini_io(efx); + + probe_data = container_of(efx, struct efx_probe_data, efx); + pci_dbg(efx->pci_dev, "shutdown successful\n"); efx_fini_devlink_and_unlock(efx); efx_fini_struct(efx); free_netdev(efx->net_dev); - probe_data = container_of(efx, struct efx_probe_data, efx); kfree(probe_data); }; @@ -1190,6 +1193,15 @@ static int efx_pci_probe(struct pci_dev *pci_dev, if (rc) goto fail2; + /* A successful cxl initialization implies a CXL region created to be + * used for PIO buffers. If there is no CXL support, or initialization + * fails, cxl_pio_initialised will be false and legacy PIO buffers + * defined at specific PCI BAR regions will be used. + */ + rc = efx_cxl_init(probe_data); + if (rc) + pci_err(pci_dev, "CXL initialization failed with error %d\n", rc); + rc = efx_pci_probe_post_io(efx); if (rc) { /* On failure, retry once immediately. diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c new file mode 100644 index 0000000000000..b7e8d85a43d37 --- /dev/null +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0-only +/**************************************************************************** + * + * Driver for AMD network controllers and boards + * Copyright (C) 2025, Advanced Micro Devices, Inc. + */ + +#include + +#include "net_driver.h" +#include "efx_cxl.h" + +#define EFX_CTPIO_BUFFER_SIZE SZ_256M + +int efx_cxl_init(struct efx_probe_data *probe_data) +{ + struct efx_nic *efx = &probe_data->efx; + struct pci_dev *pci_dev = efx->pci_dev; + struct efx_cxl *cxl; + u16 dvsec; + + probe_data->cxl_pio_initialised = false; + + /* Is the device configured with and using CXL? */ + if (!pcie_is_cxl(pci_dev)) + return 0; + + dvsec = pci_find_dvsec_capability(pci_dev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) { + pci_info(pci_dev, "CXL_DVSEC_PCIE_DEVICE capability not found\n"); + return 0; + } + + pci_dbg(pci_dev, "CXL_DVSEC_PCIE_DEVICE capability found\n"); + + /* Create a cxl_dev_state embedded in the cxl struct using cxl core api + * specifying no mbox available. + */ + cxl = devm_cxl_dev_state_create(&pci_dev->dev, CXL_DEVTYPE_DEVMEM, + pci_get_dsn(pci_dev), dvsec, + struct efx_cxl, cxlds, false); + + if (!cxl) + return -ENOMEM; + + probe_data->cxl = cxl; + + return 0; +} + +MODULE_IMPORT_NS("CXL"); diff --git a/drivers/net/ethernet/sfc/efx_cxl.h b/drivers/net/ethernet/sfc/efx_cxl.h new file mode 100644 index 0000000000000..04e46278464df --- /dev/null +++ b/drivers/net/ethernet/sfc/efx_cxl.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/**************************************************************************** + * Driver for AMD network controllers and boards + * Copyright (C) 2025, Advanced Micro Devices, Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published + * by the Free Software Foundation, incorporated herein by reference. + */ + +#ifndef EFX_CXL_H +#define EFX_CXL_H + +#ifdef CONFIG_SFC_CXL + +#include + +struct efx_probe_data; + +struct efx_cxl { + struct cxl_dev_state cxlds; + struct cxl_memdev *cxlmd; +}; + +int efx_cxl_init(struct efx_probe_data *probe_data); +#else +static inline int efx_cxl_init(struct efx_probe_data *probe_data) { return 0; } +#endif +#endif diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h index b98c259f672db..3964b2c56609c 100644 --- a/drivers/net/ethernet/sfc/net_driver.h +++ b/drivers/net/ethernet/sfc/net_driver.h @@ -1197,14 +1197,24 @@ struct efx_nic { atomic_t n_rx_noskb_drops; }; +#ifdef CONFIG_SFC_CXL +struct efx_cxl; +#endif + /** * struct efx_probe_data - State after hardware probe * @pci_dev: The PCI device * @efx: Efx NIC details + * @cxl: details of related cxl objects + * @cxl_pio_initialised: cxl initialization outcome. */ struct efx_probe_data { struct pci_dev *pci_dev; struct efx_nic efx; +#ifdef CONFIG_SFC_CXL + struct efx_cxl *cxl; + bool cxl_pio_initialised; +#endif }; static inline struct efx_nic *efx_netdev_priv(struct net_device *dev) From aed169a9228f263456adcee841e7a41774a52d97 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Wed, 13 May 2026 23:39:25 +0800 Subject: [PATCH 235/464] NVIDIA: VR: SAUCE: cxl/sfc: Map cxl regs BugLink: https://bugs.launchpad.net/bugs/2153819 Export cxl core functions for a Type2 driver being able to discover and map the device registers. Use it in sfc driver cxl initialization. Signed-off-by: Alejandro Lucero Reviewed-by: Dan Williams Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Reviewed-by: Ben Cheatham (backported from https://lore.kernel.org/r/20260423180528.17166-3-alejandro.lucero-palau@amd.com) [kobak: Kept cxl_pci_setup_regs() in the core/pci provider added by the full Type2 prerequisite series and dropped the duplicate provider hunk from drivers/cxl/pci.c.] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/pci.c | 1 + drivers/cxl/core/port.c | 1 + drivers/cxl/core/regs.c | 1 + drivers/cxl/cxlpci.h | 10 ---------- drivers/cxl/pci.c | 1 + drivers/net/ethernet/sfc/efx_cxl.c | 26 ++++++++++++++++++++++++++ include/cxl/pci.h | 22 ++++++++++++++++++++++ 7 files changed, 52 insertions(+), 10 deletions(-) create mode 100644 include/cxl/pci.h diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index d1f487b3d809a..2bcd683aa286d 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index c5aacd7054f1d..dbe30e7c383be 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index 93710cf4f0a69..20c2d9fbcfe7d 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index b826eb53cf7ba..224636588f623 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -13,16 +13,6 @@ */ #define CXL_PCI_DEFAULT_MAX_VECTORS 16 -/* Register Block Identifier (RBI) */ -enum cxl_regloc_type { - CXL_REGLOC_RBI_EMPTY = 0, - CXL_REGLOC_RBI_COMPONENT, - CXL_REGLOC_RBI_VIRT, - CXL_REGLOC_RBI_MEMDEV, - CXL_REGLOC_RBI_PMU, - CXL_REGLOC_RBI_TYPES -}; - /* * Table Access DOE, CDAT Read Entry Response * diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index a878d9fb3e58e..9e9239432b4d0 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include "cxlmem.h" #include "cxlpci.h" diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c index b7e8d85a43d37..f88c2b3fbb2c3 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.c +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -7,6 +7,8 @@ #include +#include +#include #include "net_driver.h" #include "efx_cxl.h" @@ -18,6 +20,7 @@ int efx_cxl_init(struct efx_probe_data *probe_data) struct pci_dev *pci_dev = efx->pci_dev; struct efx_cxl *cxl; u16 dvsec; + int rc; probe_data->cxl_pio_initialised = false; @@ -44,6 +47,29 @@ int efx_cxl_init(struct efx_probe_data *probe_data) if (!cxl) return -ENOMEM; + rc = cxl_pci_setup_regs(pci_dev, CXL_REGLOC_RBI_COMPONENT, + &cxl->cxlds.reg_map); + if (rc) { + pci_err(pci_dev, "No component registers\n"); + return rc; + } + + if (!cxl->cxlds.reg_map.component_map.hdm_decoder.valid) { + pci_err(pci_dev, "Expected HDM component register not found\n"); + return -ENODEV; + } + + if (!cxl->cxlds.reg_map.component_map.ras.valid) { + pci_err(pci_dev, "Expected RAS component register not found\n"); + return -ENODEV; + } + + /* Set media ready explicitly as there are neither mailbox for checking + * this state nor the CXL register involved, both not mandatory for + * type2. + */ + cxl->cxlds.media_ready = true; + probe_data->cxl = cxl; return 0; diff --git a/include/cxl/pci.h b/include/cxl/pci.h new file mode 100644 index 0000000000000..3e0000015871a --- /dev/null +++ b/include/cxl/pci.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. All rights reserved. */ + +#ifndef __CXL_CXL_PCI_H__ +#define __CXL_CXL_PCI_H__ + +/* Register Block Identifier (RBI) */ +enum cxl_regloc_type { + CXL_REGLOC_RBI_EMPTY = 0, + CXL_REGLOC_RBI_COMPONENT, + CXL_REGLOC_RBI_VIRT, + CXL_REGLOC_RBI_MEMDEV, + CXL_REGLOC_RBI_PMU, + CXL_REGLOC_RBI_TYPES +}; + +struct cxl_register_map; +struct pci_dev; + +int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, + struct cxl_register_map *map); +#endif From 26b00dd227a524753a2c347648ad5f85f7311609 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:23 +0100 Subject: [PATCH 236/464] NVIDIA: VR: SAUCE: cxl/sfc: Initialize dpa without a mailbox BugLink: https://bugs.launchpad.net/bugs/2153819 Type3 relies on mailbox CXL_MBOX_OP_IDENTIFY command for initializing memdev state params which end up being used for DPA initialization. Allow a Type2 driver to initialize DPA simply by giving the size of its volatile hardware partition. Move related functions to memdev. Add sfc driver as the client. Signed-off-by: Alejandro Lucero Reviewed-by: Dan Williams Reviewed-by: Dave Jiang Reviewed-by: Ben Cheatham Reviewed-by: Jonathan Cameron (cherry picked from https://lore.kernel.org/r/20260423180528.17166-4-alejandro.lucero-palau@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/core.h | 2 + drivers/cxl/core/mbox.c | 51 +---------------------- drivers/cxl/core/memdev.c | 66 ++++++++++++++++++++++++++++++ drivers/net/ethernet/sfc/efx_cxl.c | 5 +++ include/cxl/cxl.h | 2 + 5 files changed, 76 insertions(+), 50 deletions(-) diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 5539e941782f6..77c684744771b 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -97,6 +97,8 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr, struct dentry *cxl_debugfs_create_dir(const char *dir); int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled, enum cxl_partition_mode mode); +struct cxl_memdev_state; +int cxl_mem_get_partition_info(struct cxl_memdev_state *mds); int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size); int cxl_dpa_free(struct cxl_endpoint_decoder *cxled); resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled); diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index aa3724f51ce93..bcf77e604dc3d 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1151,7 +1151,7 @@ EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, "CXL"); * * See CXL @8.2.9.5.2.1 Get Partition Info */ -static int cxl_mem_get_partition_info(struct cxl_memdev_state *mds) +int cxl_mem_get_partition_info(struct cxl_memdev_state *mds) { struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox; struct cxl_mbox_get_partition_info pi; @@ -1307,55 +1307,6 @@ int cxl_mem_sanitize(struct cxl_memdev *cxlmd, u16 cmd) return -EBUSY; } -static void add_part(struct cxl_dpa_info *info, u64 start, u64 size, enum cxl_partition_mode mode) -{ - int i = info->nr_partitions; - - if (size == 0) - return; - - info->part[i].range = (struct range) { - .start = start, - .end = start + size - 1, - }; - info->part[i].mode = mode; - info->nr_partitions++; -} - -int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info) -{ - struct cxl_dev_state *cxlds = &mds->cxlds; - struct device *dev = cxlds->dev; - int rc; - - if (!cxlds->media_ready) { - info->size = 0; - return 0; - } - - info->size = mds->total_bytes; - - if (mds->partition_align_bytes == 0) { - add_part(info, 0, mds->volatile_only_bytes, CXL_PARTMODE_RAM); - add_part(info, mds->volatile_only_bytes, - mds->persistent_only_bytes, CXL_PARTMODE_PMEM); - return 0; - } - - rc = cxl_mem_get_partition_info(mds); - if (rc) { - dev_err(dev, "Failed to query partition information\n"); - return rc; - } - - add_part(info, 0, mds->active_volatile_bytes, CXL_PARTMODE_RAM); - add_part(info, mds->active_volatile_bytes, mds->active_persistent_bytes, - CXL_PARTMODE_PMEM); - - return 0; -} -EXPORT_SYMBOL_NS_GPL(cxl_mem_dpa_fetch, "CXL"); - int cxl_get_dirty_count(struct cxl_memdev_state *mds, u32 *count) { struct cxl_mailbox *cxl_mbox = &mds->cxlds.cxl_mbox; diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 99e422594885a..166ec6b7d0416 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -582,6 +582,72 @@ bool is_cxl_memdev(const struct device *dev) } EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, "CXL"); +static void add_part(struct cxl_dpa_info *info, u64 start, u64 size, enum cxl_partition_mode mode) +{ + int i = info->nr_partitions; + + if (size == 0) + return; + + info->part[i].range = (struct range) { + .start = start, + .end = start + size - 1, + }; + info->part[i].mode = mode; + info->nr_partitions++; +} + +int cxl_mem_dpa_fetch(struct cxl_memdev_state *mds, struct cxl_dpa_info *info) +{ + struct cxl_dev_state *cxlds = &mds->cxlds; + struct device *dev = cxlds->dev; + int rc; + + if (!cxlds->media_ready) { + info->size = 0; + return 0; + } + + info->size = mds->total_bytes; + + if (mds->partition_align_bytes == 0) { + add_part(info, 0, mds->volatile_only_bytes, CXL_PARTMODE_RAM); + add_part(info, mds->volatile_only_bytes, + mds->persistent_only_bytes, CXL_PARTMODE_PMEM); + return 0; + } + + rc = cxl_mem_get_partition_info(mds); + if (rc) { + dev_err(dev, "Failed to query partition information\n"); + return rc; + } + + add_part(info, 0, mds->active_volatile_bytes, CXL_PARTMODE_RAM); + add_part(info, mds->active_volatile_bytes, mds->active_persistent_bytes, + CXL_PARTMODE_PMEM); + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_mem_dpa_fetch, "CXL"); + +/** + * cxl_set_capacity: initialize dpa by a driver without a mailbox. + * + * @cxlds: pointer to cxl_dev_state + * @capacity: device volatile memory size + */ +int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity) +{ + struct cxl_dpa_info range_info = { + .size = capacity, + }; + + add_part(&range_info, 0, capacity, CXL_PARTMODE_RAM); + return cxl_dpa_setup(cxlds, &range_info); +} +EXPORT_SYMBOL_NS_GPL(cxl_set_capacity, "CXL"); + /** * set_exclusive_cxl_commands() - atomically disable user cxl commands * @mds: The device state to operate on diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c index f88c2b3fbb2c3..4d55c08cf2a16 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.c +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -70,6 +70,11 @@ int efx_cxl_init(struct efx_probe_data *probe_data) */ cxl->cxlds.media_ready = true; + if (cxl_set_capacity(&cxl->cxlds, EFX_CTPIO_BUFFER_SIZE)) { + pci_err(pci_dev, "dpa capacity setup failed\n"); + return -ENODEV; + } + probe_data->cxl = cxl; return 0; diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index fa72691546205..1346771edc4e4 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -223,4 +223,6 @@ struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, (drv_struct *)_devm_cxl_dev_state_create(parent, type, serial, dvsec, \ sizeof(drv_struct), mbox); \ }) + +int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity); #endif /* __CXL_CXL_H__ */ From bd04414cbc2e5520ae76aa4bf418200c752056f9 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:24 +0100 Subject: [PATCH 237/464] NVIDIA: VR: SAUCE: cxl: Prepare memdev creation for type2 BugLink: https://bugs.launchpad.net/bugs/2153819 Current cxl core is relying on a CXL_DEVTYPE_CLASSMEM type device when creating a memdev leading to problems when obtaining cxl_memdev_state references from a CXL_DEVTYPE_DEVMEM type. Modify check for obtaining cxl_memdev_state adding CXL_DEVTYPE_DEVMEM support. Make devm_cxl_add_memdev accessible from an accel driver. Signed-off-by: Alejandro Lucero Reviewed-by: Ben Cheatham Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Reviewed-by: Alison Schofield Reviewed-by: Dan Williams (cherry picked from https://lore.kernel.org/r/20260423180528.17166-5-alejandro.lucero-palau@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/memdev.c | 15 +++++++++++-- drivers/cxl/cxlmem.h | 6 ------ drivers/cxl/mem.c | 45 +++++++++++++++++++++++++++++---------- include/cxl/cxl.h | 6 ++++++ 4 files changed, 53 insertions(+), 19 deletions(-) diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 166ec6b7d0416..759b43364ed7a 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include "trace.h" #include "core.h" @@ -576,9 +577,16 @@ static const struct device_type cxl_memdev_type = { .groups = cxl_memdev_attribute_groups, }; +static const struct device_type cxl_accel_memdev_type = { + .name = "cxl_accel_memdev", + .release = cxl_memdev_release, + .devnode = cxl_memdev_devnode, +}; + bool is_cxl_memdev(const struct device *dev) { - return dev->type == &cxl_memdev_type; + return (dev->type == &cxl_memdev_type || + dev->type == &cxl_accel_memdev_type); } EXPORT_SYMBOL_NS_GPL(is_cxl_memdev, "CXL"); @@ -773,7 +781,10 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_dev_state *cxlds, dev->parent = cxlds->dev; dev->bus = &cxl_bus_type; dev->devt = MKDEV(cxl_mem_major, cxlmd->id); - dev->type = &cxl_memdev_type; + if (cxlds->type == CXL_DEVTYPE_DEVMEM) + dev->type = &cxl_accel_memdev_type; + else + dev->type = &cxl_memdev_type; device_set_pm_not_required(dev); INIT_WORK(&cxlmd->detach_work, detach_memdev); diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 281546de426e4..c98db6f18aa29 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -34,10 +34,6 @@ (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \ CXLMDEV_RESET_NEEDED_NOT) -struct cxl_memdev_attach { - int (*probe)(struct cxl_memdev *cxlmd); -}; - /** * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device * @dev: driver core device object @@ -103,8 +99,6 @@ static inline bool is_cxl_endpoint(struct cxl_port *port) struct cxl_memdev *__devm_cxl_add_memdev(struct cxl_dev_state *cxlds, const struct cxl_memdev_attach *attach); -struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, - const struct cxl_memdev_attach *attach); int devm_cxl_sanitize_setup_notifier(struct device *host, struct cxl_memdev *cxlmd); struct cxl_memdev_state; diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index fcffe24dcb42f..ff858318091f1 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -65,6 +65,26 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa) DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL, cxl_debugfs_poison_clear, "%llx\n"); +static void cxl_memdev_poison_enable(struct cxl_memdev_state *mds, + struct cxl_memdev *cxlmd, + struct dentry *dentry) +{ + /* + * Avoid poison debugfs for DEVMEM aka accelerators as they rely on + * cxl_memdev_state. + */ + if (!mds) + return; + + if (test_bit(CXL_POISON_ENABLED_INJECT, mds->poison.enabled_cmds)) + debugfs_create_file("inject_poison", 0200, dentry, cxlmd, + &cxl_poison_inject_fops); + + if (test_bit(CXL_POISON_ENABLED_CLEAR, mds->poison.enabled_cmds)) + debugfs_create_file("clear_poison", 0200, dentry, cxlmd, + &cxl_poison_clear_fops); +} + static int cxl_mem_probe(struct device *dev) { struct cxl_memdev *cxlmd = to_cxl_memdev(dev); @@ -92,12 +112,7 @@ static int cxl_mem_probe(struct device *dev) dentry = cxl_debugfs_create_dir(dev_name(dev)); debugfs_create_devm_seqfile(dev, "dpamem", dentry, cxl_mem_dpa_show); - if (test_bit(CXL_POISON_ENABLED_INJECT, mds->poison.enabled_cmds)) - debugfs_create_file("inject_poison", 0200, dentry, cxlmd, - &cxl_poison_inject_fops); - if (test_bit(CXL_POISON_ENABLED_CLEAR, mds->poison.enabled_cmds)) - debugfs_create_file("clear_poison", 0200, dentry, cxlmd, - &cxl_poison_clear_fops); + cxl_memdev_poison_enable(mds, cxlmd, dentry); rc = devm_add_action_or_reset(dev, remove_debugfs, dentry); if (rc) @@ -206,16 +221,24 @@ static ssize_t trigger_poison_list_store(struct device *dev, } static DEVICE_ATTR_WO(trigger_poison_list); -static umode_t cxl_mem_visible(struct kobject *kobj, struct attribute *a, int n) +static bool cxl_poison_attr_visible(struct kobject *kobj, struct attribute *a) { struct device *dev = kobj_to_dev(kobj); struct cxl_memdev *cxlmd = to_cxl_memdev(dev); struct cxl_memdev_state *mds = to_cxl_memdev_state(cxlmd->cxlds); - if (a == &dev_attr_trigger_poison_list.attr) - if (!test_bit(CXL_POISON_ENABLED_LIST, - mds->poison.enabled_cmds)) - return 0; + if (!mds || + !test_bit(CXL_POISON_ENABLED_LIST, mds->poison.enabled_cmds)) + return false; + + return true; +} + +static umode_t cxl_mem_visible(struct kobject *kobj, struct attribute *a, int n) +{ + if (a == &dev_attr_trigger_poison_list.attr && + !cxl_poison_attr_visible(kobj, a)) + return 0; return a->mode; } diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 1346771edc4e4..10a9b8fa2f6b7 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -149,6 +149,10 @@ struct cxl_dpa_partition { #define CXL_NR_PARTITIONS_MAX 2 +struct cxl_memdev_attach { + int (*probe)(struct cxl_memdev *cxlmd); +}; + /** * struct cxl_dev_state - The driver device state * @@ -225,4 +229,6 @@ struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, }) int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity); +struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, + const struct cxl_memdev_attach *attach); #endif /* __CXL_CXL_H__ */ From f9adf3657417734763628f08019289b8fdda75a0 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:25 +0100 Subject: [PATCH 238/464] NVIDIA: VR: SAUCE: sfc: create type2 cxl memdev BugLink: https://bugs.launchpad.net/bugs/2153819 Use cxl API for creating a cxl memory device using the type2 cxl_dev_state struct. Signed-off-by: Alejandro Lucero Reviewed-by: Martin Habets Reviewed-by: Fan Ni Acked-by: Edward Cree Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang (cherry picked from https://lore.kernel.org/r/20260423180528.17166-6-alejandro.lucero-palau@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/sfc/efx_cxl.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c index 4d55c08cf2a16..7d8a6c2133c88 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.c +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -75,6 +75,12 @@ int efx_cxl_init(struct efx_probe_data *probe_data) return -ENODEV; } + cxl->cxlmd = devm_cxl_add_memdev(&cxl->cxlds, NULL); + if (IS_ERR(cxl->cxlmd)) { + pci_err(pci_dev, "CXL accel memdev creation failed\n"); + return PTR_ERR(cxl->cxlmd); + } + probe_data->cxl = cxl; return 0; From ef9e0f936cd58f6a39c38e709134959aa5ce6bc1 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:26 +0100 Subject: [PATCH 239/464] NVIDIA: VR: SAUCE: cxl: attach region to an accelerator/type2 memdev BugLink: https://bugs.launchpad.net/bugs/2153819 Support an accelerator driver to safely work with an autodiscovered region from a committed HDM decoder through: 1) an accelerator driver cxl_attach_region struct with attach and detach callbacks. 2) a specific function, cxl_memdev_attach_region() keeping the required locks for finding a region linked to the memdev endpoint, and 3) invoking attach callback while keeping the locking allowing to work (ioremap and other internal stuff) with the related physical range by the accelerator driver, and 4) linking a detach callback to the endpoint device removal where the accelerator driver can stop using the region range. This covers the cases of a potential removal of cxl_acpi module or a accelerator memdev unbinding from cxl_mem driver through sysfs. Signed-off-by: Alejandro Lucero (backported from https://lore.kernel.org/r/20260423180528.17166-7-alejandro.lucero-palau@amd.com) [kobak: Check cxl_memdev_attach_region() errors and propagate failure so SFC probe does not continue after CXL core tears down the attached region. Set probe_data->cxl before attaching so the attach callback can use it, guard attach attempts before a valid endpoint exists, explicitly unwind attach/autoremove side effects if devres action registration fails, preserve DEVMEM target type for autodiscovered regions, and route delete / construct-failure cleanup through endpoint-owned devres actions.] [kobak: Keep no-detach DEVMEM unregister under the endpoint-device guard so attach cannot install endpoint devres actions for a region being freed.] [kobak: Avoid devres-registration failure cleanup under cxl_rwsem.region read lock: keep endpoint->dev locked, drop the region/DPA read guards before unregister_region(), and use devm_remove_action() so failed detach-action registration does not run cxl_endpoint_region_autoremove() under the read lock.] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/region.c | 193 +++++++++++++++++++++++++++-- drivers/cxl/cxl.h | 4 + drivers/net/ethernet/sfc/efx_cxl.c | 37 ++++++ drivers/net/ethernet/sfc/efx_cxl.h | 2 + include/cxl/cxl.h | 17 +++ 5 files changed, 242 insertions(+), 11 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 9ad83813a5e60..f8fdc0ec27316 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -2467,6 +2467,41 @@ static void unregister_region(void *_cxlr) put_device(&cxlr->dev); } +static void cxl_endpoint_region_autoremove(void *_cxlr); + +static void cxl_region_release_action(struct cxl_region *cxlr) +{ + struct cxl_port *port = cxlrd_to_port(cxlr->cxlrd); + + if (cxlr->type != CXL_DECODER_DEVMEM) { + devm_release_action(port->uport_dev, unregister_region, cxlr); + return; + } + + if (cxlr->params.nr_targets) { + struct cxl_endpoint_decoder *cxled = cxlr->params.targets[0]; + struct cxl_port *endpoint = cxled_to_port(cxled); + + guard(device)(&endpoint->dev); + if (cxlr->detach) { + void (*detach)(void *data) = cxlr->detach; + void *detach_data = cxlr->detach_data; + + cxlr->detach = NULL; + cxlr->detach_data = NULL; + devm_release_action(&endpoint->dev, detach, detach_data); + devm_release_action(&endpoint->dev, + cxl_endpoint_region_autoremove, + cxlr); + } else { + unregister_region(cxlr); + } + return; + } + + unregister_region(cxlr); +} + static struct lock_class_key cxl_region_key; static struct cxl_region *cxl_region_alloc(struct cxl_root_decoder *cxlrd, int id) @@ -2619,9 +2654,16 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd, if (rc) goto err; - rc = devm_add_action_or_reset(port->uport_dev, unregister_region, cxlr); - if (rc) - return ERR_PTR(rc); + /* + * For accelerators/type2, region release linked to endpoint device. + * See handling of cxl_endpoint_region_autoremove() below by + * cxl_memdev_attach_region(). + */ + if (type == CXL_DECODER_HOSTONLYMEM) { + rc = devm_add_action_or_reset(port->uport_dev, unregister_region, cxlr); + if (rc) + return ERR_PTR(rc); + } dev_dbg(port->uport_dev, "%s: created %s\n", dev_name(&cxlrd->cxlsd.cxld.dev), dev_name(dev)); @@ -2650,7 +2692,8 @@ static ssize_t create_ram_region_show(struct device *dev, } static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, - enum cxl_partition_mode mode, int id) + enum cxl_partition_mode mode, int id, + enum cxl_decoder_type type) { int rc; @@ -2672,7 +2715,7 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, return ERR_PTR(-EBUSY); } - return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM); + return devm_cxl_add_region(cxlrd, id, mode, type); } static ssize_t create_region_store(struct device *dev, const char *buf, @@ -2686,7 +2729,7 @@ static ssize_t create_region_store(struct device *dev, const char *buf, if (rc != 1) return -EINVAL; - cxlr = __create_region(cxlrd, mode, id); + cxlr = __create_region(cxlrd, mode, id, CXL_DECODER_HOSTONLYMEM); if (IS_ERR(cxlr)) return PTR_ERR(cxlr); @@ -2743,14 +2786,13 @@ static ssize_t delete_region_store(struct device *dev, const char *buf, size_t len) { struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev); - struct cxl_port *port = to_cxl_port(dev->parent); struct cxl_region *cxlr; cxlr = cxl_find_region_by_name(cxlrd, buf); if (IS_ERR(cxlr)) return PTR_ERR(cxlr); - devm_release_action(port->uport_dev, unregister_region, cxlr); + cxl_region_release_action(cxlr); put_device(&cxlr->dev); return len; @@ -3897,7 +3939,6 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, { struct cxl_endpoint_decoder *cxled = ctx->cxled; struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); - struct cxl_port *port = cxlrd_to_port(cxlrd); struct cxl_dev_state *cxlds = cxlmd->cxlds; int rc, part = READ_ONCE(cxled->part); struct cxl_region *cxlr; @@ -3912,7 +3953,8 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, do { cxlr = __create_region(cxlrd, cxlds->part[part].mode, - atomic_read(&cxlrd->region_id)); + atomic_read(&cxlrd->region_id), + cxled->cxld.target_type); } while (IS_ERR(cxlr) && PTR_ERR(cxlr) == -EBUSY); if (IS_ERR(cxlr)) { @@ -3925,7 +3967,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, rc = __construct_region(cxlr, ctx); if (rc) { - devm_release_action(port->uport_dev, unregister_region, cxlr); + cxl_region_release_action(cxlr); return ERR_PTR(rc); } @@ -4208,6 +4250,135 @@ static int cxl_region_can_probe(struct cxl_region *cxlr) return 0; } +static int first_mapped_decoder(struct device *dev, const void *data) +{ + struct cxl_endpoint_decoder *cxled; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled = to_cxl_endpoint_decoder(dev); + if (cxled->cxld.region) + return 1; + + return 0; +} + +/* + * As this is running in endpoint port remove context it does not race cxl_root + * destruction since port topologies are always removed depth first. + */ +static void cxl_endpoint_region_autoremove(void *_cxlr) +{ + unregister_region(_cxlr); +} + +/** + * cxl_memdev_attach_region - bind region to accelerator memdev + * + * @cxlmd: a pointer to cxl_memdev to use + * @attach: a pointer to region attach struct with callbacks for + * safely working with a region range by the caller + * + * Returns 0 or error. + */ +int cxl_memdev_attach_region(struct cxl_memdev *cxlmd, + struct cxl_attach_region *attach) +{ + struct cxl_port *endpoint = cxlmd->endpoint; + struct cxl_endpoint_decoder *cxled; + struct cxl_region *cxlr; + int rc; + + if (IS_ERR(endpoint)) + return PTR_ERR(endpoint); + if (!endpoint) + return -ENXIO; + + { + /* hold endpoint lock to setup autoremove of the region */ + guard(device)(&endpoint->dev); + if (!endpoint->dev.driver) + return -ENXIO; + + { + guard(rwsem_read)(&cxl_rwsem.region); + guard(rwsem_read)(&cxl_rwsem.dpa); + + /* + * TODO auto-instantiate a region, for now assume this will + * find an auto-region. + */ + struct device *dev __free(put_device) = + device_find_child(&endpoint->dev, NULL, + first_mapped_decoder); + + if (!dev) { + dev_dbg(cxlmd->cxlds->dev, + "no region found for memdev %s\n", + dev_name(&cxlmd->dev)); + return -ENXIO; + } + + cxled = to_cxl_endpoint_decoder(dev); + cxlr = cxled->cxld.region; + + if (cxlr->params.state < CXL_CONFIG_COMMIT) { + dev_dbg(cxlmd->cxlds->dev, + "region %s not committed for memdev %s\n", + dev_name(&cxlr->dev), dev_name(&cxlmd->dev)); + return -ENXIO; + } + + if (cxlr->params.nr_targets > 1) { + dev_dbg(cxlmd->cxlds->dev, + "Only attach to local non-interleaved region\n"); + return -ENXIO; + } + + attach->region = (struct range) { + .start = cxlr->params.res->start, + .end = cxlr->params.res->end, + }; + + /* + * With endpoint locked leave the caller to safely work + * with the region range. + */ + rc = attach->attach(attach->data); + if (rc) + return rc; + + /* Only teardown regions that pass validation, ignore the rest */ + rc = devm_add_action(&endpoint->dev, + cxl_endpoint_region_autoremove, cxlr); + if (rc) { + attach->detach(attach->data); + goto err_unregister; + } + + /* Link type2 driver callback for stopping use of the region range. */ + rc = devm_add_action_or_reset(&endpoint->dev, + attach->detach, attach->data); + if (rc) { + devm_remove_action(&endpoint->dev, + cxl_endpoint_region_autoremove, + cxlr); + goto err_unregister; + } + + cxlr->detach = attach->detach; + cxlr->detach_data = attach->data; + + return 0; + } +err_unregister: + unregister_region(cxlr); + return rc; + } +} +EXPORT_SYMBOL_NS_GPL(cxl_memdev_attach_region, "CXL"); + static int cxl_region_probe(struct device *dev) { struct cxl_region *cxlr = to_cxl_region(dev); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 8194447f75d30..94d7d7965c512 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -443,6 +443,8 @@ struct cxl_region_params { * @hpa_range: Address range occupied by the region * @mode: Operational mode of the mapped capacity * @type: Endpoint decoder target type + * @detach: accelerator detach callback for device-memory regions + * @detach_data: accelerator detach callback data * @cxl_nvb: nvdimm bridge for coordinating @cxlr_pmem setup / shutdown * @cxlr_pmem: (for pmem regions) cached copy of the nvdimm bridge * @flags: Region state flags @@ -458,6 +460,8 @@ struct cxl_region { struct range hpa_range; enum cxl_partition_mode mode; enum cxl_decoder_type type; + void (*detach)(void *data); + void *detach_data; struct cxl_nvdimm_bridge *cxl_nvb; struct cxl_pmem_region *cxlr_pmem; unsigned long flags; diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c index 7d8a6c2133c88..a16fefebe13a0 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.c +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -14,6 +14,36 @@ #define EFX_CTPIO_BUFFER_SIZE SZ_256M +/* Called with cxl endpoint device locked for precluding potential related + * cxl region removal triggered from user space, allowing safely mapping of + * such cxl region by the sfc driver. + */ +static int efx_cxl_map_region(void *data) { + struct efx_probe_data *probe_data = data; + struct efx_nic *efx = &probe_data->efx; + struct pci_dev *pci_dev = efx->pci_dev; + struct efx_cxl *cxl = probe_data->cxl; + struct range *cxl_pio_range = &cxl->attach_region.region; + + cxl->ctpio_cxl = ioremap(cxl_pio_range->start, + cxl_pio_range->end - cxl_pio_range->start + 1); + if (!cxl->ctpio_cxl) { + pci_err(pci_dev, "CXL ioremap region (%pra) failed\n", + cxl_pio_range); + return -ENOMEM; + } + probe_data->cxl_pio_initialised = true; + return 0; +} + +/* Called at driver exit or when user space triggers cxl region removal. */ +static void efx_cxl_unmap_region(void *data) { + struct efx_probe_data *probe_data = data; + + probe_data->cxl_pio_initialised = false; + iounmap(probe_data->cxl->ctpio_cxl); +} + int efx_cxl_init(struct efx_probe_data *probe_data) { struct efx_nic *efx = &probe_data->efx; @@ -81,8 +111,15 @@ int efx_cxl_init(struct efx_probe_data *probe_data) return PTR_ERR(cxl->cxlmd); } + cxl->attach_region.attach = efx_cxl_map_region; + cxl->attach_region.detach = efx_cxl_unmap_region; + cxl->attach_region.data = probe_data; probe_data->cxl = cxl; + rc = cxl_memdev_attach_region(cxl->cxlmd, &cxl->attach_region); + if (rc) + return rc; + return 0; } diff --git a/drivers/net/ethernet/sfc/efx_cxl.h b/drivers/net/ethernet/sfc/efx_cxl.h index 04e46278464df..1c294cd1df56c 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.h +++ b/drivers/net/ethernet/sfc/efx_cxl.h @@ -20,6 +20,8 @@ struct efx_probe_data; struct efx_cxl { struct cxl_dev_state cxlds; struct cxl_memdev *cxlmd; + struct cxl_attach_region attach_region; + void __iomem *ctpio_cxl; }; int efx_cxl_init(struct efx_probe_data *probe_data); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 10a9b8fa2f6b7..22d9435b351fb 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -7,6 +7,7 @@ #include #include +#include #include /** @@ -153,6 +154,20 @@ struct cxl_memdev_attach { int (*probe)(struct cxl_memdev *cxlmd); }; +/** + * struct cxl_attach_region - accelerator region handling + * @attach: invoked at cxl_memdev_attach_region() with endpoint device locked. + * @detach: invoked at endpoint release. + * @data: pointer referencing accelerator data for attach and detach calls. + * @region: initialised with autodiscovered region values linked to memdev. + */ +struct cxl_attach_region { + int (*attach)(void *); + void (*detach)(void *); + void *data; + struct range region; +}; + /** * struct cxl_dev_state - The driver device state * @@ -231,4 +246,6 @@ struct cxl_dev_state *_devm_cxl_dev_state_create(struct device *dev, int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity); struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, const struct cxl_memdev_attach *attach); +struct cxl_region; +int cxl_memdev_attach_region(struct cxl_memdev *cxlmd, struct cxl_attach_region *attach); #endif /* __CXL_CXL_H__ */ From 654f365103ca8b71b1a62987928ab3d833a2a8a7 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Thu, 23 Apr 2026 19:05:27 +0100 Subject: [PATCH 240/464] NVIDIA: VR: SAUCE: cxl: Avoid dax creation for accelerators BugLink: https://bugs.launchpad.net/bugs/2153819 By definition a type2 cxl device will use the host managed memory for specific functionality, therefore it should not be available to other uses like DAX. Signed-off-by: Alejandro Lucero Reviewed-by: Jonathan Cameron Reviewed-by: Davidlohr Bueso Reviewed-by: Dave Jiang Reviewed-by: Ben Cheatham (cherry picked from https://lore.kernel.org/r/20260423180528.17166-8-alejandro.lucero-palau@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/region.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index f8fdc0ec27316..9806eb9e19fac 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -4389,6 +4389,13 @@ static int cxl_region_probe(struct device *dev) if (rc) return rc; + /* + * HDM-D[B] (device-memory) regions have accelerator specific usage. + * Skip device-dax registration. + */ + if (cxlr->type == CXL_DECODER_DEVMEM) + return 0; + /* * From this point on any path that changes the region's state away from * CXL_CONFIG_COMMIT is also responsible for releasing the driver. From 9119243324bdaf26988df739985ef0ffdd4d1149 Mon Sep 17 00:00:00 2001 From: Alejandro Lucero Date: Wed, 13 May 2026 23:39:27 +0800 Subject: [PATCH 241/464] NVIDIA: VR: SAUCE: sfc: support pio mapping based on cxl BugLink: https://bugs.launchpad.net/bugs/2153819 A PIO buffer is a region of device memory to which the driver can write a packet for TX, with the device handling the transmit doorbell without requiring a DMA for getting the packet data, which helps reducing latency in certain exchanges. With CXL mem protocol this latency can be lowered further. With a device supporting CXL and successfully initialised, use the cxl region to map the memory range and use this mapping for PIO buffers. Add the disabling of those CXL-based PIO buffers if the callback for potential cxl endpoint removal by the CXL core happens. Signed-off-by: Alejandro Lucero (backported from https://lore.kernel.org/r/20260423180528.17166-9-alejandro.lucero-palau@amd.com) [kobak: Added a !EFX_USE_PIO same-module stub for efx_ef10_disable_piobufs() so non-x86 builds that still enable CONFIG_SFC_CXL do not leave efx_cxl.o unresolved.] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/sfc/ef10.c | 78 ++++++++++++++++++++++++--- drivers/net/ethernet/sfc/efx.h | 1 + drivers/net/ethernet/sfc/efx_cxl.c | 2 + drivers/net/ethernet/sfc/net_driver.h | 2 + drivers/net/ethernet/sfc/nic.h | 3 ++ 5 files changed, 79 insertions(+), 7 deletions(-) diff --git a/drivers/net/ethernet/sfc/ef10.c b/drivers/net/ethernet/sfc/ef10.c index 7e04f115bbaaa..52ad07c121833 100644 --- a/drivers/net/ethernet/sfc/ef10.c +++ b/drivers/net/ethernet/sfc/ef10.c @@ -24,6 +24,7 @@ #include #include #include +#include "efx_cxl.h" /* Hardware control for EF10 architecture including 'Huntington'. */ @@ -106,7 +107,7 @@ static int efx_ef10_get_vf_index(struct efx_nic *efx) static int efx_ef10_init_datapath_caps(struct efx_nic *efx) { - MCDI_DECLARE_BUF(outbuf, MC_CMD_GET_CAPABILITIES_V4_OUT_LEN); + MCDI_DECLARE_BUF(outbuf, MC_CMD_GET_CAPABILITIES_V7_OUT_LEN); struct efx_ef10_nic_data *nic_data = efx->nic_data; size_t outlen; int rc; @@ -177,6 +178,12 @@ static int efx_ef10_init_datapath_caps(struct efx_nic *efx) efx->num_mac_stats); } + if (outlen < MC_CMD_GET_CAPABILITIES_V7_OUT_LEN) + nic_data->datapath_caps3 = 0; + else + nic_data->datapath_caps3 = MCDI_DWORD(outbuf, + GET_CAPABILITIES_V7_OUT_FLAGS3); + return 0; } @@ -771,6 +778,35 @@ static int efx_ef10_alloc_piobufs(struct efx_nic *efx, unsigned int n) return rc; } +#ifdef CONFIG_SFC_CXL +/* Invoked from cxl core when a cxl region is removed. This is expected at + * driver exit linked to cxl core devm releases which does not require the + * below sync. + * + * However, it is required when user space actions triggger such a cxl region + * removal forcing any cxl piobuf usage to stop. Setting per tx queue piobuf + * to NULL is safe if such a tx queue is not currently in use inside + * efx_hard_start_xmit() implying tx_queue locked. + * + * After this the cxl region physical range can be safely unmap. + */ +void efx_ef10_disable_piobufs(struct efx_nic *efx) +{ + struct efx_tx_queue *tx_queue; + struct efx_channel *channel; + + local_bh_disable(); + efx_for_each_channel(channel, efx) + efx_for_each_channel_tx_queue(tx_queue, channel) { + HARD_TX_LOCK(efx->net_dev, tx_queue->core_txq, + smp_processor_id()); + tx_queue->piobuf = NULL; + HARD_TX_UNLOCK(efx->net_dev, tx_queue->core_txq); + } + local_bh_enable(); +} +#endif + static int efx_ef10_link_piobufs(struct efx_nic *efx) { struct efx_ef10_nic_data *nic_data = efx->nic_data; @@ -914,6 +950,12 @@ static void efx_ef10_forget_old_piobufs(struct efx_nic *efx) { } +#ifdef CONFIG_SFC_CXL +void efx_ef10_disable_piobufs(struct efx_nic *efx) +{ +} +#endif + #endif /* EFX_USE_PIO */ static void efx_ef10_remove(struct efx_nic *efx) @@ -1140,6 +1182,9 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx) unsigned int channel_vis, pio_write_vi_base, max_vis; struct efx_ef10_nic_data *nic_data = efx->nic_data; unsigned int uc_mem_map_size, wc_mem_map_size; +#ifdef CONFIG_SFC_CXL + struct efx_probe_data *probe_data; +#endif void __iomem *membase; int rc; @@ -1263,8 +1308,25 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx) iounmap(efx->membase); efx->membase = membase; - /* Set up the WC mapping if needed */ - if (wc_mem_map_size) { + if (!wc_mem_map_size) + goto skip_pio; + + /* Set up the WC mapping */ + +#ifdef CONFIG_SFC_CXL + probe_data = container_of(efx, struct efx_probe_data, efx); + if ((nic_data->datapath_caps3 & + (1 << MC_CMD_GET_CAPABILITIES_V7_OUT_CXL_CONFIG_ENABLE_LBN)) && + probe_data->cxl_pio_initialised) { + /* Using PIO through CXL mapping */ + nic_data->pio_write_base = probe_data->cxl->ctpio_cxl; + nic_data->pio_write_vi_base = pio_write_vi_base; + + probe_data->cxl_pio_in_use = true; + } else +#endif + { + /* Using legacy PIO BAR mapping */ nic_data->wc_membase = ioremap_wc(efx->membase_phys + uc_mem_map_size, wc_mem_map_size); @@ -1279,12 +1341,14 @@ static int efx_ef10_dimension_resources(struct efx_nic *efx) nic_data->wc_membase + (pio_write_vi_base * efx->vi_stride + ER_DZ_TX_PIOBUF - uc_mem_map_size); - - rc = efx_ef10_link_piobufs(efx); - if (rc) - efx_ef10_free_piobufs(efx); } + rc = efx_ef10_link_piobufs(efx); + if (rc) + efx_ef10_free_piobufs(efx); + +skip_pio: + netif_dbg(efx, probe, efx->net_dev, "memory BAR at %pa (virtual %p+%x UC, %p+%x WC)\n", &efx->membase_phys, efx->membase, uc_mem_map_size, diff --git a/drivers/net/ethernet/sfc/efx.h b/drivers/net/ethernet/sfc/efx.h index 45e1916866256..37fd1cf96582e 100644 --- a/drivers/net/ethernet/sfc/efx.h +++ b/drivers/net/ethernet/sfc/efx.h @@ -237,4 +237,5 @@ static inline bool efx_rwsem_assert_write_locked(struct rw_semaphore *sem) int efx_xdp_tx_buffers(struct efx_nic *efx, int n, struct xdp_frame **xdpfs, bool flush); +void efx_ef10_disable_piobufs(struct efx_nic *efx); #endif /* EFX_EFX_H */ diff --git a/drivers/net/ethernet/sfc/efx_cxl.c b/drivers/net/ethernet/sfc/efx_cxl.c index a16fefebe13a0..52b2cded76daf 100644 --- a/drivers/net/ethernet/sfc/efx_cxl.c +++ b/drivers/net/ethernet/sfc/efx_cxl.c @@ -10,6 +10,7 @@ #include #include #include "net_driver.h" +#include "efx.h" #include "efx_cxl.h" #define EFX_CTPIO_BUFFER_SIZE SZ_256M @@ -40,6 +41,7 @@ static int efx_cxl_map_region(void *data) { static void efx_cxl_unmap_region(void *data) { struct efx_probe_data *probe_data = data; + efx_ef10_disable_piobufs(&probe_data->efx); probe_data->cxl_pio_initialised = false; iounmap(probe_data->cxl->ctpio_cxl); } diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h index 3964b2c56609c..bea4eecdf842d 100644 --- a/drivers/net/ethernet/sfc/net_driver.h +++ b/drivers/net/ethernet/sfc/net_driver.h @@ -1207,6 +1207,7 @@ struct efx_cxl; * @efx: Efx NIC details * @cxl: details of related cxl objects * @cxl_pio_initialised: cxl initialization outcome. + * @cxl_pio_in_use: PIO using CXL mapping */ struct efx_probe_data { struct pci_dev *pci_dev; @@ -1214,6 +1215,7 @@ struct efx_probe_data { #ifdef CONFIG_SFC_CXL struct efx_cxl *cxl; bool cxl_pio_initialised; + bool cxl_pio_in_use; #endif }; diff --git a/drivers/net/ethernet/sfc/nic.h b/drivers/net/ethernet/sfc/nic.h index ec3b2df43b68d..7480f9995dfb8 100644 --- a/drivers/net/ethernet/sfc/nic.h +++ b/drivers/net/ethernet/sfc/nic.h @@ -152,6 +152,8 @@ enum { * %MC_CMD_GET_CAPABILITIES response) * @datapath_caps2: Further Capabilities of datapath firmware (FLAGS2 field of * %MC_CMD_GET_CAPABILITIES response) + * @datapath_caps3: Further Capabilities of datapath firmware (FLAGS3 field of + * %MC_CMD_GET_CAPABILITIES response) * @rx_dpcpu_fw_id: Firmware ID of the RxDPCPU * @tx_dpcpu_fw_id: Firmware ID of the TxDPCPU * @must_probe_vswitching: Flag: vswitching has yet to be setup after MC reboot @@ -187,6 +189,7 @@ struct efx_ef10_nic_data { bool must_check_datapath_caps; u32 datapath_caps; u32 datapath_caps2; + u32 datapath_caps3; unsigned int rx_dpcpu_fw_id; unsigned int tx_dpcpu_fw_id; bool must_probe_vswitching; From 7e38ecb9edc9ed36f7aed322e45f7fb33cc39ea7 Mon Sep 17 00:00:00 2001 From: Dan Williams Date: Tue, 10 Feb 2026 06:44:53 +0000 Subject: [PATCH 242/464] NVIDIA: VR: SAUCE: dax/hmem: Request cxl_acpi and cxl_pci before walking Soft Reserved ranges BugLink: https://bugs.launchpad.net/bugs/2153819 Ensure cxl_acpi has published CXL Window resources before HMEM walks Soft Reserved ranges. Replace MODULE_SOFTDEP("pre: cxl_acpi") with an explicit, synchronous request_module("cxl_acpi"). MODULE_SOFTDEP() only guarantees eventual loading, it does not enforce that the dependency has finished init before the current module runs. This can cause HMEM to start before cxl_acpi has populated the resource tree, breaking detection of overlaps between Soft Reserved and CXL Windows. Also, request cxl_pci before HMEM walks Soft Reserved ranges. Unlike cxl_acpi, cxl_pci attach is asynchronous and creates dependent devices that trigger further module loads. Asynchronous probe flushing (wait_for_device_probe()) is added later in the series in a deferred context before HMEM makes ownership decisions for Soft Reserved ranges. Add an additional explicit Kconfig ordering so that CXL_ACPI and CXL_PCI must be initialized before DEV_DAX_HMEM. This prevents HMEM from consuming Soft Reserved ranges before CXL drivers have had a chance to claim them. Signed-off-by: Dan Williams Signed-off-by: Smita Koralahalli Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Reviewed-by: Alison Schofield Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-2-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-2-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/Kconfig | 2 ++ drivers/dax/hmem/hmem.c | 17 ++++++++++------- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index d656e4c0eb846..3683bb3f2311b 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -48,6 +48,8 @@ config DEV_DAX_CXL tristate "CXL DAX: direct access to CXL RAM regions" depends on CXL_BUS && CXL_REGION && DEV_DAX default CXL_REGION && DEV_DAX + depends on CXL_ACPI >= DEV_DAX_HMEM + depends on CXL_PCI >= DEV_DAX_HMEM help CXL RAM regions are either mapped by platform-firmware and published in the initial system-memory map as "System RAM", mapped diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c index 1cf7c2a0ee1cb..008172fc3607e 100644 --- a/drivers/dax/hmem/hmem.c +++ b/drivers/dax/hmem/hmem.c @@ -139,6 +139,16 @@ static __init int dax_hmem_init(void) { int rc; + /* + * Ensure that cxl_acpi and cxl_pci have a chance to kick off + * CXL topology discovery at least once before scanning the + * iomem resource tree for IORES_DESC_CXL resources. + */ + if (IS_ENABLED(CONFIG_DEV_DAX_CXL)) { + request_module("cxl_acpi"); + request_module("cxl_pci"); + } + rc = platform_driver_register(&dax_hmem_platform_driver); if (rc) return rc; @@ -159,13 +169,6 @@ static __exit void dax_hmem_exit(void) module_init(dax_hmem_init); module_exit(dax_hmem_exit); -/* Allow for CXL to define its own dax regions */ -#if IS_ENABLED(CONFIG_CXL_REGION) -#if IS_MODULE(CONFIG_CXL_ACPI) -MODULE_SOFTDEP("pre: cxl_acpi"); -#endif -#endif - MODULE_ALIAS("platform:hmem*"); MODULE_ALIAS("platform:hmem_platform*"); MODULE_DESCRIPTION("HMEM DAX: direct access to 'specific purpose' memory"); From a5ab4824bf8087dae259d351175f87ed92795978 Mon Sep 17 00:00:00 2001 From: Dan Williams Date: Tue, 10 Feb 2026 06:44:54 +0000 Subject: [PATCH 243/464] NVIDIA: VR: SAUCE: dax/hmem: Gate Soft Reserved deferral on DEV_DAX_CXL BugLink: https://bugs.launchpad.net/bugs/2153819 Replace IS_ENABLED(CONFIG_CXL_REGION) with IS_ENABLED(CONFIG_DEV_DAX_CXL) so that HMEM only defers Soft Reserved ranges when CXL DAX support is enabled. This makes the coordination between HMEM and the CXL stack more precise and prevents deferral in unrelated CXL configurations. Signed-off-by: Dan Williams Signed-off-by: Smita Koralahalli Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Reviewed-by: Alison Schofield Reviewed-by: Dave Jiang Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-3-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-3-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/hmem/hmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c index 008172fc3607e..1e34243584905 100644 --- a/drivers/dax/hmem/hmem.c +++ b/drivers/dax/hmem/hmem.c @@ -66,7 +66,7 @@ static int hmem_register_device(struct device *host, int target_nid, long id; int rc; - if (IS_ENABLED(CONFIG_CXL_REGION) && + if (IS_ENABLED(CONFIG_DEV_DAX_CXL) && region_intersects(res->start, resource_size(res), IORESOURCE_MEM, IORES_DESC_CXL) != REGION_DISJOINT) { dev_dbg(host, "deferring range to CXL: %pr\n", res); From 41912166319c8a4e65f1f440d8d0d1ac44df715a Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:44:55 +0000 Subject: [PATCH 244/464] NVIDIA: VR: SAUCE: cxl/region: Skip decoder reset on detach for autodiscovered regions BugLink: https://bugs.launchpad.net/bugs/2153819 __cxl_decoder_detach() currently resets decoder programming whenever a region is detached if cxl_config_state is beyond CXL_CONFIG_ACTIVE. For autodiscovered regions, this can incorrectly tear down decoder state that may be relied upon by other consumers or by subsequent ownership decisions. Skip cxl_region_decode_reset() during detach when CXL_REGION_F_AUTO is set. Signed-off-by: Smita Koralahalli Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Reviewed-by: Alejandro Lucero Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-4-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-4-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/region.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 9806eb9e19fac..37dfb3965fbb3 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -2185,7 +2185,9 @@ __cxl_decoder_detach(struct cxl_region *cxlr, cxled->part = -1; if (p->state > CXL_CONFIG_ACTIVE) { - cxl_region_decode_reset(cxlr, p->interleave_ways); + if (!test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) + cxl_region_decode_reset(cxlr, p->interleave_ways); + p->state = CXL_CONFIG_ACTIVE; } From 84804e31332e3577f22e82d73cfd1fffd761ca9e Mon Sep 17 00:00:00 2001 From: Dan Williams Date: Tue, 10 Feb 2026 06:44:56 +0000 Subject: [PATCH 245/464] NVIDIA: VR: SAUCE: dax/cxl, hmem: Initialize hmem early and defer dax_cxl binding BugLink: https://bugs.launchpad.net/bugs/2153819 Move hmem/ earlier in the dax Makefile so that hmem_init() runs before dax_cxl. In addition, defer registration of the dax_cxl driver to a workqueue instead of using module_cxl_driver(). This ensures that dax_hmem has an opportunity to initialize and register its deferred callback and make ownership decisions before dax_cxl begins probing and claiming Soft Reserved ranges. Mark the dax_cxl driver as PROBE_PREFER_ASYNCHRONOUS so its probe runs out of line from other synchronous probing avoiding ordering dependencies while coordinating ownership decisions with dax_hmem. Signed-off-by: Dan Williams Signed-off-by: Smita Koralahalli Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-5-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-5-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/Makefile | 3 +-- drivers/dax/cxl.c | 27 ++++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/drivers/dax/Makefile b/drivers/dax/Makefile index 5ed5c39857c8b..70e996bf15261 100644 --- a/drivers/dax/Makefile +++ b/drivers/dax/Makefile @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 +obj-y += hmem/ obj-$(CONFIG_DAX) += dax.o obj-$(CONFIG_DEV_DAX) += device_dax.o obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o @@ -10,5 +11,3 @@ dax-y += bus.o device_dax-y := device.o dax_pmem-y := pmem.o dax_cxl-y := cxl.o - -obj-y += hmem/ diff --git a/drivers/dax/cxl.c b/drivers/dax/cxl.c index 13cd94d32ff7a..a2136adfa186e 100644 --- a/drivers/dax/cxl.c +++ b/drivers/dax/cxl.c @@ -38,10 +38,35 @@ static struct cxl_driver cxl_dax_region_driver = { .id = CXL_DEVICE_DAX_REGION, .drv = { .suppress_bind_attrs = true, + .probe_type = PROBE_PREFER_ASYNCHRONOUS, }, }; -module_cxl_driver(cxl_dax_region_driver); +static void cxl_dax_region_driver_register(struct work_struct *work) +{ + cxl_driver_register(&cxl_dax_region_driver); +} + +static DECLARE_WORK(cxl_dax_region_driver_work, cxl_dax_region_driver_register); + +static int __init cxl_dax_region_init(void) +{ + /* + * Need to resolve a race with dax_hmem wanting to drive regions + * instead of CXL + */ + queue_work(system_long_wq, &cxl_dax_region_driver_work); + return 0; +} +module_init(cxl_dax_region_init); + +static void __exit cxl_dax_region_exit(void) +{ + flush_work(&cxl_dax_region_driver_work); + cxl_driver_unregister(&cxl_dax_region_driver); +} +module_exit(cxl_dax_region_exit); + MODULE_ALIAS_CXL(CXL_DEVICE_DAX_REGION); MODULE_DESCRIPTION("CXL DAX: direct access to CXL regions"); MODULE_LICENSE("GPL"); From 99a11d05b89b0a349e83b43ffd01b5482a120cf1 Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:44:57 +0000 Subject: [PATCH 246/464] NVIDIA: VR: SAUCE: dax: Track all dax_region allocations under a global resource tree BugLink: https://bugs.launchpad.net/bugs/2153819 Introduce a global "DAX Regions" resource root and register each dax_region->res under it via request_resource(). Release the resource on dax_region teardown. By enforcing a single global namespace for dax_region allocations, this ensures only one of dax_hmem or dax_cxl can successfully register a dax_region for a given range. Co-developed-by: Dan Williams Signed-off-by: Dan Williams Signed-off-by: Smita Koralahalli Reviewed-by: Dave Jiang Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-6-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-6-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/bus.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c index c94c09622516e..ba288662082d7 100644 --- a/drivers/dax/bus.c +++ b/drivers/dax/bus.c @@ -10,6 +10,7 @@ #include "dax-private.h" #include "bus.h" +static struct resource dax_regions = DEFINE_RES_MEM_NAMED(0, -1, "DAX Regions"); static DEFINE_MUTEX(dax_bus_lock); /* @@ -625,6 +626,8 @@ static void dax_region_unregister(void *region) { struct dax_region *dax_region = region; + scoped_guard(rwsem_write, &dax_region_rwsem) + release_resource(&dax_region->res); sysfs_remove_groups(&dax_region->dev->kobj, dax_region_attribute_groups); dax_region_put(dax_region); @@ -635,6 +638,7 @@ struct dax_region *alloc_dax_region(struct device *parent, int region_id, unsigned long flags) { struct dax_region *dax_region; + int rc; /* * The DAX core assumes that it can store its private data in @@ -667,14 +671,27 @@ struct dax_region *alloc_dax_region(struct device *parent, int region_id, .flags = IORESOURCE_MEM | flags, }; - if (sysfs_create_groups(&parent->kobj, dax_region_attribute_groups)) { - kfree(dax_region); - return NULL; + scoped_guard(rwsem_write, &dax_region_rwsem) + rc = request_resource(&dax_regions, &dax_region->res); + if (rc) { + dev_dbg(parent, "dax_region resource conflict for %pR\n", + &dax_region->res); + goto err_res; } + if (sysfs_create_groups(&parent->kobj, dax_region_attribute_groups)) + goto err_sysfs; + if (devm_add_action_or_reset(parent, dax_region_unregister, dax_region)) return NULL; return dax_region; + +err_sysfs: + scoped_guard(rwsem_write, &dax_region_rwsem) + release_resource(&dax_region->res); +err_res: + kfree(dax_region); + return NULL; } EXPORT_SYMBOL_GPL(alloc_dax_region); From 477935b437d00cc44fedcbb027543765372ab5ab Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:44:58 +0000 Subject: [PATCH 247/464] NVIDIA: VR: SAUCE: cxl/region: Add helper to check Soft Reserved containment by CXL regions BugLink: https://bugs.launchpad.net/bugs/2153819 Add a helper to determine whether a given Soft Reserved memory range is fully contained within the committed CXL region. This helper provides a primitive for policy decisions in subsequent patches such as co-ordination with dax_hmem to determine whether CXL has fully claimed ownership of Soft Reserved memory ranges. Signed-off-by: Smita Koralahalli Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Reviewed-by: Dan Williams Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-7-Smita.KoralahalliChannabasappa@amd.com (backported from https://lore.kernel.org/r/20260210064501.157591-7-Smita.KoralahalliChannabasappa@amd.com) [kobak: Added the Soft Reserved declaration to the existing Type2 include/cxl/cxl.h header instead of recreating that header.] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/region.c | 30 ++++++++++++++++++++++++++++++ include/cxl/cxl.h | 9 +++++++++ 2 files changed, 39 insertions(+) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 37dfb3965fbb3..78ec23b21f901 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include "core.h" @@ -4227,6 +4228,35 @@ static int cxl_region_setup_poison(struct cxl_region *cxlr) return devm_add_action_or_reset(dev, remove_debugfs, dentry); } +static int region_contains_soft_reserve(struct device *dev, void *data) +{ + struct resource *res = data; + struct cxl_region *cxlr; + struct cxl_region_params *p; + + if (!is_cxl_region(dev)) + return 0; + + cxlr = to_cxl_region(dev); + p = &cxlr->params; + + if (p->state != CXL_CONFIG_COMMIT) + return 0; + + if (!p->res) + return 0; + + return resource_contains(p->res, res) ? 1 : 0; +} + +bool cxl_region_contains_soft_reserve(struct resource *res) +{ + guard(rwsem_read)(&cxl_rwsem.region); + return bus_for_each_dev(&cxl_bus_type, NULL, res, + region_contains_soft_reserve) != 0; +} +EXPORT_SYMBOL_GPL(cxl_region_contains_soft_reserve); + static int cxl_region_can_probe(struct cxl_region *cxlr) { struct cxl_region_params *p = &cxlr->params; diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 22d9435b351fb..3fbd9eac137ea 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -248,4 +248,13 @@ struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, const struct cxl_memdev_attach *attach); struct cxl_region; int cxl_memdev_attach_region(struct cxl_memdev *cxlmd, struct cxl_attach_region *attach); + +#ifdef CONFIG_CXL_REGION +bool cxl_region_contains_soft_reserve(struct resource *res); +#else +static inline bool cxl_region_contains_soft_reserve(struct resource *res) +{ + return false; +} +#endif #endif /* __CXL_CXL_H__ */ From d8af1a7a3ae2363e322db49174dfcc50379212fc Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:44:59 +0000 Subject: [PATCH 248/464] NVIDIA: VR: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination BugLink: https://bugs.launchpad.net/bugs/2153819 Add helpers to register, queue and flush the deferred work. These helpers allow dax_hmem to execute ownership resolution outside the probe context before dax_cxl binds. Signed-off-by: Smita Koralahalli Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-8-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-8-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/bus.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/dax/bus.h | 7 ++++++ 2 files changed, 65 insertions(+) diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c index ba288662082d7..523953959a16a 100644 --- a/drivers/dax/bus.c +++ b/drivers/dax/bus.c @@ -25,6 +25,64 @@ DECLARE_RWSEM(dax_region_rwsem); */ DECLARE_RWSEM(dax_dev_rwsem); +static DEFINE_MUTEX(dax_hmem_lock); +static dax_hmem_deferred_fn hmem_deferred_fn; +static void *dax_hmem_data; + +static void hmem_deferred_work(struct work_struct *work) +{ + dax_hmem_deferred_fn fn; + void *data; + + scoped_guard(mutex, &dax_hmem_lock) { + fn = hmem_deferred_fn; + data = dax_hmem_data; + } + + if (fn) + fn(data); +} + +static DECLARE_WORK(dax_hmem_work, hmem_deferred_work); + +int dax_hmem_register_work(dax_hmem_deferred_fn fn, void *data) +{ + guard(mutex)(&dax_hmem_lock); + + if (hmem_deferred_fn) + return -EINVAL; + + hmem_deferred_fn = fn; + dax_hmem_data = data; + return 0; +} +EXPORT_SYMBOL_GPL(dax_hmem_register_work); + +int dax_hmem_unregister_work(dax_hmem_deferred_fn fn, void *data) +{ + guard(mutex)(&dax_hmem_lock); + + if (hmem_deferred_fn != fn || dax_hmem_data != data) + return -EINVAL; + + hmem_deferred_fn = NULL; + dax_hmem_data = NULL; + return 0; +} +EXPORT_SYMBOL_GPL(dax_hmem_unregister_work); + +void dax_hmem_queue_work(void) +{ + queue_work(system_long_wq, &dax_hmem_work); +} +EXPORT_SYMBOL_GPL(dax_hmem_queue_work); + +void dax_hmem_flush_work(void) +{ + flush_work(&dax_hmem_work); +} +EXPORT_SYMBOL_GPL(dax_hmem_flush_work); + #define DAX_NAME_LEN 30 struct dax_id { struct list_head list; diff --git a/drivers/dax/bus.h b/drivers/dax/bus.h index cbbf64443098c..b58a88e8089c0 100644 --- a/drivers/dax/bus.h +++ b/drivers/dax/bus.h @@ -41,6 +41,13 @@ struct dax_device_driver { void (*remove)(struct dev_dax *dev); }; +typedef void (*dax_hmem_deferred_fn)(void *data); + +int dax_hmem_register_work(dax_hmem_deferred_fn fn, void *data); +int dax_hmem_unregister_work(dax_hmem_deferred_fn fn, void *data); +void dax_hmem_queue_work(void); +void dax_hmem_flush_work(void); + int __dax_driver_register(struct dax_device_driver *dax_drv, struct module *module, const char *mod_name); #define dax_driver_register(driver) \ From 44d9aee6ac021be58ae3eedb5bfe47b0cb3c29fa Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:45:00 +0000 Subject: [PATCH 249/464] NVIDIA: VR: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges BugLink: https://bugs.launchpad.net/bugs/2153819 The current probe time ownership check for Soft Reserved memory based solely on CXL window intersection is insufficient. dax_hmem probing is not always guaranteed to run after CXL enumeration and region assembly, which can lead to incorrect ownership decisions before the CXL stack has finished publishing windows and assembling committed regions. Introduce deferred ownership handling for Soft Reserved ranges that intersect CXL windows. When such a range is encountered during dax_hmem probe, schedule deferred work and wait for the CXL stack to complete enumeration and region assembly before deciding ownership. Evaluate ownership of Soft Reserved ranges based on CXL region containment. - If all Soft Reserved ranges are fully contained within committed CXL regions, DROP handling Soft Reserved ranges from dax_hmem and allow dax_cxl to bind. - If any Soft Reserved range is not fully claimed by committed CXL region, REGISTER the Soft Reserved ranges with dax_hmem. Use dax_cxl_mode to coordinate ownership decisions for Soft Reserved ranges. Once, ownership resolution is complete, flush the deferred work from dax_cxl before allowing dax_cxl to bind. This enforces a strict ownership. Either CXL fully claims the Soft reserved ranges or it relinquishes it entirely. Co-developed-by: Dan Williams Signed-off-by: Dan Williams Signed-off-by: Smita Koralahalli Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-9-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-9-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/bus.c | 3 ++ drivers/dax/bus.h | 19 ++++++++++ drivers/dax/cxl.c | 1 + drivers/dax/hmem/hmem.c | 78 +++++++++++++++++++++++++++++++++++++++-- 4 files changed, 99 insertions(+), 2 deletions(-) diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c index 523953959a16a..94c9d947a8a39 100644 --- a/drivers/dax/bus.c +++ b/drivers/dax/bus.c @@ -25,6 +25,9 @@ DECLARE_RWSEM(dax_region_rwsem); */ DECLARE_RWSEM(dax_dev_rwsem); +enum dax_cxl_mode dax_cxl_mode = DAX_CXL_MODE_DEFER; +EXPORT_SYMBOL_NS_GPL(dax_cxl_mode, "CXL"); + static DEFINE_MUTEX(dax_hmem_lock); static dax_hmem_deferred_fn hmem_deferred_fn; static void *dax_hmem_data; diff --git a/drivers/dax/bus.h b/drivers/dax/bus.h index b58a88e8089c0..82616ff52fd14 100644 --- a/drivers/dax/bus.h +++ b/drivers/dax/bus.h @@ -41,6 +41,25 @@ struct dax_device_driver { void (*remove)(struct dev_dax *dev); }; +/* + * enum dax_cxl_mode - State machine to determine ownership for CXL + * tagged Soft Reserved memory ranges. + * @DAX_CXL_MODE_DEFER: Ownership resolution pending. Set while waiting + * for CXL enumeration and region assembly to complete. + * @DAX_CXL_MODE_REGISTER: CXL regions do not fully cover Soft Reserved + * ranges. Fall back to registering those ranges via dax_hmem. + * @DAX_CXL_MODE_DROP: All Soft Reserved ranges intersecting CXL windows + * are fully contained within committed CXL regions. Drop HMEM handling + * and allow dax_cxl to bind. + */ +enum dax_cxl_mode { + DAX_CXL_MODE_DEFER, + DAX_CXL_MODE_REGISTER, + DAX_CXL_MODE_DROP, +}; + +extern enum dax_cxl_mode dax_cxl_mode; + typedef void (*dax_hmem_deferred_fn)(void *data); int dax_hmem_register_work(dax_hmem_deferred_fn fn, void *data); diff --git a/drivers/dax/cxl.c b/drivers/dax/cxl.c index a2136adfa186e..3ab39b77843d5 100644 --- a/drivers/dax/cxl.c +++ b/drivers/dax/cxl.c @@ -44,6 +44,7 @@ static struct cxl_driver cxl_dax_region_driver = { static void cxl_dax_region_driver_register(struct work_struct *work) { + dax_hmem_flush_work(); cxl_driver_register(&cxl_dax_region_driver); } diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c index 1e34243584905..85854e25254b2 100644 --- a/drivers/dax/hmem/hmem.c +++ b/drivers/dax/hmem/hmem.c @@ -3,6 +3,7 @@ #include #include #include +#include #include "../bus.h" static bool region_idle; @@ -69,8 +70,18 @@ static int hmem_register_device(struct device *host, int target_nid, if (IS_ENABLED(CONFIG_DEV_DAX_CXL) && region_intersects(res->start, resource_size(res), IORESOURCE_MEM, IORES_DESC_CXL) != REGION_DISJOINT) { - dev_dbg(host, "deferring range to CXL: %pr\n", res); - return 0; + switch (dax_cxl_mode) { + case DAX_CXL_MODE_DEFER: + dev_dbg(host, "deferring range to CXL: %pr\n", res); + dax_hmem_queue_work(); + return 0; + case DAX_CXL_MODE_REGISTER: + dev_dbg(host, "registering CXL range: %pr\n", res); + break; + case DAX_CXL_MODE_DROP: + dev_dbg(host, "dropping CXL range: %pr\n", res); + return 0; + } } rc = region_intersects_soft_reserve(res->start, resource_size(res)); @@ -123,8 +134,70 @@ static int hmem_register_device(struct device *host, int target_nid, return rc; } +static int hmem_register_cxl_device(struct device *host, int target_nid, + const struct resource *res) +{ + if (region_intersects(res->start, resource_size(res), IORESOURCE_MEM, + IORES_DESC_CXL) != REGION_DISJOINT) + return hmem_register_device(host, target_nid, res); + + return 0; +} + +static int soft_reserve_has_cxl_match(struct device *host, int target_nid, + const struct resource *res) +{ + if (region_intersects(res->start, resource_size(res), IORESOURCE_MEM, + IORES_DESC_CXL) != REGION_DISJOINT) { + if (!cxl_region_contains_soft_reserve((struct resource *)res)) + return 1; + } + + return 0; +} + +static void process_defer_work(void *data) +{ + struct platform_device *pdev = data; + int rc; + + /* relies on cxl_acpi and cxl_pci having had a chance to load */ + wait_for_device_probe(); + + rc = walk_hmem_resources(&pdev->dev, soft_reserve_has_cxl_match); + + if (!rc) { + dax_cxl_mode = DAX_CXL_MODE_DROP; + dev_dbg(&pdev->dev, "All Soft Reserved ranges claimed by CXL\n"); + } else { + dax_cxl_mode = DAX_CXL_MODE_REGISTER; + dev_warn(&pdev->dev, + "Soft Reserved not fully contained in CXL; using HMEM\n"); + } + + walk_hmem_resources(&pdev->dev, hmem_register_cxl_device); +} + +static void kill_defer_work(void *data) +{ + struct platform_device *pdev = data; + + dax_hmem_flush_work(); + dax_hmem_unregister_work(process_defer_work, pdev); +} + static int dax_hmem_platform_probe(struct platform_device *pdev) { + int rc; + + rc = dax_hmem_register_work(process_defer_work, pdev); + if (rc) + return rc; + + rc = devm_add_action_or_reset(&pdev->dev, kill_defer_work, pdev); + if (rc) + return rc; + return walk_hmem_resources(&pdev->dev, hmem_register_device); } @@ -174,3 +247,4 @@ MODULE_ALIAS("platform:hmem_platform*"); MODULE_DESCRIPTION("HMEM DAX: direct access to 'specific purpose' memory"); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Intel Corporation"); +MODULE_IMPORT_NS("CXL"); From 380f8265ffea22f7cab3c969f5e03f1178821b9d Mon Sep 17 00:00:00 2001 From: Smita Koralahalli Date: Tue, 10 Feb 2026 06:45:01 +0000 Subject: [PATCH 250/464] NVIDIA: VR: SAUCE: dax/hmem: Reintroduce Soft Reserved ranges back into the iomem tree BugLink: https://bugs.launchpad.net/bugs/2153819 Reworked from a patch by Alison Schofield Reintroduce Soft Reserved range into the iomem_resource tree for HMEM to consume. This restores visibility in /proc/iomem for ranges actively in use, while avoiding the early-boot conflicts that occurred when Soft Reserved was published into iomem before CXL window and region discovery. Link: https://lore.kernel.org/linux-cxl/29312c0765224ae76862d59a17748c8188fb95f1.1692638817.git.alison.schofield@intel.com/ Co-developed-by: Alison Schofield Signed-off-by: Alison Schofield Co-developed-by: Zhijian Li Signed-off-by: Zhijian Li Signed-off-by: Smita Koralahalli Reviewed-by: Dave Jiang Reviewed-by: Jonathan Cameron Reviewed-by: Dan Williams Tested-by: Tomasz Wolski Link: https://lore.kernel.org/r/20260210064501.157591-10-Smita.KoralahalliChannabasappa@amd.com (cherry picked from https://lore.kernel.org/r/20260210064501.157591-10-Smita.KoralahalliChannabasappa@amd.com) Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/dax/hmem/hmem.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/drivers/dax/hmem/hmem.c b/drivers/dax/hmem/hmem.c index 85854e25254b2..c07bf5fe833dc 100644 --- a/drivers/dax/hmem/hmem.c +++ b/drivers/dax/hmem/hmem.c @@ -59,6 +59,34 @@ static void release_hmem(void *pdev) platform_device_unregister(pdev); } +static void remove_soft_reserved(void *r) +{ + remove_resource(r); + kfree(r); +} + +static int add_soft_reserve_into_iomem(struct device *host, + const struct resource *res) +{ + int rc; + + struct resource *soft __free(kfree) = + kmalloc(sizeof(*res), GFP_KERNEL); + if (!soft) + return -ENOMEM; + + *soft = DEFINE_RES_NAMED_DESC(res->start, (res->end - res->start + 1), + "Soft Reserved", IORESOURCE_MEM, + IORES_DESC_SOFT_RESERVED); + + rc = insert_resource(&iomem_resource, soft); + if (rc) + return rc; + + return devm_add_action_or_reset(host, remove_soft_reserved, + no_free_ptr(soft)); +} + static int hmem_register_device(struct device *host, int target_nid, const struct resource *res) { @@ -88,7 +116,9 @@ static int hmem_register_device(struct device *host, int target_nid, if (rc != REGION_INTERSECTS) return 0; - /* TODO: Add Soft-Reserved memory back to iomem */ + rc = add_soft_reserve_into_iomem(host, res); + if (rc) + return rc; id = memregion_alloc(GFP_KERNEL); if (id < 0) { From e774a2ddfcf497884322cb2144e8c73e152dc4c2 Mon Sep 17 00:00:00 2001 From: Robert Richter Date: Tue, 28 Oct 2025 10:47:53 +0100 Subject: [PATCH 251/464] NVIDIA: VR: SAUCE: cxl/region: Support multi-level interleaving with smaller granularities for lower levels BugLink: https://bugs.launchpad.net/bugs/2153819 The CXL specification supports multi-level interleaving "as long as all the levels use different, but consecutive, HPA bits to select the target and no Interleave Set has more than 8 devices" (from 3.2). Currently the kernel expects that a decoder's "interleave granularity is a multiple of @parent_port granularity". That is, the granularity of a lower level is bigger than those of the parent and uses the outer HPA bits as selector. It works e.g. for the following 8-way config: * cross-link (cross-hostbridge config in CFMWS): * 4-way * 256 granularity * Selector: HPA[8:9] * sub-link (CXL Host bridge config of the HDM): * 2-way * 1024 granularity * Selector: HPA[10] Now, if the outer HPA bits are used for the cross-hostbridge, an 8-way config could look like this: * cross-link (cross-hostbridge config in CFMWS): * 4-way * 512 granularity * Selector: HPA[9:10] * sub-link (CXL Host bridge config of the HDM): * 2-way * 256 granularity * Selector: HPA[8] The enumeration of decoders for this configuration fails then with following error: cxl region0: pci0000:00:port1 cxl_port_setup_targets expected iw: 2 ig: 1024 [mem 0x10000000000-0x1ffffffffff flags 0x200] cxl region0: pci0000:00:port1 cxl_port_setup_targets got iw: 2 ig: 256 state: enabled 0x10000000000:0x1ffffffffff cxl_port endpoint12: failed to attach decoder12.0 to region0: -6 Note that this happens only if firmware is setting up the decoders (CXL_REGION_F_AUTO). For userspace region assembly the granularities are chosen to increase from root down to the lower levels. That is, outer HPA bits are always used for lower interleaving levels. Rework the implementation to also support multi-level interleaving with smaller granularities for lower levels. Determine the interleave set of autodetected decoders. Check that it is a subset of the root interleave. The HPA selector bits are extracted for all decoders of the set and checked that there is no overlap and bits are consecutive. All decoders can be programmed now to use any bit range within the region's target selector. Signed-off-by: Robert Richter (backported from https://lore.kernel.org/all/20251028094754.72816-1-rrichter@amd.com/) [kobak: resolved conflicts with cxlr->cxlrd and spa_maps_hpa()] Signed-off-by: Koba Ko Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/region.c | 201 ++++++++++++++++++++------------------ 1 file changed, 108 insertions(+), 93 deletions(-) diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 78ec23b21f901..455a2c090589b 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -1376,57 +1376,119 @@ static int check_interleave_cap(struct cxl_decoder *cxld, int iw, int ig) return 0; } +static inline u64 get_selector(u64 ways, u64 gran) +{ + if (!is_power_of_2(ways)) + ways /= 3; + + if (!is_power_of_2(ways) || !is_power_of_2(gran)) + return 0; + + return (ways - 1) * gran; +} + static int cxl_port_setup_targets(struct cxl_port *port, struct cxl_region *cxlr, struct cxl_endpoint_decoder *cxled) { struct cxl_root_decoder *cxlrd = cxlr->cxlrd; - int parent_iw, parent_ig, ig, iw, rc, pos = cxled->pos; struct cxl_port *parent_port = to_cxl_port(port->dev.parent); struct cxl_region_ref *cxl_rr = cxl_rr_load(port, cxlr); struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); struct cxl_ep *ep = cxl_ep_load(port, cxlmd); struct cxl_region_params *p = &cxlr->params; struct cxl_decoder *cxld = cxl_rr->decoder; - struct cxl_switch_decoder *cxlsd; + struct cxl_switch_decoder *cxlsd = to_cxl_switch_decoder(&cxld->dev); struct cxl_port *iter = port; - u16 eig, peig; - u8 eiw, peiw; + int ig, iw = cxl_rr->nr_targets, rc, pos = cxled->pos; + int distance, parent_distance; + u64 selector, cxlr_sel; + u16 eig; + u8 eiw; /* * While root level decoders support x3, x6, x12, switch level * decoders only support powers of 2 up to x16. */ - if (!is_power_of_2(cxl_rr->nr_targets)) { + if (!is_power_of_2(iw)) { dev_dbg(&cxlr->dev, "%s:%s: invalid target count %d\n", - dev_name(port->uport_dev), dev_name(&port->dev), - cxl_rr->nr_targets); + dev_name(port->uport_dev), dev_name(&port->dev), iw); return -EINVAL; } - cxlsd = to_cxl_switch_decoder(&cxld->dev); - if (cxl_rr->nr_targets_set) { - int i, distance = 1; - struct cxl_region_ref *cxl_rr_iter; + if (iw > 8 || iw > cxlsd->nr_targets) { + dev_dbg(&cxlr->dev, + "%s:%s:%s: ways: %d overflows targets: %d\n", + dev_name(port->uport_dev), dev_name(&port->dev), + dev_name(&cxld->dev), iw, cxlsd->nr_targets); + return -ENXIO; + } - /* - * The "distance" between peer downstream ports represents which - * endpoint positions in the region interleave a given port can - * host. - * - * For example, at the root of a hierarchy the distance is - * always 1 as every index targets a different host-bridge. At - * each subsequent switch level those ports map every Nth region - * position where N is the width of the switch == distance. - */ - do { - cxl_rr_iter = cxl_rr_load(iter, cxlr); - distance *= cxl_rr_iter->nr_targets; - iter = to_cxl_port(iter->dev.parent); - } while (!is_cxl_root(iter)); - distance *= cxlrd->cxlsd.cxld.interleave_ways; + /* + * Calculate the effective granularity and ways to determine + * HPA bits used as target selectors of the interleave set. + * Use this to check if the root decoder and all subsequent + * HDM decoders only use bits from that range as selectors. + * + * The "distance" between peer downstream ports represents which + * endpoint positions in the region interleave a given port can + * host. + * + * For example, at the root of a hierarchy the distance is + * always 1 as every index targets a different host-bridge. At + * each subsequent switch level those ports map every Nth region + * position where N is the width of the switch == distance. + */ + + /* Start with the root decoders selector and distance. */ + selector = get_selector(cxlrd->cxlsd.cxld.interleave_ways, + cxlrd->cxlsd.cxld.interleave_granularity); + distance = cxlrd->cxlsd.cxld.interleave_ways; + if (!is_power_of_2(distance)) + distance /= 3; + + for (iter = parent_port; !is_cxl_root(iter); + iter = to_cxl_port(iter->dev.parent)) { + struct cxl_region_ref *cxl_rr_iter = cxl_rr_load(iter, cxlr); + struct cxl_decoder *cxld_iter = cxl_rr_iter->decoder; + u64 cxld_sel; + + if (cxld_iter->interleave_ways == 1) + continue; + + cxld_sel = get_selector(cxld_iter->interleave_ways, + cxld_iter->interleave_granularity); + + if (cxld_sel & selector) { + dev_dbg(&cxlr->dev, "%s:%s: overlapping selectors: %#llx:%#llx\n", + dev_name(iter->uport_dev), + dev_name(&iter->dev), cxld_sel, selector); + return -ENXIO; + } - for (i = 0; i < cxl_rr->nr_targets_set; i++) + selector |= cxld_sel; + distance *= cxl_rr_iter->nr_targets; + } + + parent_distance = distance; + distance *= iw; + + /* The combined selector bits must fit the region selector. */ + cxlr_sel = get_selector(p->interleave_ways, + p->interleave_granularity); + + if ((cxlr_sel & selector) != selector) { + dev_dbg(&cxlr->dev, "%s:%s: invalid selectors: %#llx:%#llx\n", + dev_name(iter->uport_dev), + dev_name(&iter->dev), cxlr_sel, selector); + return -ENXIO; + } + + /* Calculate remaining selector bits available for use. */ + selector = cxlr_sel & ~selector; + + if (cxl_rr->nr_targets_set) { + for (int i = 0; i < cxl_rr->nr_targets_set; i++) if (ep->dport == cxlsd->target[i]) { rc = check_last_peer(cxled, ep, cxl_rr, distance); @@ -1437,87 +1499,40 @@ static int cxl_port_setup_targets(struct cxl_port *port, goto add_target; } - if (is_cxl_root(parent_port)) { + if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) + ig = cxld->interleave_granularity; + else /* + * Set the interleave granularity with each interleave + * level to a multiple of it's parent port interleave + * ways. Beginning with the granularity of the root + * decoder set to the region granularity (starting + * with the inner selector bits of the HPA), the + * granularity is increased with each level. Calculate + * this using the parent distance and region + * granularity. + * * Root decoder IG is always set to value in CFMWS which * may be different than this region's IG. We can use the * region's IG here since interleave_granularity_store() * does not allow interleaved host-bridges with * root IG != region IG. */ - parent_ig = p->interleave_granularity; - parent_iw = cxlrd->cxlsd.cxld.interleave_ways; - /* - * For purposes of address bit routing, use power-of-2 math for - * switch ports. - */ - if (!is_power_of_2(parent_iw)) - parent_iw /= 3; - } else { - struct cxl_region_ref *parent_rr; - struct cxl_decoder *parent_cxld; - - parent_rr = cxl_rr_load(parent_port, cxlr); - parent_cxld = parent_rr->decoder; - parent_ig = parent_cxld->interleave_granularity; - parent_iw = parent_cxld->interleave_ways; - } - - rc = granularity_to_eig(parent_ig, &peig); - if (rc) { - dev_dbg(&cxlr->dev, "%s:%s: invalid parent granularity: %d\n", - dev_name(parent_port->uport_dev), - dev_name(&parent_port->dev), parent_ig); - return rc; - } - - rc = ways_to_eiw(parent_iw, &peiw); - if (rc) { - dev_dbg(&cxlr->dev, "%s:%s: invalid parent interleave: %d\n", - dev_name(parent_port->uport_dev), - dev_name(&parent_port->dev), parent_iw); - return rc; - } + ig = p->interleave_granularity * parent_distance; - iw = cxl_rr->nr_targets; rc = ways_to_eiw(iw, &eiw); - if (rc) { - dev_dbg(&cxlr->dev, "%s:%s: invalid port interleave: %d\n", - dev_name(port->uport_dev), dev_name(&port->dev), iw); - return rc; - } - - /* - * Interleave granularity is a multiple of @parent_port granularity. - * Multiplier is the parent port interleave ways. - */ - rc = granularity_to_eig(parent_ig * parent_iw, &eig); - if (rc) { - dev_dbg(&cxlr->dev, - "%s: invalid granularity calculation (%d * %d)\n", - dev_name(&parent_port->dev), parent_ig, parent_iw); - return rc; - } - - rc = eig_to_granularity(eig, &ig); - if (rc) { - dev_dbg(&cxlr->dev, "%s:%s: invalid interleave: %d\n", - dev_name(port->uport_dev), dev_name(&port->dev), - 256 << eig); - return rc; - } + if (!rc) + rc = granularity_to_eig(ig, &eig); - if (iw > 8 || iw > cxlsd->nr_targets) { - dev_dbg(&cxlr->dev, - "%s:%s:%s: ways: %d overflows targets: %d\n", + if (rc || (iw > 1 && ~selector & get_selector(iw, ig))) { + dev_dbg(&cxlr->dev, "%s:%s: invalid port interleave: %d:%d:%#llx\n", dev_name(port->uport_dev), dev_name(&port->dev), - dev_name(&cxld->dev), iw, cxlsd->nr_targets); + iw, ig, selector); return -ENXIO; } if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) { if (cxld->interleave_ways != iw || - (iw > 1 && cxld->interleave_granularity != ig) || !spa_maps_hpa(p, &cxld->hpa_range) || ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)) { dev_err(&cxlr->dev, From ecc92b794b83fc22ff3b24eecf25072073569b6e Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Thu, 14 May 2026 01:04:53 +0800 Subject: [PATCH 252/464] NVIDIA: VR: SAUCE: [Config] CXL config annotations for Type-2 device and RAS support BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Source: https://github.com/NVIDIA/NV-Kernels/commit/f80636d27f65bb38e54c3601e7efbdde16ed0b40 Add Ubuntu kernel config annotations for CXL-related configs introduced or changed by the CXL Type-2, RAS, and autodiscovered-region support backports. CONFIG_CXL_BUS, CONFIG_CXL_PCI, CONFIG_CXL_MEM, and CONFIG_CXL_PORT are built in for Type-2 device support. CONFIG_CXL_RAS and the EINJ symbols cover CXL RAS/error-injection support. CONFIG_SFC_CXL remains disabled for NVIDIA platforms. Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit f80636d27f65bb38e54c3601e7efbdde16ed0b40 nv-kernels/24.04_linux-nvidia-6.17-next) [kobak: Backported annotation overrides from debian.nvidia-6.17 to debian.nvidia-bos; PCIEAER_CXL is overridden as removed instead of editing debian.master.] Signed-off-by: Koba Ko --- debian.nvidia-bos/config/annotations | 39 ++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 392f5a93126da..c6338b139f0fd 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -6,6 +6,15 @@ include "../../debian.master/config/annotations" +CONFIG_ACPI_APEI_EINJ policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_ACPI_APEI_EINJ note<'Required for CONFIG_ACPI_APEI_EINJ_CXL'> + +CONFIG_ACPI_APEI_EINJ_CXL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_ACPI_APEI_EINJ_CXL note<'CXL protocol error injection support via APEI EINJ'> + +CONFIG_ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION note<'Override debian.master amd64-only; arm64 selects this via arch/arm64/Kconfig since 4d873c5dc3ed'> + CONFIG_ARM64_ERRATUM_1902691 policy<{'arm64': 'y'}> CONFIG_ARM64_ERRATUM_1902691 note<'Required for Grace enablement'> @@ -36,6 +45,9 @@ CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enable CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablement'> +CONFIG_CACHEMAINT_FOR_HOTPLUG policy<{'amd64': '-', 'arm64': 'n'}> +CONFIG_CACHEMAINT_FOR_HOTPLUG note<'Optional HiSilicon HHA cache maintenance driver; depends on GENERIC_CPU_CACHE_MAINTENANCE; not needed for NVIDIA platforms'> + CONFIG_ARM_FFA_TRANSPORT policy<{'arm64': 'y'}> CONFIG_ARM_FFA_TRANSPORT note<'LP: #2111511'> @@ -111,9 +123,24 @@ CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE note<'LP: #2028576: Perf governo CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL note<'LP: #2028576: Perf governor required for NVIDIA workloads'> +CONFIG_CXL_BUS policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CXL_BUS note<'Enable CXL bus support built-in; required for CXL Type-2 device and RAS support'> + +CONFIG_CXL_MEM policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CXL_MEM note<'Auto-selected by CXL_PCI; required for CXL memory expansion and Type-2 device support'> + CONFIG_CXL_MEM_RAW_COMMANDS policy<{'amd64': 'n', 'arm64': 'y'}> CONFIG_CXL_MEM_RAW_COMMANDS note<'Enable CXL raw commands for memory devices'> +CONFIG_CXL_PCI policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CXL_PCI note<'Enable CXL PCI management built-in; auto-selects CXL_MEM; required for CXL Type-2 device support'> + +CONFIG_CXL_PORT policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CXL_PORT note<'Required for CXL port enumeration; defaults to CXL_BUS value'> + +CONFIG_CXL_RAS policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_CXL_RAS note<'New def_bool replacing PCIEAER_CXL; auto-enabled with ACPI_APEI_GHES+PCIEAER+CXL_BUS; CXL RAS error handling support'> + CONFIG_DRM_NOUVEAU policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_DRM_NOUVEAU note<'Disable nouveau for NVIDIA kernels'> @@ -135,6 +162,12 @@ CONFIG_EFI_CAPSULE_LOADER note<'LP: #2067111'> CONFIG_ETM4X_IMPDEF_FEATURE policy<{'arm64': 'n'}> CONFIG_ETM4X_IMPDEF_FEATURE note<'Required for Grace enablement'> +CONFIG_FWCTL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_FWCTL note<'Selected by CXL_BUS when CXL_FEATURES is enabled; required for CXL feature mailbox access'> + +CONFIG_GENERIC_CPU_CACHE_MAINTENANCE policy<{'amd64': '-', 'arm64': 'y'}> +CONFIG_GENERIC_CPU_CACHE_MAINTENANCE note<'Selected by arm64 via arch/arm64/Kconfig since 4d873c5dc3ed; not selected by x86'> + CONFIG_GPIO_AAEON policy<{'amd64': '-'}> CONFIG_GPIO_AAEON note<'Disable all Ubuntu ODM drivers'> @@ -180,6 +213,9 @@ CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> +CONFIG_PCIEAER_CXL policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_PCIEAER_CXL note<'Removed by commit d18f1b7beadf (PCI/AER: Replace PCIEAER_CXL symbol with CXL_RAS)'> + CONFIG_NVGRACE_EGM policy<{'arm64': 'm'}> CONFIG_NVGRACE_EGM note<'LP: #2119656'> @@ -207,6 +243,9 @@ CONFIG_SAMPLE_CORESIGHT_SYSCFG note<'Required for Grace enablem CONFIG_SENSORS_AAEON policy<{'amd64': '-'}> CONFIG_SENSORS_AAEON note<'Disable all Ubuntu ODM drivers'> +CONFIG_SFC_CXL policy<{'amd64': 'n', 'arm64': 'n'}> +CONFIG_SFC_CXL note<'Solarflare SFC9100-family CXL Type-2 device support; not needed for NVIDIA platforms'> + CONFIG_SPI_TEGRA210_QUAD policy<{'arm64': 'y'}> CONFIG_SPI_TEGRA210_QUAD note<'Ensures the TPM is available before the IMA driver initializes'> From 9f76671572f27a29feceba68b0a157d70c7d3a71 Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Thu, 14 May 2026 01:05:24 +0800 Subject: [PATCH 253/464] NVIDIA: VR: SAUCE: [Config] Enable CXL DAX and KMEM built-in for CXL memory access BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Source: https://github.com/NVIDIA/NV-Kernels/commit/c5c11cf704ac7c320de7b53e26189c9d10d6b18f Override debian.master policy for DEV_DAX, DEV_DAX_CXL, and DEV_DAX_KMEM so CXL memory regions are available as raw DAX devices and as hotplugged System-RAM without relying on module load ordering. Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit c5c11cf704ac7c320de7b53e26189c9d10d6b18f nv-kernels/24.04_linux-nvidia-6.17-next) [kobak: Backported annotation overrides from debian.nvidia-6.17 to debian.nvidia-bos.] Signed-off-by: Koba Ko --- debian.nvidia-bos/config/annotations | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index c6338b139f0fd..0ad2528908f19 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -264,6 +264,15 @@ CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM dri CONFIG_ULTRASOC_SMB policy<{'arm64': 'n'}> CONFIG_ULTRASOC_SMB note<'Required for Grace enablement'> +CONFIG_DEV_DAX policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX note<'Override debian.master m->y; required built-in for DEV_DAX_CXL=y'> + +CONFIG_DEV_DAX_CXL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX_CXL note<'Override debian.master m->y; CXL RAM region DAX access; depends on CXL_BUS+CXL_REGION+DEV_DAX'> + +CONFIG_DEV_DAX_KMEM policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX_KMEM note<'Override debian.master m->y; map CXL DAX devices as System-RAM'> + CONFIG_VFIO_CONTAINER policy<{'amd64': 'y', 'arm64': 'n'}> CONFIG_VFIO_CONTAINER note<'LP: #2095028'> From 30e4f67b68e69e200e735c28f86c5ef65916c620 Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Thu, 14 May 2026 01:05:40 +0800 Subject: [PATCH 254/464] NVIDIA: VR: SAUCE: [Config] Add PCI_CXL annotation for CXL state save/restore BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Source: https://github.com/NVIDIA/NV-Kernels/commit/a5544cbc1ab9ec25eff39cbcb67eb67133a261a7 Add Ubuntu kernel config annotation for CONFIG_PCI_CXL introduced by the CXL DVSEC and HDM state save/restore series. CONFIG_PCI_CXL is a hidden bool auto-enabled when CXL_BUS=y. It gates compilation of drivers/pci/cxl.o, which saves and restores CXL DVSEC control/range registers and HDM decoder state across PCI resets and link transitions. Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit a5544cbc1ab9ec25eff39cbcb67eb67133a261a7 nv-kernels/24.04_linux-nvidia-6.17-next) [kobak: Backported annotation override from debian.nvidia-6.17 to debian.nvidia-bos.] Signed-off-by: Koba Ko --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 0ad2528908f19..854c543bb2a1c 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -273,6 +273,9 @@ CONFIG_DEV_DAX_CXL note<'Override debian.master m-> CONFIG_DEV_DAX_KMEM policy<{'amd64': 'y', 'arm64': 'y'}> CONFIG_DEV_DAX_KMEM note<'Override debian.master m->y; map CXL DAX devices as System-RAM'> +CONFIG_PCI_CXL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_PCI_CXL note<'Hidden bool; auto-enabled by CXL_BUS; PCI core CXL DVSEC and HDM state save/restore support'> + CONFIG_VFIO_CONTAINER policy<{'amd64': 'y', 'arm64': 'n'}> CONFIG_VFIO_CONTAINER note<'LP: #2095028'> From a49119485418c44a42b96b6ee1f6fbdec0a748a1 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Tue, 19 May 2026 02:00:36 +0800 Subject: [PATCH 255/464] NVIDIA: SAUCE: Revert "NVIDIA: VR: SAUCE: cxl: add support for cxl reset" BugLink: https://bugs.launchpad.net/bugs/2153819 This reverts commit 4f089d910e1cfc4e1983533ab4aa8f6a1b4b296c. Drop the older monolithic CXL reset method before applying the refreshed save/restore and reset series. Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Matt Ochs Signed-off-by: Brad Figg --- drivers/pci/pci.c | 146 ---------------------------------- include/linux/pci.h | 2 +- include/uapi/linux/pci_regs.h | 14 ---- 3 files changed, 1 insertion(+), 161 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 0b4b7371341f6..9298a461bd302 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4980,151 +4980,6 @@ static int cxl_reset_bus_function(struct pci_dev *dev, bool probe) return rc; } -static int cxl_reset_prepare(struct pci_dev *dev, u16 dvsec) -{ - u32 timeout_us = 100, timeout_tot_us = 10000; - u16 reg, cap; - int rc; - - if (!pci_wait_for_pending_transaction(dev)) - pci_err(dev, "timed out waiting for pending transaction; performing cxl reset anyway\n"); - - /* Check if the device is cache capable. */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, &cap); - if (rc) - return rc; - - if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE)) - return 0; - - /* Disable cache. WB and invalidate cache if capability is advertised */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); - if (rc) - return rc; - reg |= PCI_DVSEC_CXL_DISABLE_CACHING; - /* - * DEVCTL2 bits are written only once. So check WB+I capability while - * keeping disable caching set. - */ - if (cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE) - reg |= PCI_DVSEC_CXL_INIT_CACHE_WBI; - pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); - - /* - * From Section 9.6: "Software may leverage the cache size reported in - * the DVSEC CXL Capability2 register to compute a suitable timeout - * value". - * Given there is no conversion factor for cache size -> timeout, - * setting timer for default 10ms. - */ - do { - if (timeout_tot_us == 0) - return -ETIMEDOUT; - usleep_range(timeout_us, timeout_us + 1); - timeout_tot_us -= timeout_us; - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, - ®); - if (rc) - return rc; - } while (!(reg & PCI_DVSEC_CXL_CACHE_INV)); - - return 0; -} - -static int cxl_reset_init(struct pci_dev *dev, u16 dvsec) -{ - /* - * Timeout values ref CXL Spec v3.2 Ch 8 Control and Status Registers, - * under section 8.1.3.1 DVSEC CXL Capability. - */ - u32 reset_timeouts_ms[] = { 10, 100, 1000, 10000, 100000 }; - u16 reg; - u32 timeout_ms; - int rc, ind; - - /* Check if CXL Reset MEM CLR is supported. */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); - if (rc) - return rc; - - if (reg & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE) { - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, - ®); - if (rc) - return rc; - - reg |= PCI_DVSEC_CXL_RST_MEM_CLR_EN; - pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); - } - - /* Read timeout value. */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); - if (rc) - return rc; - ind = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, reg); - timeout_ms = reset_timeouts_ms[ind]; - - /* Write reset config. */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); - if (rc) - return rc; - - reg |= PCI_DVSEC_CXL_INIT_CXL_RST; - pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); - - /* Wait till timeout and then check reset status is complete. */ - msleep(timeout_ms); - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_STATUS2, ®); - if (rc) - return rc; - if (reg & PCI_DVSEC_CXL_RST_ERR || - ~reg & PCI_DVSEC_CXL_RST_DONE) - return -ETIMEDOUT; - - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, ®); - if (rc) - return rc; - reg &= (~PCI_DVSEC_CXL_DISABLE_CACHING); - pci_write_config_word(dev, dvsec + PCI_DVSEC_CXL_CTRL2, reg); - - return 0; -} - -/** - * cxl_reset - initiate a cxl reset - * @dev: device to reset - * @probe: if true, return 0 if device can be reset this way - * - * Initiate a cxl reset on @dev. - */ -static int cxl_reset(struct pci_dev *dev, bool probe) -{ - u16 dvsec, reg; - int rc; - - dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL, - PCI_DVSEC_CXL_DEVICE); - if (!dvsec) - return -ENOTTY; - - /* Check if CXL Reset is supported. */ - rc = pci_read_config_word(dev, dvsec + PCI_DVSEC_CXL_CAP, ®); - if (rc) - return -ENOTTY; - - if ((reg & PCI_DVSEC_CXL_RST_CAPABLE) == 0) - return -ENOTTY; - - if (probe) - return 0; - - rc = cxl_reset_prepare(dev, dvsec); - if (rc) - return rc; - - return cxl_reset_init(dev, dvsec); -} - void pci_dev_lock(struct pci_dev *dev) { /* block PM suspend, driver probe, etc. */ @@ -5212,7 +5067,6 @@ const struct pci_reset_fn_method pci_reset_fn_methods[] = { { pci_dev_acpi_reset, .name = "acpi" }, { pcie_reset_flr, .name = "flr" }, { pci_af_flr, .name = "af_flr" }, - { cxl_reset, .name = "cxl_reset" }, { pci_pm_reset, .name = "pm" }, { pci_reset_bus_function, .name = "bus" }, { cxl_reset_bus_function, .name = "cxl_bus" }, diff --git a/include/linux/pci.h b/include/linux/pci.h index 9758af74a5452..14bbdea91919e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -51,7 +51,7 @@ PCI_STATUS_PARITY) /* Number of reset methods used in pci_reset_fn_methods array in pci.c */ -#define PCI_NUM_RESET_METHODS 9 +#define PCI_NUM_RESET_METHODS 8 #define PCI_RESET_PROBE true #define PCI_RESET_DO_RESET false diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 543275ff9ed62..718fb630f5bb7 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1349,24 +1349,10 @@ /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */ #define PCI_DVSEC_CXL_DEVICE 0 #define PCI_DVSEC_CXL_CAP 0xA -#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0) #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2) #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) -#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) -#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) -#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) -#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11) #define PCI_DVSEC_CXL_CTRL 0xC #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) -#define PCI_DVSEC_CXL_CTRL2 0x10 -#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) -#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) -#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) -#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) -#define PCI_DVSEC_CXL_STATUS2 0x12 -#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) -#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) -#define PCI_DVSEC_CXL_RST_ERR _BITUL(2) #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0) From 48e0b8a140d4756ba2c51811bb338bcabf3b8108 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 08:00:15 +0000 Subject: [PATCH 256/464] NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC control, lock, and range register definitions BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 PCI: Add CXL DVSEC control, lock, and range register definitions Add register offset and field definitions for CXL DVSEC registers needed by CXL state save/restore across resets: - CTRL2 (offset 0x10) and LOCK (offset 0x14) registers - CONFIG_LOCK bit in the LOCK register - RWL (read-write-when-locked) field masks for CTRL and range base registers. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit 07ad5f1ef09ec2c6dd872bd61e533cc2da8ed32b nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- include/uapi/linux/pci_regs.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 718fb630f5bb7..148328bd4cb37 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1353,6 +1353,10 @@ #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) #define PCI_DVSEC_CXL_CTRL 0xC #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) +#define PCI_DVSEC_CXL_CTRL_RWL 0x5FED +#define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_LOCK 0x14 +#define PCI_DVSEC_CXL_LOCK_CONFIG _BITUL(0) #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) #define PCI_DVSEC_CXL_RANGE_SIZE_LOW(i) (0x1C + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_INFO_VALID _BITUL(0) @@ -1360,8 +1364,10 @@ #define PCI_DVSEC_CXL_MEM_ACTIVE_TIMEOUT __GENMASK(15, 13) #define PCI_DVSEC_CXL_MEM_SIZE_LOW __GENMASK(31, 28) #define PCI_DVSEC_CXL_RANGE_BASE_HIGH(i) (0x20 + (i * 0x10)) +#define PCI_DVSEC_CXL_RANGE_BASE_HI_RWL 0xFFFFFFFF #define PCI_DVSEC_CXL_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) #define PCI_DVSEC_CXL_MEM_BASE_LOW __GENMASK(31, 28) +#define PCI_DVSEC_CXL_RANGE_BASE_LO_RWL 0xF0000000 #define CXL_DVSEC_RANGE_MAX 2 From 9a100acbd560d5070bd2741c77f7dcb3e3a15a26 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 08:00:16 +0000 Subject: [PATCH 257/464] NVIDIA: VR: SAUCE: cxl: Move HDM decoder and register map definitions to include/cxl/cxl.h BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Move CXL HDM decoder register defines, register map structs (cxl_reg_map, cxl_component_reg_map, cxl_device_reg_map, cxl_pmu_reg_map, cxl_register_map), cxl_hdm_decoder_count(), enum cxl_regloc_type, and cxl_find_regblock()/cxl_setup_regs() declarations from internal CXL headers to include/cxl/pci.h. This makes them accessible to code outside the CXL subsystem, in particular the PCI core CXL state save/restore support added in a subsequent patch. No functional change. Signed-off-by: Srirangan Madhavan (backported from https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit b5e166cae47a8356338c607c99d98007b83d3324 nv-kernels/24.04_linux-nvidia-6.17-next) [koba: Also move CXL_CM_CAP_CAP_ID_RAS, CXL_CM_CAP_CAP_ID_HDM, and CXL_CM_CAP_CAP_HDM_VERSION into public include/cxl/cxl.h to keep the public CXL header layout consistent.] Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/cxl/cxl.h | 61 ----------------------------------------------- include/cxl/cxl.h | 58 ++++++++++++++++++++++++++++++++++++++++++++ include/cxl/pci.h | 5 +++- 3 files changed, 62 insertions(+), 62 deletions(-) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 94d7d7965c512..c7bffa399581e 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -24,63 +24,6 @@ extern const struct nvdimm_security_ops *cxl_security_ops; * (port-driver, region-driver, nvdimm object-drivers... etc). */ -/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ -#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K - -/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers*/ -#define CXL_CM_OFFSET 0x1000 -#define CXL_CM_CAP_HDR_OFFSET 0x0 -#define CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0) -#define CM_CAP_HDR_CAP_ID 1 -#define CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16) -#define CM_CAP_HDR_CAP_VERSION 1 -#define CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20) -#define CM_CAP_HDR_CACHE_MEM_VERSION 1 -#define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24) -#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20) - -#define CXL_CM_CAP_CAP_ID_RAS 0x2 -#define CXL_CM_CAP_CAP_ID_HDM 0x5 -#define CXL_CM_CAP_CAP_HDM_VERSION 1 - -/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ -#define CXL_HDM_DECODER_CAP_OFFSET 0x0 -#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0) -#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) -#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) -#define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11) -#define CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12) -#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 -#define CXL_HDM_DECODER_ENABLE BIT(1) -#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) -#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14) -#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18) -#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c) -#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20) -#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) -#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER0_CTRL_LOCK BIT(8) -#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) -#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) -#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11) -#define CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12) -#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24) -#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28) -#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i) -#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i) - -/* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */ -#define CXL_DECODER_MIN_GRANULARITY 256 -#define CXL_DECODER_MAX_ENCODED_IG 6 - -static inline int cxl_hdm_decoder_count(u32 cap_hdr) -{ - int val = FIELD_GET(CXL_HDM_DECODER_COUNT_MASK, cap_hdr); - - return val ? val * 2 : 1; -} - /* Encode defined in CXL 2.0 8.2.5.12.7 HDM Decoder Control Register */ static inline int eig_to_granularity(u16 eig, unsigned int *granularity) { @@ -214,13 +157,9 @@ int cxl_map_device_regs(const struct cxl_register_map *map, int cxl_map_pmu_regs(struct cxl_register_map *map, struct cxl_pmu_regs *regs); #define CXL_INSTANCES_COUNT -1 -enum cxl_regloc_type; int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type); int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map, unsigned int index); -int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, - struct cxl_register_map *map); -int cxl_setup_regs(struct cxl_register_map *map); struct cxl_dport; int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 3fbd9eac137ea..1c496c1e846c2 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -5,6 +5,7 @@ #ifndef __CXL_CXL_H__ #define __CXL_CXL_H__ +#include #include #include #include @@ -71,6 +72,63 @@ struct cxl_regs { ); }; +#define CXL_CM_CAP_CAP_ID_RAS 0x2 +#define CXL_CM_CAP_CAP_ID_HDM 0x5 +#define CXL_CM_CAP_CAP_HDM_VERSION 1 + +/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ +#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K + +/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers */ +#define CXL_CM_OFFSET 0x1000 +#define CXL_CM_CAP_HDR_OFFSET 0x0 +#define CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0) +#define CM_CAP_HDR_CAP_ID 1 +#define CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16) +#define CM_CAP_HDR_CAP_VERSION 1 +#define CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20) +#define CM_CAP_HDR_CACHE_MEM_VERSION 1 +#define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24) +#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20) + +/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ +#define CXL_HDM_DECODER_CAP_OFFSET 0x0 +#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) +#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) +#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) +#define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11) +#define CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12) +#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE BIT(1) +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18) +#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c) +#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) +#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) +#define CXL_HDM_DECODER0_CTRL_LOCK BIT(8) +#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) +#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11) +#define CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12) +#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24) +#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28) +#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i) +#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i) + +/* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */ +#define CXL_DECODER_MIN_GRANULARITY 256 +#define CXL_DECODER_MAX_ENCODED_IG 6 + +static inline int cxl_hdm_decoder_count(u32 cap_hdr) +{ + int val = FIELD_GET(CXL_HDM_DECODER_COUNT_MASK, cap_hdr); + + return val ? val * 2 : 1; +} + struct cxl_reg_map { bool valid; int id; diff --git a/include/cxl/pci.h b/include/cxl/pci.h index 3e0000015871a..edbf980c283f1 100644 --- a/include/cxl/pci.h +++ b/include/cxl/pci.h @@ -14,9 +14,12 @@ enum cxl_regloc_type { CXL_REGLOC_RBI_TYPES }; -struct cxl_register_map; struct pci_dev; +struct cxl_register_map; int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); +int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, + struct cxl_register_map *map); +int cxl_setup_regs(struct cxl_register_map *map); #endif From f4352fb4622d426850fca8142e524178b2143b7e Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 08:00:17 +0000 Subject: [PATCH 258/464] NVIDIA: VR: SAUCE: PCI: Add virtual extended cap save buffer for CXL state BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Add pci_add_virtual_ext_cap_save_buffer() to allocate save buffers using virtual cap IDs (above PCI_EXT_CAP_ID_MAX) that don't require a real capability in config space. The existing pci_add_ext_cap_save_buffer() cannot be used for CXL DVSEC state because it calls pci_find_saved_ext_cap() which searches for a matching capability in PCI config space. The CXL state saved here is a synthetic snapshot (DVSEC+HDM) and should not be tied to a real extended-cap instance. A virtual extended-cap save buffer API (cap IDs above PCI_EXT_CAP_ID_MAX) allows PCI to track this state without a backing config space capability. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit b3a768fe26282e3cbf250b4c4657cf11fea6ada4 nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/pci/pci.c | 20 ++++++++++++++++++++ drivers/pci/pci.h | 18 ++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 9298a461bd302..10c874730e0df 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3445,6 +3445,26 @@ int pci_add_ext_cap_save_buffer(struct pci_dev *dev, u16 cap, unsigned int size) return _pci_add_cap_save_buffer(dev, cap, true, size); } +int pci_add_virtual_ext_cap_save_buffer(struct pci_dev *dev, u16 cap, + unsigned int size) +{ + struct pci_cap_saved_state *save_state; + + if (cap <= PCI_EXT_CAP_ID_MAX) + return -EINVAL; + + save_state = kzalloc(sizeof(*save_state) + size, GFP_KERNEL); + if (!save_state) + return -ENOMEM; + + save_state->cap.cap_nr = cap; + save_state->cap.cap_extended = true; + save_state->cap.size = size; + pci_add_saved_cap(dev, save_state); + + return 0; +} + /** * pci_allocate_cap_save_buffers - allocate buffers for saving capabilities * @dev: the PCI device diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 13fa71f965900..cae087dec6fb1 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -245,15 +245,33 @@ struct pci_cap_saved_state { struct pci_cap_saved_data cap; }; +/* + * Virtual extended cap ID for CXL DVSEC state in the cap save chain. + */ +#define PCI_EXT_CAP_ID_CXL_DVSEC_VIRTUAL 0xFFFF +static_assert(PCI_EXT_CAP_ID_MAX < PCI_EXT_CAP_ID_CXL_DVSEC_VIRTUAL); + void pci_allocate_cap_save_buffers(struct pci_dev *dev); void pci_free_cap_save_buffers(struct pci_dev *dev); int pci_add_cap_save_buffer(struct pci_dev *dev, char cap, unsigned int size); int pci_add_ext_cap_save_buffer(struct pci_dev *dev, u16 cap, unsigned int size); +int pci_add_virtual_ext_cap_save_buffer(struct pci_dev *dev, u16 cap, + unsigned int size); struct pci_cap_saved_state *pci_find_saved_cap(struct pci_dev *dev, char cap); struct pci_cap_saved_state *pci_find_saved_ext_cap(struct pci_dev *dev, u16 cap); +#ifdef CONFIG_PCI_CXL +void pci_allocate_cxl_save_buffer(struct pci_dev *dev); +void pci_save_cxl_state(struct pci_dev *dev); +void pci_restore_cxl_state(struct pci_dev *dev); +#else +static inline void pci_allocate_cxl_save_buffer(struct pci_dev *dev) { } +static inline void pci_save_cxl_state(struct pci_dev *dev) { } +static inline void pci_restore_cxl_state(struct pci_dev *dev) { } +#endif + #define PCI_PM_D2_DELAY 200 /* usec; see PCIe r4.0, sec 5.9.1 */ #define PCI_PM_D3HOT_WAIT 10 /* msec */ #define PCI_PM_D3COLD_WAIT 100 /* msec */ From 608c2f90be92d402c1e61ce091f3b68933088e76 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 08:00:18 +0000 Subject: [PATCH 259/464] NVIDIA: VR: SAUCE: PCI: Add cxl DVSEC state save/restore across resets BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Save and restore CXL DVSEC control registers (CTRL, CTRL2), range base registers, and lock state across PCI resets. When the DVSEC CONFIG_LOCK bit is set, certain DVSEC fields become read-only and hardware may have updated them. Blindly restoring saved values would be silently ignored or conflict with hardware state. Instead, a read-merge-write approach is used: current hardware values are read for the RWL (read-write-when-locked) fields and merged with saved state, so only writable bits are restored while locked bits retain their hardware values. Hooked into pci_save_state()/pci_restore_state() so all PCI reset paths automatically preserve CXL DVSEC configuration. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/) [jan: Resolve minor conflict in drivers/pci/Makefile due to code line shifts ] Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit 0bb0dc0223282c6c1bd7b7cdb345848d12d41e4d nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/pci/Kconfig | 4 + drivers/pci/Makefile | 1 + drivers/pci/cxl.c | 177 +++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 3 + 4 files changed, 185 insertions(+) create mode 100644 drivers/pci/cxl.c diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index e3f848ffb52a7..6b96650b3f311 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -119,6 +119,10 @@ config XEN_PCIDEV_FRONTEND The PCI device frontend driver allows the kernel to import arbitrary PCI devices from a PCI backend to support PCI driver domains. +config PCI_CXL + bool + default y if CXL_BUS + config PCI_ATS bool diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile index 41ebc3b9a5182..b6b5c9dbaaac7 100644 --- a/drivers/pci/Makefile +++ b/drivers/pci/Makefile @@ -40,6 +40,7 @@ obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o obj-$(CONFIG_PCI_NPEM) += npem.o obj-$(CONFIG_PCIE_TPH) += tph.o obj-$(CONFIG_CARDBUS) += setup-cardbus.o +obj-$(CONFIG_PCI_CXL) += cxl.o # Endpoint library must be initialized before its users obj-$(CONFIG_PCI_ENDPOINT) += endpoint/ diff --git a/drivers/pci/cxl.c b/drivers/pci/cxl.c new file mode 100644 index 0000000000000..abcf70de91715 --- /dev/null +++ b/drivers/pci/cxl.c @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * CXL PCI state save/restore support. + * + * Saves and restores CXL DVSEC registers across PCI resets and link + * disable/enable transitions. Hooked into pci_save_state() / + * pci_restore_state() via the PCI capability save chain. + */ +#include +#include +#include "pci.h" + +struct cxl_pci_state { + u16 dvsec; + u16 dvsec_ctrl; + u16 dvsec_ctrl2; + u32 range_base_hi[CXL_DVSEC_RANGE_MAX]; + u32 range_base_lo[CXL_DVSEC_RANGE_MAX]; + u16 dvsec_lock; + bool dvsec_valid; +}; + +static void cxl_save_dvsec(struct pci_dev *pdev, struct cxl_pci_state *state) +{ + int rc_ctrl, rc_ctrl2; + u16 dvsec; + int i; + + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return; + + state->dvsec = dvsec; + rc_ctrl = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL, + &state->dvsec_ctrl); + rc_ctrl2 = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + &state->dvsec_ctrl2); + if (rc_ctrl || rc_ctrl2) { + pci_warn(pdev, + "CXL: DVSEC read failed (ctrl rc=%d, ctrl2 rc=%d)\n", + rc_ctrl, rc_ctrl2); + return; + } + + for (i = 0; i < CXL_DVSEC_RANGE_MAX; i++) { + pci_read_config_dword(pdev, + dvsec + PCI_DVSEC_CXL_RANGE_BASE_HIGH(i), + &state->range_base_hi[i]); + pci_read_config_dword(pdev, + dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(i), + &state->range_base_lo[i]); + } + + pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_LOCK, + &state->dvsec_lock); + + state->dvsec_valid = true; +} + +static u32 cxl_merge_rwl(u32 saved, u32 current_hw, u32 rwl_mask) +{ + return (current_hw & rwl_mask) | (saved & ~rwl_mask); +} + +static void cxl_restore_dvsec(struct pci_dev *pdev, + const struct cxl_pci_state *state) +{ + u16 lock_reg = 0; + int i; + + if (!state->dvsec_valid) + return; + + pci_read_config_word(pdev, state->dvsec + PCI_DVSEC_CXL_LOCK, + &lock_reg); + + if (lock_reg & PCI_DVSEC_CXL_LOCK_CONFIG) { + u16 hw_ctrl; + u32 hw_range_hi, hw_range_lo; + + pci_read_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_CTRL, + &hw_ctrl); + pci_write_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_CTRL, + cxl_merge_rwl(state->dvsec_ctrl, hw_ctrl, + PCI_DVSEC_CXL_CTRL_RWL)); + + pci_write_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_CTRL2, + state->dvsec_ctrl2); + + for (i = 0; i < CXL_DVSEC_RANGE_MAX; i++) { + pci_read_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_HIGH(i), + &hw_range_hi); + pci_write_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_HIGH(i), + cxl_merge_rwl(state->range_base_hi[i], + hw_range_hi, + PCI_DVSEC_CXL_RANGE_BASE_HI_RWL)); + + pci_read_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(i), + &hw_range_lo); + pci_write_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(i), + cxl_merge_rwl(state->range_base_lo[i], + hw_range_lo, + PCI_DVSEC_CXL_RANGE_BASE_LO_RWL)); + } + } else { + pci_write_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_CTRL, + state->dvsec_ctrl); + pci_write_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_CTRL2, + state->dvsec_ctrl2); + for (i = 0; i < CXL_DVSEC_RANGE_MAX; i++) { + pci_write_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_HIGH(i), + state->range_base_hi[i]); + pci_write_config_dword(pdev, + state->dvsec + PCI_DVSEC_CXL_RANGE_BASE_LOW(i), + state->range_base_lo[i]); + } + + pci_write_config_word(pdev, + state->dvsec + PCI_DVSEC_CXL_LOCK, + state->dvsec_lock); + } +} + +void pci_allocate_cxl_save_buffer(struct pci_dev *dev) +{ + if (!pcie_is_cxl(dev)) + return; + + if (pci_add_virtual_ext_cap_save_buffer(dev, + PCI_EXT_CAP_ID_CXL_DVSEC_VIRTUAL, + sizeof(struct cxl_pci_state))) + pci_err(dev, "unable to allocate CXL save buffer\n"); +} + +void pci_save_cxl_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *save_state; + struct cxl_pci_state *state; + + save_state = pci_find_saved_ext_cap(pdev, + PCI_EXT_CAP_ID_CXL_DVSEC_VIRTUAL); + if (!save_state) + return; + + state = (struct cxl_pci_state *)save_state->cap.data; + state->dvsec_valid = false; + + cxl_save_dvsec(pdev, state); +} + +void pci_restore_cxl_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *save_state; + struct cxl_pci_state *state; + + save_state = pci_find_saved_ext_cap(pdev, + PCI_EXT_CAP_ID_CXL_DVSEC_VIRTUAL); + if (!save_state) + return; + + state = (struct cxl_pci_state *)save_state->cap.data; + if (!state->dvsec_valid) + return; + + cxl_restore_dvsec(pdev, state); +} diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 10c874730e0df..1b7e69f9ff116 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1759,6 +1759,7 @@ int pci_save_state(struct pci_dev *dev) pci_save_aer_state(dev); pci_save_ptm_state(dev); pci_save_tph_state(dev); + pci_save_cxl_state(dev); return pci_save_vc_state(dev); } EXPORT_SYMBOL(pci_save_state); @@ -1841,6 +1842,7 @@ void pci_restore_state(struct pci_dev *dev) pci_restore_aer_state(dev); pci_restore_config_space(dev); + pci_restore_cxl_state(dev); pci_restore_pcix_state(dev); pci_restore_msi_state(dev); @@ -3488,6 +3490,7 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev) pci_err(dev, "unable to allocate suspend buffer for LTR\n"); pci_allocate_vc_save_buffers(dev); + pci_allocate_cxl_save_buffer(dev); } void pci_free_cap_save_buffers(struct pci_dev *dev) From b059513dbc3bb94bfcd087981e9871fc54a81af2 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 08:00:19 +0000 Subject: [PATCH 260/464] NVIDIA: VR: SAUCE: PCI: Add HDM decoder state save/restore BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Save and restore CXL HDM decoder registers (global control, per-decoder base/size/target-list, and commit state) across PCI resets. On restore, decoders that were committed are reprogrammed and recommitted with a 10ms timeout. Locked decoders that are already committed are skipped, since their state is protected by hardware and reprogramming them would fail. The Register Locator DVSEC is parsed directly via PCI config space reads rather than calling cxl_find_regblock()/cxl_setup_regs(), since this code lives in the PCI core and must not depend on CXL module symbols. MSE is temporarily enabled during save/restore to allow MMIO access to the HDM decoder register block. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306080026.116789-1-smadhavan@nvidia.com/) [jan: Include in drivers/pci/cxl.c due to conflict resolution in "4acbc27592b8 NVIDIA: VR: SAUCE: cxl: Move HDM decoder and register map definitions to include/cxl/cxl.h"] Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit 578f9164046dd36b52bd0872095199a13f4164d1 nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/pci/cxl.c | 308 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 305 insertions(+), 3 deletions(-) diff --git a/drivers/pci/cxl.c b/drivers/pci/cxl.c index abcf70de91715..eee28a3755a23 100644 --- a/drivers/pci/cxl.c +++ b/drivers/pci/cxl.c @@ -2,15 +2,32 @@ /* * CXL PCI state save/restore support. * - * Saves and restores CXL DVSEC registers across PCI resets and link - * disable/enable transitions. Hooked into pci_save_state() / + * Saves and restores CXL DVSEC and HDM decoder registers across PCI resets + * and link disable/enable transitions. Hooked into pci_save_state() / * pci_restore_state() via the PCI capability save chain. */ #include +#include +#include +#include +#include #include #include "pci.h" +#define CXL_HDM_MAX_DECODERS 32 + +struct cxl_hdm_decoder_snapshot { + u32 base_lo; + u32 base_hi; + u32 size_lo; + u32 size_hi; + u32 ctrl; + u32 tl_lo; + u32 tl_hi; +}; + struct cxl_pci_state { + /* DVSEC saved state */ u16 dvsec; u16 dvsec_ctrl; u16 dvsec_ctrl2; @@ -18,6 +35,15 @@ struct cxl_pci_state { u32 range_base_lo[CXL_DVSEC_RANGE_MAX]; u16 dvsec_lock; bool dvsec_valid; + + /* HDM decoder saved state */ + int hdm_bar; + unsigned long hdm_bar_offset; + unsigned long hdm_map_size; + u32 hdm_global_ctrl; + int hdm_count; + struct cxl_hdm_decoder_snapshot decoders[CXL_HDM_MAX_DECODERS]; + bool hdm_valid; }; static void cxl_save_dvsec(struct pci_dev *pdev, struct cxl_pci_state *state) @@ -132,6 +158,279 @@ static void cxl_restore_dvsec(struct pci_dev *pdev, } } +struct pci_cmd_saved { + struct pci_dev *pdev; + u16 cmd; +}; + +DEFINE_FREE(restore_pci_cmd, struct pci_cmd_saved, + if (!(_T.cmd & PCI_COMMAND_MEMORY)) + pci_write_config_word(_T.pdev, PCI_COMMAND, _T.cmd)) + +/** + * cxl_find_component_regblock - Find the Component Register Block via + * the Register Locator DVSEC + * @pdev: PCI device to scan + * @bir: output BAR index + * @offset: output offset within the BAR + * + * Parses the Register Locator DVSEC (ID 8) directly via PCI config space + * reads. No dependency on CXL module symbols. + * + * Return: 0 on success, -ENODEV if not found. + */ +static int cxl_find_component_regblock(struct pci_dev *pdev, + int *bir, u64 *offset) +{ + u32 regloc_size, regblocks; + u16 regloc; + int i; + + regloc = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_REG_LOCATOR); + if (!regloc) + return -ENODEV; + + pci_read_config_dword(pdev, regloc + PCI_DVSEC_HEADER1, ®loc_size); + regloc_size = PCI_DVSEC_HEADER1_LEN(regloc_size); + regblocks = (regloc_size - PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1) / 8; + + for (i = 0; i < regblocks; i++) { + u32 reg_lo, reg_hi; + unsigned int off; + + off = regloc + PCI_DVSEC_CXL_REG_LOCATOR_BLOCK1 + i * 8; + pci_read_config_dword(pdev, off, ®_lo); + pci_read_config_dword(pdev, off + 4, ®_hi); + + if (FIELD_GET(PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_ID, reg_lo) != + CXL_REGLOC_RBI_COMPONENT) + continue; + + *bir = FIELD_GET(PCI_DVSEC_CXL_REG_LOCATOR_BIR, reg_lo); + *offset = ((u64)reg_hi << 32) | + (reg_lo & PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW); + return 0; + } + + return -ENODEV; +} + +/* + * Discover and map HDM decoder registers. + * Caller must pci_iounmap() the returned pointer. + */ +static void __iomem *cxl_hdm_map(struct pci_dev *pdev, int *bar_out, + unsigned long *offset_out, + unsigned long *size_out) +{ + int bir; + u64 reg_offset; + void __iomem *comp_base, *cm_base; + u32 cap_hdr; + int cap, cap_count; + unsigned long hdm_offset = 0, hdm_size = 0; + void __iomem *hdm; + + if (cxl_find_component_regblock(pdev, &bir, ®_offset)) + return NULL; + + comp_base = pci_iomap_range(pdev, bir, reg_offset, + CXL_CM_OFFSET + SZ_4K); + if (!comp_base) + return NULL; + + cm_base = comp_base + CXL_CM_OFFSET; + cap_hdr = readl(cm_base); + + if (FIELD_GET(CXL_CM_CAP_HDR_ID_MASK, cap_hdr) != CM_CAP_HDR_CAP_ID) { + pci_iounmap(pdev, comp_base); + return NULL; + } + + cap_count = FIELD_GET(CXL_CM_CAP_HDR_ARRAY_SIZE_MASK, cap_hdr); + + for (cap = 1; cap <= cap_count; cap++) { + u16 cap_id; + u32 cap_off; + u32 hdr; + + if (cap * sizeof(u32) >= SZ_4K) + break; + + hdr = readl(cm_base + cap * 4); + cap_id = FIELD_GET(CXL_CM_CAP_HDR_ID_MASK, hdr); + cap_off = FIELD_GET(CXL_CM_CAP_PTR_MASK, hdr); + + if (cap_id != CXL_CM_CAP_CAP_ID_HDM) + continue; + + if (cap_off + sizeof(u32) > SZ_4K) + break; + + hdr = readl(cm_base + cap_off); + hdm_offset = CXL_CM_OFFSET + cap_off; + hdm_size = 0x20 * cxl_hdm_decoder_count(hdr) + 0x10; + break; + } + + pci_iounmap(pdev, comp_base); + + if (!hdm_size) + return NULL; + + hdm = pci_iomap_range(pdev, bir, reg_offset + hdm_offset, hdm_size); + if (!hdm) + return NULL; + + *bar_out = bir; + *offset_out = reg_offset + hdm_offset; + *size_out = hdm_size; + return hdm; +} + +static void cxl_save_hdm(struct pci_dev *pdev, void __iomem *hdm, + struct cxl_pci_state *state, int count) +{ + int i; + + state->hdm_count = min_t(int, count, CXL_HDM_MAX_DECODERS); + state->hdm_global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET); + + for (i = 0; i < state->hdm_count; i++) { + struct cxl_hdm_decoder_snapshot *d = &state->decoders[i]; + + d->base_lo = readl(hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(i)); + d->base_hi = readl(hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i)); + d->size_lo = readl(hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i)); + d->size_hi = readl(hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i)); + d->ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); + d->tl_lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(i)); + d->tl_hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(i)); + } +} + +static void cxl_restore_hdm(struct pci_dev *pdev, void __iomem *hdm, + const struct cxl_pci_state *state) +{ + int i; + + writel(state->hdm_global_ctrl, hdm + CXL_HDM_DECODER_CTRL_OFFSET); + + for (i = 0; i < state->hdm_count; i++) { + const struct cxl_hdm_decoder_snapshot *d = &state->decoders[i]; + unsigned long timeout; + u32 ctrl; + + if (!(d->ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)) + continue; + + ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); + if ((ctrl & CXL_HDM_DECODER0_CTRL_LOCK) && + (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)) + continue; + + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) { + ctrl &= ~CXL_HDM_DECODER0_CTRL_COMMIT; + writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); + } + + writel(d->base_lo, hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(i)); + writel(d->base_hi, hdm + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i)); + writel(d->size_lo, hdm + CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i)); + writel(d->size_hi, hdm + CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i)); + writel(d->tl_lo, hdm + CXL_HDM_DECODER0_TL_LOW(i)); + writel(d->tl_hi, hdm + CXL_HDM_DECODER0_TL_HIGH(i)); + + wmb(); + + ctrl = d->ctrl & ~(CXL_HDM_DECODER0_CTRL_COMMITTED | + CXL_HDM_DECODER0_CTRL_COMMIT_ERROR); + ctrl |= CXL_HDM_DECODER0_CTRL_COMMIT; + writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); + + timeout = jiffies + msecs_to_jiffies(10); + for (;;) { + ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(i)); + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) + break; + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMIT_ERROR) { + pci_warn(pdev, + "HDM decoder %d commit error on restore\n", + i); + break; + } + if (time_after(jiffies, timeout)) { + pci_warn(pdev, + "HDM decoder %d commit timeout on restore\n", + i); + break; + } + cpu_relax(); + } + } +} + +static void cxl_save_hdm_decoders(struct pci_dev *pdev, + struct cxl_pci_state *state) +{ + int hdm_bar; + unsigned long hdm_bar_offset, hdm_map_size; + void __iomem *hdm; + u16 cmd; + u32 cap; + struct pci_cmd_saved saved __free(restore_pci_cmd) = { + .pdev = pdev, .cmd = PCI_COMMAND_MEMORY, + }; + + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + saved.cmd = cmd; + if (!(cmd & PCI_COMMAND_MEMORY)) + pci_write_config_word(pdev, PCI_COMMAND, + cmd | PCI_COMMAND_MEMORY); + + hdm = cxl_hdm_map(pdev, &hdm_bar, &hdm_bar_offset, &hdm_map_size); + if (!hdm) + return; + + cap = readl(hdm + CXL_HDM_DECODER_CAP_OFFSET); + cxl_save_hdm(pdev, hdm, state, cxl_hdm_decoder_count(cap)); + state->hdm_bar = hdm_bar; + state->hdm_bar_offset = hdm_bar_offset; + state->hdm_map_size = hdm_map_size; + state->hdm_valid = true; + pci_iounmap(pdev, hdm); +} + +static void cxl_restore_hdm_decoders(struct pci_dev *pdev, + const struct cxl_pci_state *state) +{ + void __iomem *hdm; + u16 cmd; + struct pci_cmd_saved saved __free(restore_pci_cmd) = { + .pdev = pdev, .cmd = PCI_COMMAND_MEMORY, + }; + + if (!state->hdm_valid) + return; + + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + saved.cmd = cmd; + if (!(cmd & PCI_COMMAND_MEMORY)) + pci_write_config_word(pdev, PCI_COMMAND, + cmd | PCI_COMMAND_MEMORY); + + hdm = pci_iomap_range(pdev, state->hdm_bar, state->hdm_bar_offset, + state->hdm_map_size); + if (!hdm) { + pci_warn(pdev, "CXL: failed to map HDM for restore\n"); + return; + } + + cxl_restore_hdm(pdev, hdm, state); + pci_iounmap(pdev, hdm); +} + void pci_allocate_cxl_save_buffer(struct pci_dev *dev) { if (!pcie_is_cxl(dev)) @@ -155,8 +454,10 @@ void pci_save_cxl_state(struct pci_dev *pdev) state = (struct cxl_pci_state *)save_state->cap.data; state->dvsec_valid = false; + state->hdm_valid = false; cxl_save_dvsec(pdev, state); + cxl_save_hdm_decoders(pdev, state); } void pci_restore_cxl_state(struct pci_dev *pdev) @@ -170,8 +471,9 @@ void pci_restore_cxl_state(struct pci_dev *pdev) return; state = (struct cxl_pci_state *)save_state->cap.data; - if (!state->dvsec_valid) + if (!state->dvsec_valid && !state->hdm_valid) return; cxl_restore_dvsec(pdev, state); + cxl_restore_hdm_decoders(pdev, state); } From 34b908edeee7e30d7680f28703925e918a6e2fc1 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:16 +0000 Subject: [PATCH 261/464] NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC reset and capability register definitions BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Add CXL DVSEC register definitions needed for CXL device reset per CXL r3.2 section 8.1.3.1: - Capability bits: RST_CAPABLE, CACHE_CAPABLE, CACHE_WBI_CAPABLE, RST_TIMEOUT, RST_MEM_CLR_CAPABLE - Control2 register: DISABLE_CACHING, INIT_CACHE_WBI, INIT_CXL_RST, RST_MEM_CLR_EN - Status2 register: CACHE_INV, RST_DONE, RST_ERR - Non-CXL Function Map DVSEC register offset Signed-off-by: Srirangan Madhavan (backported from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) [jan: Resolve conflicts where PCI_DVSEC_CXL_CACHE_CAPABLE is already added by "72bd823fb4f1 NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable devices"] Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit 8f77fbe3816c18bf2a7c8aa61bea89c00860790d nv-kernels/24.04_linux-nvidia-6.17-next) [koba: Preserve PCI_DVSEC_CXL_CACHE_CAPABLE because drivers/pci/ats.c still uses it for CXL.cache ATS dependency from commit 37654885ec90c (6.17 source commit 72bd823fb4f1).] Signed-off-by: Koba Ko Acked-by: Matt Ochs --- include/uapi/linux/pci_regs.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index 148328bd4cb37..c4a6c0a916dc7 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1349,12 +1349,25 @@ /* CXL r4.0, 8.1.3: PCIe DVSEC for CXL Device */ #define PCI_DVSEC_CXL_DEVICE 0 #define PCI_DVSEC_CXL_CAP 0xA +#define PCI_DVSEC_CXL_CACHE_CAPABLE _BITUL(0) #define PCI_DVSEC_CXL_MEM_CAPABLE _BITUL(2) #define PCI_DVSEC_CXL_HDM_COUNT __GENMASK(5, 4) +#define PCI_DVSEC_CXL_CACHE_WBI_CAPABLE _BITUL(6) +#define PCI_DVSEC_CXL_RST_CAPABLE _BITUL(7) +#define PCI_DVSEC_CXL_RST_TIMEOUT __GENMASK(10, 8) +#define PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE _BITUL(11) #define PCI_DVSEC_CXL_CTRL 0xC #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) #define PCI_DVSEC_CXL_CTRL_RWL 0x5FED #define PCI_DVSEC_CXL_CTRL2 0x10 +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) +#define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) +#define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) +#define PCI_DVSEC_CXL_STATUS2 0x12 +#define PCI_DVSEC_CXL_CACHE_INV _BITUL(0) +#define PCI_DVSEC_CXL_RST_DONE _BITUL(1) +#define PCI_DVSEC_CXL_RST_ERR _BITUL(2) #define PCI_DVSEC_CXL_LOCK 0x14 #define PCI_DVSEC_CXL_LOCK_CONFIG _BITUL(0) #define PCI_DVSEC_CXL_RANGE_SIZE_HIGH(i) (0x18 + (i * 0x10)) @@ -1373,6 +1386,7 @@ /* CXL r4.0, 8.1.4: Non-CXL Function Map DVSEC */ #define PCI_DVSEC_CXL_FUNCTION_MAP 2 +#define PCI_DVSEC_CXL_FUNCTION_MAP_REG 0x0C /* CXL r4.0, 8.1.5: Extensions DVSEC for Ports */ #define PCI_DVSEC_CXL_PORT 3 From 5324088768b5984bd637ac5ee85873bffed028ee Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:17 +0000 Subject: [PATCH 262/464] NVIDIA: VR: SAUCE: PCI: Export pci_dev_save_and_disable() and pci_dev_restore() BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Export pci_dev_save_and_disable() and pci_dev_restore() so that subsystems performing non-standard reset sequences (e.g. CXL) can reuse the PCI core standard pre/post reset lifecycle: driver reset_prepare/reset_done callbacks, PCI config space save/restore, and device disable/re-enable. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit a14a427ac483aa939aba9173673636bb71a07784 nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/pci/pci.c | 21 +++++++++++++++++++-- include/linux/pci.h | 3 +++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 1b7e69f9ff116..2f8e7dfd391eb 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -5031,7 +5031,15 @@ void pci_dev_unlock(struct pci_dev *dev) } EXPORT_SYMBOL_GPL(pci_dev_unlock); -static void pci_dev_save_and_disable(struct pci_dev *dev) +/** + * pci_dev_save_and_disable - Save device state and disable it + * @dev: PCI device to save and disable + * + * Save the PCI configuration state, invoke the driver's reset_prepare + * callback (if any), and disable the device by clearing the Command register. + * The device lock must be held by the caller. + */ +void pci_dev_save_and_disable(struct pci_dev *dev) { const struct pci_error_handlers *err_handler = dev->driver ? dev->driver->err_handler : NULL; @@ -5064,8 +5072,16 @@ static void pci_dev_save_and_disable(struct pci_dev *dev) */ pci_write_config_word(dev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE); } +EXPORT_SYMBOL_GPL(pci_dev_save_and_disable); -static void pci_dev_restore(struct pci_dev *dev) +/** + * pci_dev_restore - Restore device state after reset + * @dev: PCI device to restore + * + * Restore the saved PCI configuration state and invoke the driver's + * reset_done callback (if any). The device lock must be held by the caller. + */ +void pci_dev_restore(struct pci_dev *dev) { const struct pci_error_handlers *err_handler = dev->driver ? dev->driver->err_handler : NULL; @@ -5082,6 +5098,7 @@ static void pci_dev_restore(struct pci_dev *dev) else if (dev->driver) pci_warn(dev, "reset done"); } +EXPORT_SYMBOL_GPL(pci_dev_restore); /* dev->reset_methods[] is a 0-terminated list of indices into this array */ const struct pci_reset_fn_method pci_reset_fn_methods[] = { diff --git a/include/linux/pci.h b/include/linux/pci.h index 14bbdea91919e..6137e0b4b964a 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2002,6 +2002,9 @@ int pci_dev_trylock(struct pci_dev *dev); void pci_dev_unlock(struct pci_dev *dev); DEFINE_GUARD(pci_dev, struct pci_dev *, pci_dev_lock(_T), pci_dev_unlock(_T)) +void pci_dev_save_and_disable(struct pci_dev *dev); +void pci_dev_restore(struct pci_dev *dev); + /* * PCI domain support. Sometimes called PCI segment (eg by ACPI), * a PCI domain is defined to be a set of PCI buses which share From 1c5ac1eb313fae4dbca2577b03682164170de391 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:18 +0000 Subject: [PATCH 263/464] NVIDIA: VR: SAUCE: cxl: Add memory offlining and cache flush helpers BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Add infrastructure for quiescing the CXL data path before reset: - Memory offlining: check if CXL-backed memory is online and offline it via offline_and_remove_memory() before reset, per CXL spec requirement to quiesce all CXL.mem transactions before issuing CXL Reset. - CPU cache flush: invalidate cache lines before reset as a safety measure after memory offline. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit 98bfbf9c3f88013ffbff4b08a1da0043606d0269 nv-kernels/24.04_linux-nvidia-6.17-next) [koba: Use a real System RAM walker callback so resource walks never invoke a NULL function pointer.] Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/cxl/core/pci.c | 120 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 2bcd683aa286d..84e3f037ea71d 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -4,6 +4,8 @@ #include #include #include +#include +#include #include #include #include @@ -927,3 +929,121 @@ int cxl_port_get_possible_dports(struct cxl_port *port) return ctx.count; } + +/* + * CXL Reset support - core-provided reset logic for CXL devices. + * + * These functions implement the CXL reset sequence. + */ + +/* + * If CXL memory backed by this decoder is online as System RAM, offline + * and remove it per CXL spec requirements before issuing CXL Reset. + * Returns 0 if memory was not online or was successfully offlined. + */ +static int cxl_is_system_ram(struct resource *res, void *arg) +{ + return 1; +} + +static int __maybe_unused cxl_offline_memory(struct device *dev, void *data) +{ + struct cxl_endpoint_decoder *cxled; + struct cxl_region *cxlr; + struct cxl_region_params *p; + int rc; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled = to_cxl_endpoint_decoder(dev); + guard(rwsem_read)(&cxl_rwsem.region); + + cxlr = cxled->cxld.region; + if (!cxlr) + return 0; + + p = &cxlr->params; + if (!p->res) + return 0; + + if (walk_iomem_res_desc(IORES_DESC_NONE, + IORESOURCE_SYSTEM_RAM | IORESOURCE_BUSY, + p->res->start, p->res->end, NULL, + cxl_is_system_ram) <= 0) + return 0; + + dev_info(dev, "Offlining CXL memory [%pr] for reset\n", p->res); + +#ifdef CONFIG_MEMORY_HOTREMOVE + rc = offline_and_remove_memory(p->res->start, resource_size(p->res)); + if (rc) { + dev_err(dev, + "Failed to offline CXL memory [%pr]: %d\n", + p->res, rc); + return rc; + } +#else + dev_err(dev, "Memory hotremove not supported, cannot offline CXL memory\n"); + rc = -EOPNOTSUPP; + return rc; +#endif + + return 0; +} + +static int __maybe_unused cxl_reset_prepare_memdev(struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint; + struct device *dev; + + if (!cxlmd || !cxlmd->cxlds) + return -ENODEV; + + dev = cxlmd->cxlds->dev; + endpoint = cxlmd->endpoint; + if (!endpoint) + return 0; + + return device_for_each_child(&endpoint->dev, NULL, + cxl_offline_memory); +} + +static int __maybe_unused cxl_decoder_flush_cache(struct device *dev, void *data) +{ + struct cxl_endpoint_decoder *cxled; + struct cxl_region *cxlr; + struct resource *res; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled = to_cxl_endpoint_decoder(dev); + guard(rwsem_read)(&cxl_rwsem.region); + + cxlr = cxled->cxld.region; + if (!cxlr || !cxlr->params.res) + return 0; + + res = cxlr->params.res; + cpu_cache_invalidate_memregion(res->start, resource_size(res)); + return 0; +} + +static int __maybe_unused cxl_reset_flush_cpu_caches(struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint; + + if (!cxlmd) + return 0; + + endpoint = cxlmd->endpoint; + if (!endpoint || IS_ERR(endpoint)) + return 0; + + if (!cpu_cache_has_invalidate_memregion()) + return 0; + + device_for_each_child(&endpoint->dev, NULL, cxl_decoder_flush_cache); + return 0; +} From 708d9558659ec381a29902c6b3f6458d21757f97 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:19 +0000 Subject: [PATCH 264/464] NVIDIA: VR: SAUCE: cxl: Add multi-function sibling coordination for CXL reset BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Add sibling PCI function save/disable/restore coordination for CXL reset. Before reset, all CXL.cachemem sibling functions are locked, saved, and disabled; after reset they are restored. The Non-CXL Function Map DVSEC and per-function DVSEC capability register are consulted to skip non-CXL and CXL.io-only functions. A global mutex serializes concurrent resets to prevent deadlocks between sibling functions. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit 9a08c0246be53f047ed4128455f708b7a4350261 nv-kernels/24.04_linux-nvidia-6.17-next) [koba: Propagate sibling collection allocation failures after pci_walk_bus() so reset aborts instead of proceeding with a partial sibling list.] Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/cxl/core/pci.c | 156 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 84e3f037ea71d..5949c5d15787b 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -16,6 +16,9 @@ #include "core.h" #include "trace.h" +/* Initial sibling array capacity: covers max non-ARI functions per slot */ +#define CXL_RESET_SIBLINGS_INIT 8 + /** * DOC: cxl core pci * @@ -1047,3 +1050,156 @@ static int __maybe_unused cxl_reset_flush_cpu_caches(struct cxl_memdev *cxlmd) device_for_each_child(&endpoint->dev, NULL, cxl_decoder_flush_cache); return 0; } + +/* + * Serialize all CXL reset operations globally. + */ +static DEFINE_MUTEX(cxl_reset_mutex); + +struct cxl_reset_context { + struct pci_dev *target; + struct pci_dev **pci_functions; + int pci_func_count; + int pci_func_cap; +}; + +/* + * Check if a sibling function is non-CXL using the Non-CXL Function Map + * DVSEC. Returns true if fn is listed as non-CXL, false otherwise (including + * on any read failure). + */ +static bool cxl_is_non_cxl_function(struct pci_dev *pdev, + u16 func_map_dvsec, int fn) +{ + int reg, bit; + u32 map; + + if (pci_ari_enabled(pdev->bus)) { + reg = fn / 32; + bit = fn % 32; + } else { + reg = 0; + bit = fn; + } + + if (pci_read_config_dword(pdev, + func_map_dvsec + PCI_DVSEC_CXL_FUNCTION_MAP_REG + (reg * 4), + &map)) + return false; + + return map & BIT(bit); +} + +struct cxl_reset_walk_ctx { + struct cxl_reset_context *ctx; + u16 func_map_dvsec; + int error; + bool ari; +}; + +static int cxl_reset_collect_sibling(struct pci_dev *func, void *data) +{ + struct cxl_reset_walk_ctx *wctx = data; + struct cxl_reset_context *ctx = wctx->ctx; + struct pci_dev *pdev = ctx->target; + u16 dvsec, cap; + int fn; + + if (func == pdev) + return 0; + + if (!wctx->ari && + PCI_SLOT(func->devfn) != PCI_SLOT(pdev->devfn)) + return 0; + + fn = wctx->ari ? func->devfn : PCI_FUNC(func->devfn); + if (wctx->func_map_dvsec && + cxl_is_non_cxl_function(pdev, wctx->func_map_dvsec, fn)) + return 0; + + /* Only coordinate with siblings that have CXL.cachemem */ + dvsec = pci_find_dvsec_capability(func, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return 0; + if (pci_read_config_word(func, dvsec + PCI_DVSEC_CXL_CAP, &cap)) + return 0; + if (!(cap & (PCI_DVSEC_CXL_CACHE_CAPABLE | + PCI_DVSEC_CXL_MEM_CAPABLE))) + return 0; + + /* Grow sibling array; double capacity for ARI devices when running out of space */ + if (ctx->pci_func_count >= ctx->pci_func_cap) { + struct pci_dev **new; + int new_cap = ctx->pci_func_cap ? ctx->pci_func_cap * 2 + : CXL_RESET_SIBLINGS_INIT; + + new = krealloc(ctx->pci_functions, + new_cap * sizeof(*new), GFP_KERNEL); + if (!new) { + wctx->error = -ENOMEM; + return 1; + } + ctx->pci_functions = new; + ctx->pci_func_cap = new_cap; + } + + pci_dev_get(func); + ctx->pci_functions[ctx->pci_func_count++] = func; + return 0; +} + +static void __maybe_unused cxl_pci_functions_reset_release(struct cxl_reset_context *ctx) +{ + int i; + + for (i = 0; i < ctx->pci_func_count; i++) + pci_dev_put(ctx->pci_functions[i]); + kfree(ctx->pci_functions); + ctx->pci_functions = NULL; + ctx->pci_func_count = 0; + ctx->pci_func_cap = 0; +} + +static int __maybe_unused cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) +{ + struct pci_dev *pdev = ctx->target; + struct cxl_reset_walk_ctx wctx; + int i; + + ctx->pci_func_count = 0; + ctx->pci_functions = NULL; + ctx->pci_func_cap = 0; + + wctx.ctx = ctx; + wctx.ari = pci_ari_enabled(pdev->bus); + wctx.error = 0; + wctx.func_map_dvsec = pci_find_dvsec_capability(pdev, + PCI_VENDOR_ID_CXL, PCI_DVSEC_CXL_FUNCTION_MAP); + + /* Collect CXL.cachemem siblings under pci_bus_sem */ + pci_walk_bus(pdev->bus, cxl_reset_collect_sibling, &wctx); + if (wctx.error) { + cxl_pci_functions_reset_release(ctx); + return wctx.error; + } + + /* Lock and save/disable siblings outside pci_bus_sem */ + for (i = 0; i < ctx->pci_func_count; i++) { + pci_dev_lock(ctx->pci_functions[i]); + pci_dev_save_and_disable(ctx->pci_functions[i]); + } + + return 0; +} + +static void __maybe_unused cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) +{ + int i; + + for (i = 0; i < ctx->pci_func_count; i++) { + pci_dev_restore(ctx->pci_functions[i]); + pci_dev_unlock(ctx->pci_functions[i]); + } + cxl_pci_functions_reset_release(ctx); +} From 55d40e199130af531ce845a50f7de41913897d7f Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:20 +0000 Subject: [PATCH 265/464] NVIDIA: VR: SAUCE: cxl: Add CXL DVSEC reset sequence and flow orchestration BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 cxl_dev_reset() implements the hardware reset sequence: optionally enable memory clear, initiate reset via CTRL2, wait for completion, and re-enable caching. cxl_do_reset() orchestrates the full reset flow: 1. CXL pre-reset: mem offlining and cache flush (when memdev present) 2. PCI save/disable: pci_dev_save_and_disable() automatically saves CXL DVSEC and HDM decoder state via PCI core hooks 3. Sibling coordination: save/disable CXL.cachemem sibling functions 4. Execute CXL DVSEC reset 5. Sibling restore: always runs to re-enable sibling functions 6. PCI restore: pci_dev_restore() automatically restores CXL state The CXL-specific DVSEC and HDM save/restore is handled by the PCI core's CXL save/restore infrastructure (drivers/pci/cxl.c). Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (backported from commit 92fb80732a4ea34b76cbe51b15e95ff04f49cb89 nv-kernels/24.04_linux-nvidia-6.17-next) [koba: Treat error-valued cxlmd->endpoint as no endpoint to avoid dereferencing ERR_PTR before endpoint attach.] [koba: Check sibling collection failure before starting the CXL reset so allocation failure restores the target and aborts.] [koba: Limit the memdev device lock to endpoint-dependent memory preparation and cache flush, restore memory quiesce before PCI disable, and track sibling reset preparation so reset_done cleanup only runs after successful sibling prepare.] [koba: Guard reset_done() against NULL/ERR_PTR memdev endpoints before decoder reset detection.] Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/cxl/core/pci.c | 196 ++++++++++++++++++++++++++++++++++++++++- drivers/cxl/pci.c | 8 +- 2 files changed, 198 insertions(+), 6 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 5949c5d15787b..c97ee3d257a87 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1005,7 +1005,7 @@ static int __maybe_unused cxl_reset_prepare_memdev(struct cxl_memdev *cxlmd) dev = cxlmd->cxlds->dev; endpoint = cxlmd->endpoint; - if (!endpoint) + if (!endpoint || IS_ERR(endpoint)) return 0; return device_for_each_child(&endpoint->dev, NULL, @@ -1149,7 +1149,7 @@ static int cxl_reset_collect_sibling(struct pci_dev *func, void *data) return 0; } -static void __maybe_unused cxl_pci_functions_reset_release(struct cxl_reset_context *ctx) +static void cxl_pci_functions_reset_release(struct cxl_reset_context *ctx) { int i; @@ -1161,7 +1161,7 @@ static void __maybe_unused cxl_pci_functions_reset_release(struct cxl_reset_cont ctx->pci_func_cap = 0; } -static int __maybe_unused cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) +static int cxl_pci_functions_reset_prepare(struct cxl_reset_context *ctx) { struct pci_dev *pdev = ctx->target; struct cxl_reset_walk_ctx wctx; @@ -1193,7 +1193,7 @@ static int __maybe_unused cxl_pci_functions_reset_prepare(struct cxl_reset_conte return 0; } -static void __maybe_unused cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) +static void cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) { int i; @@ -1203,3 +1203,191 @@ static void __maybe_unused cxl_pci_functions_reset_done(struct cxl_reset_context } cxl_pci_functions_reset_release(ctx); } + +/* + * CXL device reset execution + */ +static int cxl_dev_reset(struct pci_dev *pdev, int dvsec) +{ + static const u32 reset_timeout_ms[] = { 10, 100, 1000, 10000, 100000 }; + u16 cap, ctrl2, status2; + u32 timeout_ms; + int rc, idx; + + if (!pci_wait_for_pending_transaction(pdev)) + pci_err(pdev, "timed out waiting for pending transactions\n"); + + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap); + if (rc) + return rc; + + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2); + if (rc) + return rc; + + /* + * Disable caching and initiate cache writeback+invalidation if the + * device supports it. Poll for completion. + * Per CXL r3.2 section 9.6, software may use the cache size from + * DVSEC CXL Capability2 to compute a suitable timeout; we use a + * default of 10ms. + */ + if (cap & PCI_DVSEC_CXL_CACHE_WBI_CAPABLE) { + u32 wbi_poll_us = 100; + s32 wbi_remaining_us = 10000; + + ctrl2 |= PCI_DVSEC_CXL_DISABLE_CACHING; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + ctrl2); + if (rc) + return rc; + + ctrl2 |= PCI_DVSEC_CXL_INIT_CACHE_WBI; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + ctrl2); + if (rc) + return rc; + + do { + usleep_range(wbi_poll_us, wbi_poll_us + 1); + wbi_remaining_us -= wbi_poll_us; + rc = pci_read_config_word(pdev, + dvsec + PCI_DVSEC_CXL_STATUS2, + &status2); + if (rc) + return rc; + } while (!(status2 & PCI_DVSEC_CXL_CACHE_INV) && + wbi_remaining_us > 0); + + if (!(status2 & PCI_DVSEC_CXL_CACHE_INV)) { + pci_err(pdev, "CXL cache WB+I timed out\n"); + return -ETIMEDOUT; + } + } else if (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) { + ctrl2 |= PCI_DVSEC_CXL_DISABLE_CACHING; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + ctrl2); + if (rc) + return rc; + } + + if (cap & PCI_DVSEC_CXL_RST_MEM_CLR_CAPABLE) { + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + &ctrl2); + if (rc) + return rc; + + ctrl2 |= PCI_DVSEC_CXL_RST_MEM_CLR_EN; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, + ctrl2); + if (rc) + return rc; + } + + idx = FIELD_GET(PCI_DVSEC_CXL_RST_TIMEOUT, cap); + if (idx >= ARRAY_SIZE(reset_timeout_ms)) + idx = ARRAY_SIZE(reset_timeout_ms) - 1; + timeout_ms = reset_timeout_ms[idx]; + + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2); + if (rc) + return rc; + + ctrl2 |= PCI_DVSEC_CXL_INIT_CXL_RST; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); + if (rc) + return rc; + + msleep(timeout_ms); + + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_STATUS2, + &status2); + if (rc) + return rc; + + if (status2 & PCI_DVSEC_CXL_RST_ERR) { + pci_err(pdev, "CXL reset error\n"); + return -EIO; + } + + if (!(status2 & PCI_DVSEC_CXL_RST_DONE)) { + pci_err(pdev, "CXL reset timeout\n"); + return -ETIMEDOUT; + } + + rc = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, &ctrl2); + if (rc) + return rc; + + ctrl2 &= ~PCI_DVSEC_CXL_DISABLE_CACHING; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); + if (rc) + return rc; + + return 0; +} + +static int match_memdev_by_parent(struct device *dev, const void *parent) +{ + return is_cxl_memdev(dev) && dev->parent == parent; +} + +static int __cxl_do_reset(struct pci_dev *pdev, struct cxl_memdev *cxlmd, + int dvsec) +{ + struct cxl_reset_context ctx = { .target = pdev }; + bool siblings_prepared = false; + int rc; + + mutex_lock(&cxl_reset_mutex); + pci_dev_lock(pdev); + + if (cxlmd) { + guard(device)(&cxlmd->dev); + + rc = cxl_reset_prepare_memdev(cxlmd); + if (rc) + goto out_unlock; + + cxl_reset_flush_cpu_caches(cxlmd); + } + + pci_dev_save_and_disable(pdev); + + rc = cxl_pci_functions_reset_prepare(&ctx); + if (!rc) { + siblings_prepared = true; + rc = cxl_dev_reset(pdev, dvsec); + } + + if (siblings_prepared) + cxl_pci_functions_reset_done(&ctx); + + pci_dev_restore(pdev); + +out_unlock: + pci_dev_unlock(pdev); + mutex_unlock(&cxl_reset_mutex); + + return rc; +} + +static int cxl_do_reset(struct pci_dev *pdev) +{ + int dvsec; + + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return -ENODEV; + + struct device *memdev __free(put_device) = + bus_find_device(&cxl_bus_type, NULL, &pdev->dev, + match_memdev_by_parent); + if (!memdev) + return __cxl_do_reset(pdev, NULL, dvsec); + + struct cxl_memdev *cxlmd = to_cxl_memdev(memdev); + + return __cxl_do_reset(pdev, cxlmd, dvsec); +} diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 9e9239432b4d0..964ce1b7950f5 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -965,6 +965,7 @@ static void cxl_reset_done(struct pci_dev *pdev) { struct cxl_dev_state *cxlds = pci_get_drvdata(pdev); struct cxl_memdev *cxlmd = cxlds->cxlmd; + struct cxl_port *endpoint; struct device *dev = &pdev->dev; /* @@ -977,8 +978,11 @@ static void cxl_reset_done(struct pci_dev *pdev) if (!cxlmd->dev.driver) return; - if (cxlmd->endpoint && - cxl_endpoint_decoder_reset_detected(cxlmd->endpoint)) { + endpoint = cxlmd->endpoint; + if (!endpoint || IS_ERR(endpoint)) + return; + + if (cxl_endpoint_decoder_reset_detected(endpoint)) { dev_crit(dev, "SBR happened without memory regions removal.\n"); dev_crit(dev, "System may be unstable if regions hosted system memory.\n"); add_taint(TAINT_USER, LOCKDEP_STILL_OK); From 3613021b3b31b35302eb4ed2d8a5c2616c2b2c7c Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:21 +0000 Subject: [PATCH 266/464] NVIDIA: VR: SAUCE: cxl: Add cxl_reset sysfs interface for PCI devices BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Add a "cxl_reset" sysfs attribute to PCI devices that support CXL Reset (CXL r3.2 section 8.1.3.1). The attribute is visible only on devices with both CXL.cache and CXL.mem capabilities and the CXL Reset Capable bit set in the DVSEC. Writing "1" to the attribute triggers the full CXL reset flow via cxl_do_reset(). The interface is decoupled from memdev creation: when a CXL memdev exists, memory offlining and cache flush are performed; otherwise reset proceeds without the memory management. The sysfs attribute is managed entirely by the CXL module using sysfs_create_group() / sysfs_remove_group() rather than the PCI core's static attribute groups. This avoids cross-module symbol dependencies between the PCI core (always built-in) and CXL_BUS (potentially modular). At module init, existing PCI devices are scanned and a PCI bus notifier handles hot-plug/unplug. kernfs_drain() makes sure that any in-flight store() completes before sysfs_remove_group() returns, preventing use-after-free during module unload. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit 6e96f7e341a4eb1b9631e40b43d120b2b9e2c6e2 nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- drivers/cxl/core/core.h | 2 + drivers/cxl/core/pci.c | 113 ++++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/port.c | 3 ++ 3 files changed, 118 insertions(+) diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 77c684744771b..3861409ea69e3 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -138,6 +138,8 @@ extern struct cxl_rwsem cxl_rwsem; int cxl_memdev_init(void); void cxl_memdev_exit(void); void cxl_mbox_init(void); +void cxl_reset_sysfs_init(void); +void cxl_reset_sysfs_exit(void); enum cxl_poison_trace_type { CXL_POISON_TRACE_LIST, diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index c97ee3d257a87..6944f0f74790b 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1391,3 +1391,116 @@ static int cxl_do_reset(struct pci_dev *pdev) return __cxl_do_reset(pdev, cxlmd, dvsec); } + +/* + * CXL reset sysfs attribute management. + * + * The cxl_reset attribute is added to PCI devices that advertise CXL Reset + * capability. Managed entirely by the CXL module via subsys_interface on + * pci_bus_type, avoiding cross-module symbol dependencies between the PCI + * core (built-in) and CXL (potentially modular). + * + * subsys_interface handles existing devices at register time and hot-plug + * add/remove automatically. On unregister, remove_dev runs for all tracked + * devices under bus core serialization. + */ + +static bool pci_cxl_reset_capable(struct pci_dev *pdev) +{ + int dvsec; + u16 cap; + + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return false; + + if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap)) + return false; + + if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE) || + !(cap & PCI_DVSEC_CXL_MEM_CAPABLE)) + return false; + + return !!(cap & PCI_DVSEC_CXL_RST_CAPABLE); +} + +static ssize_t cxl_reset_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + int rc; + + if (!sysfs_streq(buf, "1")) + return -EINVAL; + + rc = cxl_do_reset(pdev); + return rc ? rc : count; +} +static DEVICE_ATTR_WO(cxl_reset); + +static umode_t cxl_reset_attr_is_visible(struct kobject *kobj, + struct attribute *a, int n) +{ + struct pci_dev *pdev = to_pci_dev(kobj_to_dev(kobj)); + + if (!pci_cxl_reset_capable(pdev)) + return 0; + + return a->mode; +} + +static struct attribute *cxl_reset_attrs[] = { + &dev_attr_cxl_reset.attr, + NULL, +}; + +static const struct attribute_group cxl_reset_attr_group = { + .attrs = cxl_reset_attrs, + .is_visible = cxl_reset_attr_is_visible, +}; + +static int cxl_reset_add_dev(struct device *dev, + struct subsys_interface *sif) +{ + struct pci_dev *pdev = to_pci_dev(dev); + + if (!pci_cxl_reset_capable(pdev)) + return 0; + + return sysfs_create_group(&dev->kobj, &cxl_reset_attr_group); +} + +static void cxl_reset_remove_dev(struct device *dev, + struct subsys_interface *sif) +{ + struct pci_dev *pdev = to_pci_dev(dev); + + if (!pci_cxl_reset_capable(pdev)) + return; + + sysfs_remove_group(&dev->kobj, &cxl_reset_attr_group); +} + +static struct subsys_interface cxl_reset_interface = { + .name = "cxl_reset", + .subsys = &pci_bus_type, + .add_dev = cxl_reset_add_dev, + .remove_dev = cxl_reset_remove_dev, +}; + +void cxl_reset_sysfs_init(void) +{ + int rc; + + rc = subsys_interface_register(&cxl_reset_interface); + if (rc) + pr_warn("CXL: failed to register cxl_reset interface (%d)\n", + rc); +} + +void cxl_reset_sysfs_exit(void) +{ + subsys_interface_unregister(&cxl_reset_interface); +} diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index dbe30e7c383be..f4f7dad473beb 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -2531,6 +2531,8 @@ static __init int cxl_core_init(void) if (rc) goto err_ras; + cxl_reset_sysfs_init(); + return 0; err_ras: @@ -2546,6 +2548,7 @@ static __init int cxl_core_init(void) static void cxl_core_exit(void) { + cxl_reset_sysfs_exit(); cxl_ras_exit(); cxl_region_exit(); bus_unregister(&cxl_bus_type); From 18c4bcc5e1e8aedaeaee731f3ad1c345db60d092 Mon Sep 17 00:00:00 2001 From: Srirangan Madhavan Date: Fri, 6 Mar 2026 09:23:22 +0000 Subject: [PATCH 267/464] NVIDIA: VR: SAUCE: Documentation: ABI: Add CXL PCI cxl_reset sysfs attribute BugLink: https://bugs.launchpad.net/bugs/2153819 BugLink: https://bugs.launchpad.net/bugs/2143032 Document the cxl_reset sysfs attribute added to PCI devices that support CXL Reset. Signed-off-by: Srirangan Madhavan (cherry picked from https://lore.kernel.org/linux-cxl/20260306092322.148765-1-smadhavan@nvidia.com/) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg (cherry picked from commit 33b53e15c379ec657cc0a4905df161eb9162a3cf nv-kernels/24.04_linux-nvidia-6.17-next) Signed-off-by: Koba Ko Acked-by: Matt Ochs --- Documentation/ABI/testing/sysfs-bus-pci | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci index b767db2c52cb7..d67c733626b83 100644 --- a/Documentation/ABI/testing/sysfs-bus-pci +++ b/Documentation/ABI/testing/sysfs-bus-pci @@ -174,6 +174,28 @@ Description: similiar to writing 1 to their individual "reset" file, so use with caution. +What: /sys/bus/pci/devices/.../cxl_reset +Date: February 2026 +Contact: linux-cxl@vger.kernel.org +Description: + This attribute is only visible when the device advertises + CXL Reset Capable in the CXL DVSEC Capability register + (CXL r3.2, section 8.1.3). + + Writing 1 to this file triggers a CXL device reset which + affects CXL.cache and CXL.mem state on all CXL functions + (i.e. those not listed in the Non-CXL Function Map DVSEC, + section 8.1.4), not just CXL.io/PCIe state. This is + separate from the standard PCI reset interface because CXL + Reset has different scope. + + The reset will fail with -EBUSY if any CXL regions using this + device have drivers bound. Active regions are torn down as + part of the reset sequence. + + This attribute is registered by the CXL core when a CXL device + is discovered, independent of which driver binds the PCI device. + What: /sys/bus/pci/devices/.../vpd Date: February 2008 Contact: Ben Hutchings From ceb7605474d4e3839c819f7bf398b1fed23333c5 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 21 May 2026 15:52:26 -0500 Subject: [PATCH 268/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index afcbeefd4ec40..ee910844fe6dd 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2008.8) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Thu, 21 May 2026 15:52:26 -0500 + linux-nvidia-bos (7.0.0-2005.5) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2005.5 -proposed tracker (LP: #2148362) From a33a55b5480876d9c3ab2c53e4d5e559f7b9bfb5 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 21 May 2026 15:55:07 -0500 Subject: [PATCH 269/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2153497 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index 445d5824e700a..572c4a12d5dcf 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2148362 d2026.04.13-1 +2153497 d2026.05.20-1 From d1a42d73fe5c1f770f9ae4ed1bfeb41c9f803a3d Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 21 May 2026 16:27:02 -0500 Subject: [PATCH 270/464] UBUNTU: [Config] nvidia-bos: update annotations Some options were not ordered as the annotations tool expected them to be. Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 37 ++++++++++++++-------------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 854c543bb2a1c..562d154cdd155 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -45,9 +45,6 @@ CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE note<'Required for Grace enable CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_TRBE_WRITE_OUT_OF_RANGE note<'Required for Grace enablement'> -CONFIG_CACHEMAINT_FOR_HOTPLUG policy<{'amd64': '-', 'arm64': 'n'}> -CONFIG_CACHEMAINT_FOR_HOTPLUG note<'Optional HiSilicon HHA cache maintenance driver; depends on GENERIC_CPU_CACHE_MAINTENANCE; not needed for NVIDIA platforms'> - CONFIG_ARM_FFA_TRANSPORT policy<{'arm64': 'y'}> CONFIG_ARM_FFA_TRANSPORT note<'LP: #2111511'> @@ -57,6 +54,9 @@ CONFIG_ARM_LFA note<'LP: #2138342'> CONFIG_ARM_SMMU_V3_IOMMUFD policy<{'arm64': 'y'}> CONFIG_ARM_SMMU_V3_IOMMUFD note<'LP: #2095028'> +CONFIG_CACHEMAINT_FOR_HOTPLUG policy<{'amd64': '-', 'arm64': 'n'}> +CONFIG_CACHEMAINT_FOR_HOTPLUG note<'Optional HiSilicon HHA cache maintenance driver; depends on GENERIC_CPU_CACHE_MAINTENANCE; not needed for NVIDIA platforms'> + CONFIG_CMA_SIZE_MBYTES policy<{'amd64': '0', 'arm64': '0'}> CONFIG_CMA_SIZE_MBYTES note<'LP: #2150898'> @@ -141,6 +141,15 @@ CONFIG_CXL_PORT note<'Required for CXL port enum CONFIG_CXL_RAS policy<{'amd64': 'y', 'arm64': 'y'}> CONFIG_CXL_RAS note<'New def_bool replacing PCIEAER_CXL; auto-enabled with ACPI_APEI_GHES+PCIEAER+CXL_BUS; CXL RAS error handling support'> +CONFIG_DEV_DAX policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX note<'Override debian.master m-'> + +CONFIG_DEV_DAX_CXL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX_CXL note<'Override debian.master m-'> + +CONFIG_DEV_DAX_KMEM policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_DEV_DAX_KMEM note<'Override debian.master m-'> + CONFIG_DRM_NOUVEAU policy<{'amd64': 'n', 'arm64': 'n'}> CONFIG_DRM_NOUVEAU note<'Disable nouveau for NVIDIA kernels'> @@ -213,9 +222,6 @@ CONFIG_NOUVEAU_PLATFORM_DRIVER note<'Disable nouveau for NVIDIA CONFIG_NR_CPUS policy<{'amd64': '8192', 'arm64': '512'}> CONFIG_NR_CPUS note<'LP: #1864198'> -CONFIG_PCIEAER_CXL policy<{'amd64': '-', 'arm64': '-'}> -CONFIG_PCIEAER_CXL note<'Removed by commit d18f1b7beadf (PCI/AER: Replace PCIEAER_CXL symbol with CXL_RAS)'> - CONFIG_NVGRACE_EGM policy<{'arm64': 'm'}> CONFIG_NVGRACE_EGM note<'LP: #2119656'> @@ -228,6 +234,12 @@ CONFIG_NVIDIA_TEGRA410_C2C_PMU note<'LP: #2139315'> CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU policy<{'arm64': 'm'}> CONFIG_NVIDIA_TEGRA410_CMEM_LATENCY_PMU note<'LP: #2139315'> +CONFIG_PCIEAER_CXL policy<{'amd64': '-', 'arm64': '-'}> +CONFIG_PCIEAER_CXL note<'Removed by commit d18f1b7beadf (PCI/AER: Replace PCIEAER_CXL symbol with CXL_RAS)'> + +CONFIG_PCI_CXL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_PCI_CXL note<'Hidden bool; auto-enabled by CXL_BUS; PCI core CXL DVSEC and HDM state save/restore support'> + CONFIG_PID_IN_CONTEXTIDR policy<{'arm64': 'y'}> CONFIG_PID_IN_CONTEXTIDR note<'Required for Grace enablement'> @@ -264,18 +276,6 @@ CONFIG_UBUNTU_ODM_DRIVERS note<'Disable all Ubuntu ODM dri CONFIG_ULTRASOC_SMB policy<{'arm64': 'n'}> CONFIG_ULTRASOC_SMB note<'Required for Grace enablement'> -CONFIG_DEV_DAX policy<{'amd64': 'y', 'arm64': 'y'}> -CONFIG_DEV_DAX note<'Override debian.master m->y; required built-in for DEV_DAX_CXL=y'> - -CONFIG_DEV_DAX_CXL policy<{'amd64': 'y', 'arm64': 'y'}> -CONFIG_DEV_DAX_CXL note<'Override debian.master m->y; CXL RAM region DAX access; depends on CXL_BUS+CXL_REGION+DEV_DAX'> - -CONFIG_DEV_DAX_KMEM policy<{'amd64': 'y', 'arm64': 'y'}> -CONFIG_DEV_DAX_KMEM note<'Override debian.master m->y; map CXL DAX devices as System-RAM'> - -CONFIG_PCI_CXL policy<{'amd64': 'y', 'arm64': 'y'}> -CONFIG_PCI_CXL note<'Hidden bool; auto-enabled by CXL_BUS; PCI core CXL DVSEC and HDM state save/restore support'> - CONFIG_VFIO_CONTAINER policy<{'amd64': 'y', 'arm64': 'n'}> CONFIG_VFIO_CONTAINER note<'LP: #2095028'> @@ -286,4 +286,5 @@ CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> # ---- Annotations without notes ---- CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_HISI_SOC_HHA policy<{'arm64': '-'}> CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> From 19aa3e21e77c6ddd6bd00e49a15803fc64733d6c Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Fri, 22 May 2026 08:38:19 -0500 Subject: [PATCH 271/464] UBUNTU: [Packaging] dkms-build: Pass --force to `dkms build` This is necessary to bypass dependencies declared by the nvidia-fs dkms.conf that are present on the system, detected by the nvidia-fs build, but not in the source directory used by dkms and so not detected by dkms. Ignore: yes Signed-off-by: Jacob Martin --- debian/scripts/dkms-build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/scripts/dkms-build b/debian/scripts/dkms-build index 8e9adc56f03de..d64946a808e9b 100755 --- a/debian/scripts/dkms-build +++ b/debian/scripts/dkms-build @@ -171,7 +171,7 @@ echo "II: dkms-build building $package" fakeroot="" [ $(id -u) -ne 0 ] && fakeroot="/usr/bin/fakeroot" rc=0 -$fakeroot /usr/sbin/dkms build --no-prepare-kernel --no-clean-kernel \ +$fakeroot /usr/sbin/dkms build --force --no-prepare-kernel --no-clean-kernel \ -k "$abi_flavour" ${ARCH:+-a $ARCH} \ --sourcetree "$dkms_dir/source" \ --dkmstree "$dkms_dir/build" \ From 76ef1609c586547c7f9493a91a1a12ad0d0609d1 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Fri, 22 May 2026 11:40:49 -0500 Subject: [PATCH 272/464] UBUNTU: [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel-versions (adhoc/d2026.05.20) BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/dkms-versions | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/dkms-versions b/debian.nvidia-bos/dkms-versions index fd8e9c633b136..ba40c69369282 100644 --- a/debian.nvidia-bos/dkms-versions +++ b/debian.nvidia-bos/dkms-versions @@ -1,2 +1,2 @@ -zfs-linux 2.4.1-1ubuntu1 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=riscv64 arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms off_series=true +zfs-linux 2.4.1-1ubuntu5 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=riscv64 arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms off_series=true v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms off_series=true From 83276807c072dc6b74d04bd9d74d0920f39d983b Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 27 May 2026 15:46:15 -0500 Subject: [PATCH 273/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2008.8 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 290 +++++++++++++++++++++++++++++++++++- 1 file changed, 285 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index ee910844fe6dd..5de21acf22d47 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,290 @@ -linux-nvidia-bos (7.0.0-2008.8) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2008.8) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2007.7 -proposed tracker (LP: #2153497) - -- Jacob Martin Thu, 21 May 2026 15:52:26 -0500 + * GRO managed-frag use-after-free leading to local privilege escalation + (LP: #2154172) + - net: gro: don't merge zcopy skbs + + * apparmor (LP: #2151747) + - SAUCE: apparmor: pass big_resp to handler + - SAUCE: apparmor: remove redundant kref_init for listener->count + - SAUCE: apparmor: fix NULL pointer dereference in unpack_pdb + + * apparmor (LP: #2151747) // CVE-2026-47337 + - SAUCE: apparmor: fix NULL pointer dereference in bind_map_addr + + * apparmor (LP: #2151747) // CVE-2026-47334 + - SAUCE: apparmor: fix sleep prone memory allocation under a spin_lock + + * apparmor (LP: #2151747) // CVE-2026-47333 + - SAUCE: apparmor: fix dfa unpacking size of the notification filter + + * apparmor (LP: #2151747) // CVE-2026-47332 + - SAUCE: apparmor: fix size check against type instead of pointer + + * apparmor: LLVM/clang build failure due to uninitialized variable in + notify.c (LP: #2148809) // CVE-2026-47330 + - SAUCE: apparmor: initialize variable used in uninitialized context + + * apparmor (LP: #2151747) // CVE-2026-47329 + - SAUCE: apparmor: fix name validation bypass on notification + + * apparmor (LP: #2151747) // CVE-2026-47327 // CVE-2026-47328 + - SAUCE: apparmor: fix glob memory leak after kstrdup + + * apparmor (LP: #2151747) // CVE-2026-47326 + - SAUCE: apparmor: fix inverted NULL check after aa_get_buffer + + * CVE-2026-46300 + - net: skbuff: preserve shared-frag marker during coalescing + - net: skbuff: propagate shared-frag marker through frag-transfer helpers + + * net/rds: reset op_nents when zerocopy page pin fails (LP: #2153962) + - net/rds: reset op_nents when zerocopy page pin fails + + * CVE-2026-46333 + - ptrace: slightly saner 'get_dumpable()' logic + + * CVE-2026-43500 + - rxrpc: Fix conn-level packet handling to unshare RESPONSE packets + - rxrpc: Fix potential UAF after skb_unshare() failure + - rxrpc: Fix rxrpc_input_call_event() to only unshare DATA packets + - rxrpc: Also unshare DATA/RESPONSE packets when paged frags are present + + * CVE-2026-43284 + - xfrm: esp: avoid in-place decrypt on shared skb frags + + * Packaging resync (LP: #1786013) + - [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel- + versions (adhoc/d2026.05.20) + + * Add CXL Type-2 device support, RAS error handling, reset, state + save/restore, and interleaving support (LP: #2143032) // CXL: Backport + Type-2, state save/restore, and reset support (LP: #2153819) + - NVIDIA: VR: SAUCE: [Config] CXL config annotations for Type-2 device and + RAS support + - NVIDIA: VR: SAUCE: [Config] Enable CXL DAX and KMEM built-in for CXL + memory access + - NVIDIA: VR: SAUCE: [Config] Add PCI_CXL annotation for CXL state + save/restore + - NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC control, lock, and range register + definitions + - NVIDIA: VR: SAUCE: cxl: Move HDM decoder and register map definitions to + include/cxl/cxl.h + - NVIDIA: VR: SAUCE: PCI: Add virtual extended cap save buffer for CXL + state + - NVIDIA: VR: SAUCE: PCI: Add cxl DVSEC state save/restore across resets + - NVIDIA: VR: SAUCE: PCI: Add HDM decoder state save/restore + - NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC reset and capability register + definitions + - NVIDIA: VR: SAUCE: PCI: Export pci_dev_save_and_disable() and + pci_dev_restore() + - NVIDIA: VR: SAUCE: cxl: Add memory offlining and cache flush helpers + - NVIDIA: VR: SAUCE: cxl: Add multi-function sibling coordination for CXL + reset + - NVIDIA: VR: SAUCE: cxl: Add CXL DVSEC reset sequence and flow + orchestration + - NVIDIA: VR: SAUCE: cxl: Add cxl_reset sysfs interface for PCI devices + - NVIDIA: VR: SAUCE: Documentation: ABI: Add CXL PCI cxl_reset sysfs + attribute + + * CXL: Backport Type-2, state save/restore, and reset support (LP: #2153819) + - cxl: support Type2 when initializing cxl_dev_state + - cxl: export internal structs for external Type2 drivers + - cxl: Move pci generic code from cxl_pci to core/cxl_pci + - cxl/pci: Remove redundant cxl_pci_find_port() call + - NVIDIA: VR: SAUCE: sfc: add cxl support + - NVIDIA: VR: SAUCE: cxl/sfc: Map cxl regs + - NVIDIA: VR: SAUCE: cxl/sfc: Initialize dpa without a mailbox + - NVIDIA: VR: SAUCE: cxl: Prepare memdev creation for type2 + - NVIDIA: VR: SAUCE: sfc: create type2 cxl memdev + - NVIDIA: VR: SAUCE: cxl: attach region to an accelerator/type2 memdev + - NVIDIA: VR: SAUCE: cxl: Avoid dax creation for accelerators + - NVIDIA: VR: SAUCE: sfc: support pio mapping based on cxl + - NVIDIA: VR: SAUCE: dax/hmem: Request cxl_acpi and cxl_pci before walking + Soft Reserved ranges + - NVIDIA: VR: SAUCE: dax/hmem: Gate Soft Reserved deferral on DEV_DAX_CXL + - NVIDIA: VR: SAUCE: cxl/region: Skip decoder reset on detach for + autodiscovered regions + - NVIDIA: VR: SAUCE: dax/cxl, hmem: Initialize hmem early and defer + dax_cxl binding + - NVIDIA: VR: SAUCE: dax: Track all dax_region allocations under a global + resource tree + - NVIDIA: VR: SAUCE: cxl/region: Add helper to check Soft Reserved + containment by CXL regions + - NVIDIA: VR: SAUCE: dax: Add deferred-work helpers for dax_hmem and + dax_cxl coordination + - NVIDIA: VR: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft + Reserved memory ranges + - NVIDIA: VR: SAUCE: dax/hmem: Reintroduce Soft Reserved ranges back into + the iomem tree + - NVIDIA: VR: SAUCE: cxl/region: Support multi-level interleaving with + smaller granularities for lower levels + - NVIDIA: SAUCE: Revert "NVIDIA: VR: SAUCE: cxl: add support for cxl + reset" + + * Installer fails internally with a RSync error due to page fault + (LP: #2150640) + - NVIDIA: SAUCE: ovl: keep err zero after successful ovl_cache_get() + + * Refresh series: Allow ATS to be always on for certain ATS-capable devices + (LP: #2150727) + - Revert "NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on" + - Revert "NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for non-CXL + NVIDIA GPUs" + - Revert "NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache + capable devices" + - NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for CXL.cache capable + devices + - NVIDIA: VR: SAUCE: PCI: Allow ATS to be always on for pre-CXL devices + - NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Allow ATS to be always on + + * Pull CPPC mailing list patches for Spark (LP: #2131705) + - ACPI: CPPC: Add cppc_get_perf() API to read performance controls + - ACPI: CPPC: Warn on missing mandatory DESIRED_PERF register + - ACPI: CPPC: Extend cppc_set_epp_perf() for FFH/SystemMemory + - cpufreq: CPPC: Update cached perf_ctrls on sysfs write + - cpufreq: cppc: Update MIN_PERF/MAX_PERF in target callbacks + - ACPI: CPPC: add APIs and sysfs interface for perf_limited + - cpufreq: CPPC: Add sysfs documentation for perf_limited + - ACPI: CPPC: Move reference performance to capabilities + - ACPI: CPPC: Fix uninitialized ref variable in cppc_get_perf_caps() + - ACPI: CPPC: Check cpc_read() return values consistently + - cpufreq: Remove max_freq_req update for pre-existing policy + - cpufreq: Add boost_freq_req QoS request + - cpufreq: Allocate QoS freq_req objects with policy + - cpufreq/amd-pstate: Cache the max frequency in cpudata + - NVIDIA: SAUCE: cpufreq: Extract cpufreq_policy_init_qos() function + - NVIDIA: SAUCE: cpufreq: Set default policy->min/max values for all + drivers + - NVIDIA: SAUCE: cpufreq: Remove driver default policy->min/max init + - NVIDIA: SAUCE: cpufreq: Use policy->min/max init as QoS request + - NVIDIA: SAUCE: cpufreq: CPPC: add autonomous mode boot parameter support + + * Backport Vera PMU support (LP: #2149756) + - Revert "NVIDIA: VR: SAUCE: perf vendor events arm64: Add Tegra410 + Olympus PMU events" + - Revert "NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 C2C PMU" + - Revert "NVIDIA: VR: SAUCE: perf: add NVIDIA Tegra410 CPU Memory Latency + PMU" + - Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT + PMU" + - Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 PCIE + PMU" + - Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: Add arm_cspmu_acpi_dev_get" + - Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU" + - Revert "NVIDIA: VR: SAUCE: perf/arm_cspmu: nvidia: Rename doc to + Tegra241" + - perf/arm_cspmu: nvidia: Rename doc to Tegra241 + - perf/arm_cspmu: nvidia: Add Tegra410 UCF PMU + - perf/arm_cspmu: Add arm_cspmu_acpi_dev_get + - perf/arm_cspmu: nvidia: Add Tegra410 PCIE PMU + - perf/arm_cspmu: nvidia: Add Tegra410 PCIE-TGT PMU + - perf: add NVIDIA Tegra410 CPU Memory Latency PMU + - perf: add NVIDIA Tegra410 C2C PMU + - perf vendor events arm64: Add Tegra410 Olympus PMU events + - NVIDIA: VR: SAUCE: perf/arm_pmu: Skip PMCCNTR_EL0 on NVIDIA Olympus + + * Backport lan743x driver patches (LP: #2152064) + - net: microchip: lan743x: add ethtool nway_reset support + - net: lan743x: fix SGMII detection on PCI1xxxx B0+ during warm reset + - net: lan743x: rename chip_rev to fpga_rev + + * Backport SMT-aware asymmetric CPU capacity idle selection (LP: #2150671) + - NVIDIA: VR: SAUCE: sched/fair: Attach sched_domain_shared to + sd_asym_cpucapacity + - NVIDIA: VR: SAUCE: sched/fair: Prefer fully-idle SMT cores in asym- + capacity idle selection + - NVIDIA: VR: SAUCE: sched/fair: Reject misfit pulls onto busy SMT + siblings on asym-capacity + - NVIDIA: VR: SAUCE: sched/fair: Add SIS_UTIL support to + select_idle_capacity() + + * Forward-port of the full Arm Live Firmware Activation (LFA) v2 series + (LP: #2150652) + - Revert "NVIDIA: VR: SAUCE: firmware: smccc: register as platform driver" + - Revert "NVIDIA: VR: SAUCE: firmware: smccc: add timeout, touch wdt" + - Revert "NVIDIA: VR: SAUCE: firmware: smccc: add support for Live + Firmware Activation (LFA)" + - NVIDIA: VR: SAUCE: dt-bindings: arm: Add Live Firmware Activation + binding + - NVIDIA: VR: SAUCE: firmware: smccc: Add support for Live Firmware + Activation (LFA) + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Move image rescanning + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Add timeout and trigger + watchdog + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Register ACPI notification + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Add auto_activate sysfs file + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Register DT interrupt + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: introduce SMC access lock + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: handle LFA_BUSY in PRIME and + ACTIVATE + - NVIDIA: VR: SAUCE: firmware: smccc: lfa: Emit a uevent on inventory + updates + + * Introduce a sharded cache affinity scope (LP: #2150467) + - workqueue: fix parse_affn_scope() prefix matching bug + - workqueue: fix typo in WQ_AFFN_SMT comment + - workqueue: add WQ_AFFN_CACHE_SHARD affinity scope + - workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope + - tools/workqueue: add CACHE_SHARD support to wq_dump.py + - workqueue: add test_workqueue benchmark module + - docs: workqueue: document WQ_AFFN_CACHE_SHARD affinity scope + - workqueue: avoid unguarded 64-bit division + - workqueue: validate cpumask_first() result in + llc_populate_cpu_shard_id() + - [Config] nvidia: Defaults for CONFIG_TEST_WORKQUEUE + + * UBUNTU: [Config] nvidia: Disable default CMA reservation (LP: #2150898) + - [Config] nvidia: Disable default CMA reservation + + * Backport Use device ID range for DGX Spark iGPU (LP: #2150487) + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Use device ID range for DGX Spark iGPU + iommu quirk + + * Backport NVIDIA: SAUCE: iommu/arm-smmu-v3: Use identity domain for ASPEED + BMC devices (LP: #2150470) + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Use identity domain for ASPEED BMC + devices + + * Update GDS/NVMe SAUCE for v6.17 (LP: #2134960) // [linux-nvidia-7.0]: + Forward-port GDS/NVFS content (LP: #2150289) + - NVIDIA: SAUCE: Patch NVMe/NVMeoF driver to support GDS on Linux 7.0 + Kernel + + * Backport Set LED_HW_PLUGGABLE for NPEM and fix class init ordering issue + of CXL/fwctl (LP: #2149918) + - PCI/NPEM: Set LED_HW_PLUGGABLE for hotplug-capable ports + - fwctl: Fix class init ordering to avoid NULL pointer dereference on + device removal + + * gpio: tegra186: Simplify GPIO line name prefix and support multi-socket + devices (LP: #2148664) + - gpio: tegra186: Simplify GPIO line name prefix handling + - gpio: tegra186: Support multi-socket devices + - Revert "NVIDIA: SAUCE: serial: 8250_mtk: Add ACPI support" + - NVIDIA: SAUCE: MEDIATEK: serial: 8250_mtk: Add ACPI support + + * fix r8169 vs r8127 contention for Spark (LP: #2144345) + - NVIDIA: SAUCE: r8169: remove PCI IDs claimed by r8127 driver + + * Backport of the vfio/nvgrace-gpu Blackwell-Next GPU readiness check (v3) + from LKML to 26.04_linux-nvidia. (LP: #2148701) + - NVIDIA: SAUCE: vfio/nvgrace-gpu: Add Blackwell-Next GPU readiness check + via CXL DVSEC + + [ Ubuntu: 7.0.0-15.15 ] + + * resolute/linux: 7.0.0-15.15 -proposed tracker (LP: #2148866) + * Qualcomm X1E: Speaker overdrive causes hardware protection shutdown + (LP: #2149808) + - SAUCE: ASoC: qcom: x1e80100: limit speaker volumes + * intel-ipu7 / intel-ipu7-isys modules are shipped unsigned in latest + Resolute kernels, breaking Secure Boot systems (LP: #2148718) + - [packaging] add intel-ipu7 to signature inclusion list + + -- Jacob Martin Wed, 27 May 2026 15:46:15 -0500 linux-nvidia-bos (7.0.0-2005.5) resolute; urgency=medium From d8331e5ffdbd6621a9cedcd8ea921bc86a086512 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Mon, 18 May 2026 16:05:20 +0800 Subject: [PATCH 274/464] NVIDIA: VR: SAUCE: tegra: bpmp: Move channel, resource init to helper BugLink: https://bugs.launchpad.net/bugs/2147212 Refactor the BPMP driver by moving channel initialization and Device Tree resource parsing into separate helper functions. This prepares the driver for ACPI support, where these helpers will be skipped because channel initialization is handled by ACPI AML methods on ACPI-based systems. Signed-off-by: Aniruddha Rao (backported from V4 internal mail <20260423140823.2848045-2-anrao@nvidia.com>) [kobak: Preserve threaded channel count/semaphore initialization after the helper split and align rx_channel allocation continuation.] Signed-off-by: Koba Ko Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/tegra/bpmp.c | 96 +++++++++++++++++++++-------------- 1 file changed, 58 insertions(+), 38 deletions(-) diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c index e74bba7ccc443..37a51f7540ad0 100644 --- a/drivers/firmware/tegra/bpmp.c +++ b/drivers/firmware/tegra/bpmp.c @@ -699,19 +699,9 @@ void tegra_bpmp_handle_rx(struct tegra_bpmp *bpmp) spin_unlock(&bpmp->lock); } -static int tegra_bpmp_probe(struct platform_device *pdev) +static int tegra_bpmp_init_channels(struct tegra_bpmp *bpmp) { - struct tegra_bpmp *bpmp; - char tag[TAG_SZ]; size_t size; - int err; - - bpmp = devm_kzalloc(&pdev->dev, sizeof(*bpmp), GFP_KERNEL); - if (!bpmp) - return -ENOMEM; - - bpmp->soc = of_device_get_match_data(&pdev->dev); - bpmp->dev = &pdev->dev; INIT_LIST_HEAD(&bpmp->mrqs); spin_lock_init(&bpmp->lock); @@ -721,37 +711,85 @@ static int tegra_bpmp_probe(struct platform_device *pdev) size = BITS_TO_LONGS(bpmp->threaded.count) * sizeof(long); - bpmp->threaded.allocated = devm_kzalloc(&pdev->dev, size, GFP_KERNEL); + bpmp->threaded.allocated = devm_kzalloc(bpmp->dev, size, GFP_KERNEL); if (!bpmp->threaded.allocated) return -ENOMEM; - bpmp->threaded.busy = devm_kzalloc(&pdev->dev, size, GFP_KERNEL); + bpmp->threaded.busy = devm_kzalloc(bpmp->dev, size, GFP_KERNEL); if (!bpmp->threaded.busy) return -ENOMEM; spin_lock_init(&bpmp->atomic_tx_lock); - bpmp->tx_channel = devm_kzalloc(&pdev->dev, sizeof(*bpmp->tx_channel), + bpmp->tx_channel = devm_kzalloc(bpmp->dev, sizeof(*bpmp->tx_channel), GFP_KERNEL); if (!bpmp->tx_channel) return -ENOMEM; - bpmp->rx_channel = devm_kzalloc(&pdev->dev, sizeof(*bpmp->rx_channel), - GFP_KERNEL); + bpmp->rx_channel = devm_kzalloc(bpmp->dev, sizeof(*bpmp->rx_channel), + GFP_KERNEL); if (!bpmp->rx_channel) return -ENOMEM; - bpmp->threaded_channels = devm_kcalloc(&pdev->dev, bpmp->threaded.count, + bpmp->threaded_channels = devm_kcalloc(bpmp->dev, bpmp->threaded.count, sizeof(*bpmp->threaded_channels), GFP_KERNEL); if (!bpmp->threaded_channels) return -ENOMEM; - platform_set_drvdata(pdev, bpmp); + return 0; +} + +static int tegra_bpmp_init_resources(struct tegra_bpmp *bpmp) +{ + int err; - err = bpmp->soc->ops->init(bpmp); + err = of_platform_default_populate(bpmp->dev->of_node, NULL, bpmp->dev); if (err < 0) return err; + if (of_property_present(bpmp->dev->of_node, "#clock-cells")) { + err = tegra_bpmp_init_clocks(bpmp); + if (err < 0) + return err; + } + + if (of_property_present(bpmp->dev->of_node, "#reset-cells")) { + err = tegra_bpmp_init_resets(bpmp); + if (err < 0) + return err; + } + + if (of_property_present(bpmp->dev->of_node, "#power-domain-cells")) + err = tegra_bpmp_init_powergates(bpmp); + + return err; +} + +static int tegra_bpmp_probe(struct platform_device *pdev) +{ + struct tegra_bpmp *bpmp; + char tag[TAG_SZ]; + int err; + + bpmp = devm_kzalloc(&pdev->dev, sizeof(*bpmp), GFP_KERNEL); + if (!bpmp) + return -ENOMEM; + + bpmp->soc = device_get_match_data(&pdev->dev); + bpmp->dev = &pdev->dev; + + err = tegra_bpmp_init_channels(bpmp); + if (err < 0) + return err; + + platform_set_drvdata(pdev, bpmp); + + if (bpmp->soc->ops->init) { + err = bpmp->soc->ops->init(bpmp); + if (err < 0) + return err; + } + err = tegra_bpmp_request_mrq(bpmp, MRQ_PING, tegra_bpmp_mrq_handle_ping, bpmp); if (err < 0) @@ -771,28 +809,10 @@ static int tegra_bpmp_probe(struct platform_device *pdev) dev_info(&pdev->dev, "firmware: %.*s\n", (int)sizeof(tag), tag); - err = of_platform_default_populate(pdev->dev.of_node, NULL, &pdev->dev); + err = tegra_bpmp_init_resources(bpmp); if (err < 0) goto free_mrq; - if (of_property_present(pdev->dev.of_node, "#clock-cells")) { - err = tegra_bpmp_init_clocks(bpmp); - if (err < 0) - goto free_mrq; - } - - if (of_property_present(pdev->dev.of_node, "#reset-cells")) { - err = tegra_bpmp_init_resets(bpmp); - if (err < 0) - goto free_mrq; - } - - if (of_property_present(pdev->dev.of_node, "#power-domain-cells")) { - err = tegra_bpmp_init_powergates(bpmp); - if (err < 0) - goto free_mrq; - } - err = tegra_bpmp_init_debugfs(bpmp); if (err < 0) dev_err(&pdev->dev, "debugfs initialization failed: %d\n", err); From 32922da409c36ec6a7704937245fbde69644b5b2 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Mon, 18 May 2026 16:05:21 +0800 Subject: [PATCH 275/464] NVIDIA: VR: SAUCE: tegra: bpmp: Add ACPI support BugLink: https://bugs.launchpad.net/bugs/2147212 This patch adds required changes in the Tegra BPMP driver to make it compatible with ACPI based platforms. On ACPI systems, IPC is handled through the AML method instead of the core kernel framework using Mailboxes and IVC. Bypass clock, reset and powergate init calls as these are not controlled by the Linux drivers on ACPI based systems. Signed-off-by: Aniruddha Rao (backported from V4 internal mail <20260423140823.2848045-3-anrao@nvidia.com>) [kobak: Add !ACPI_HANDLE(bpmp->dev) NULL guard around bpmp->soc->ops->init because ACPI match driver_data=0 makes bpmp->soc NULL; make BPMP debugfs directory per-device on ACPI systems to avoid duplicate /sys/kernel/debug/bpmp collision on dual NVDA3001 instances; remove unused i; heap allocate the ACPI BMRQ package to avoid the frame-size warning; reject short BMRQ replies before copying response data; add CONFIG_ACPI stub for the ACPI helper; restore the public irqs_disabled() guard before ACPI/DT transport selection.] Signed-off-by: Koba Ko Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- drivers/firmware/tegra/bpmp-debugfs.c | 19 ++- drivers/firmware/tegra/bpmp-private.h | 8 ++ drivers/firmware/tegra/bpmp.c | 177 +++++++++++++++++++++++--- 3 files changed, 183 insertions(+), 21 deletions(-) diff --git a/drivers/firmware/tegra/bpmp-debugfs.c b/drivers/firmware/tegra/bpmp-debugfs.c index 4221fed70ad48..9898aa9396c56 100644 --- a/drivers/firmware/tegra/bpmp-debugfs.c +++ b/drivers/firmware/tegra/bpmp-debugfs.c @@ -2,6 +2,7 @@ /* * Copyright (c) 2017, NVIDIA CORPORATION. All rights reserved. */ +#include #include #include #include @@ -771,6 +772,8 @@ static int bpmp_populate_debugfs_shmem(struct tegra_bpmp *bpmp) int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp) { + const char *root_name = "bpmp"; + char *acpi_root_name = NULL; struct dentry *root; bool inband; int err; @@ -780,13 +783,23 @@ int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp) if (!inband && !tegra_bpmp_mrq_is_supported(bpmp, MRQ_DEBUGFS)) return 0; - root = debugfs_create_dir("bpmp", NULL); + if (ACPI_HANDLE(bpmp->dev)) { + acpi_root_name = kasprintf(GFP_KERNEL, "bpmp-%s", + dev_name(bpmp->dev)); + if (!acpi_root_name) + return -ENOMEM; + + root_name = acpi_root_name; + } + + root = debugfs_create_dir(root_name, NULL); + kfree(acpi_root_name); if (IS_ERR(root)) - return -ENOMEM; + return PTR_ERR(root); bpmp->debugfs_mirror = debugfs_create_dir("debug", root); if (IS_ERR(bpmp->debugfs_mirror)) { - err = -ENOMEM; + err = PTR_ERR(bpmp->debugfs_mirror); goto out; } diff --git a/drivers/firmware/tegra/bpmp-private.h b/drivers/firmware/tegra/bpmp-private.h index 07c3d46abb874..6cb5b8b544561 100644 --- a/drivers/firmware/tegra/bpmp-private.h +++ b/drivers/firmware/tegra/bpmp-private.h @@ -26,4 +26,12 @@ struct tegra_bpmp_ops { extern const struct tegra_bpmp_ops tegra186_bpmp_ops; extern const struct tegra_bpmp_ops tegra210_bpmp_ops; +#define TEGRA_BPMP_ACPI_BMRQ_DATA_SZ 3960U + +struct tegra_bpmp_acpi_message { + u64 status; + u8 *data_ptr; + u8 data[TEGRA_BPMP_ACPI_BMRQ_DATA_SZ]; +}; + #endif diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c index 37a51f7540ad0..77b1fcae3d31b 100644 --- a/drivers/firmware/tegra/bpmp.c +++ b/drivers/firmware/tegra/bpmp.c @@ -3,6 +3,7 @@ * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. */ +#include #include #include #include @@ -309,12 +310,117 @@ static ssize_t tegra_bpmp_channel_write(struct tegra_bpmp_channel *channel, static int __maybe_unused tegra_bpmp_resume(struct device *dev); +#if IS_ENABLED(CONFIG_ACPI) +static int tegra_bpmp_transfer_acpi(struct tegra_bpmp *bpmp, + struct tegra_bpmp_message *msg) +{ + acpi_status status; + union acpi_object params[2]; + struct acpi_object_list param_list; + struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL }; + union acpi_object *obj; + struct tegra_bpmp_acpi_message *pkg; + struct acpi_buffer format = { sizeof("NB"), "NB" }; + struct acpi_buffer extract; + size_t rbuf_len, rdata_len; + + if (!tegra_bpmp_message_valid(msg)) + return -EINVAL; + + params[0].type = ACPI_TYPE_INTEGER; + params[0].integer.value = msg->mrq; + + params[1].type = ACPI_TYPE_BUFFER; + params[1].buffer.length = msg->tx.size; + params[1].buffer.pointer = (u8 *)msg->tx.data; + + param_list.count = 2; + param_list.pointer = params; + + status = acpi_evaluate_object(ACPI_HANDLE(bpmp->dev), "BMRQ", + ¶m_list, &output); + if (ACPI_FAILURE(status)) { + acpi_evaluation_failure_warn(ACPI_HANDLE(bpmp->dev), "BMRQ", + status); + return -ENODEV; + } + + obj = (union acpi_object *)output.pointer; + /* Validate returned type */ + if (!obj || obj->type != ACPI_TYPE_PACKAGE) { + dev_err(bpmp->dev, "Invalid BMRQ data\n"); + kfree(output.pointer); + return -ENODATA; + } + + if (obj->package.count < 2 || + obj->package.elements[1].type != ACPI_TYPE_BUFFER) { + dev_err(bpmp->dev, "Invalid BMRQ data\n"); + kfree(output.pointer); + return -ENODATA; + } + + rdata_len = obj->package.elements[1].buffer.length; + rbuf_len = sizeof(u64) + sizeof(u8 *) + rdata_len; + if (rbuf_len > sizeof(*pkg)) { + dev_err(bpmp->dev, "BMRQ: reply buffer too large (%zu)\n", rbuf_len); + kfree(output.pointer); + return -EINVAL; + } + + pkg = kzalloc(sizeof(*pkg), GFP_KERNEL); + if (!pkg) { + kfree(output.pointer); + return -ENOMEM; + } + + extract.length = rbuf_len; + extract.pointer = pkg; + + status = acpi_extract_package(obj, &format, &extract); + if (ACPI_FAILURE(status)) { + dev_err(bpmp->dev, "BMRQ: failed to parse package (%s)\n", + acpi_format_exception(status)); + kfree(pkg); + kfree(output.pointer); + return -EINVAL; + } + + msg->rx.ret = (int)pkg->status; + if (msg->rx.data && msg->rx.size) { + if (rdata_len < msg->rx.size) { + dev_err(bpmp->dev, "BMRQ: reply buffer too small (%zu)\n", + rdata_len); + kfree(pkg); + kfree(output.pointer); + return -EMSGSIZE; + } + + memcpy(msg->rx.data, pkg->data, msg->rx.size); + } + + /* Free memory allocated by ACPI core */ + kfree(pkg); + kfree(output.pointer); + return 0; +} +#else +static int tegra_bpmp_transfer_acpi(struct tegra_bpmp *bpmp, + struct tegra_bpmp_message *msg) +{ + return -EOPNOTSUPP; +} +#endif + int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp, struct tegra_bpmp_message *msg) { struct tegra_bpmp_channel *channel; int err; + if (WARN_ON(ACPI_HANDLE(bpmp->dev))) + return -EOPNOTSUPP; + if (WARN_ON(!irqs_disabled())) return -EPERM; @@ -355,8 +461,8 @@ int tegra_bpmp_transfer_atomic(struct tegra_bpmp *bpmp, } EXPORT_SYMBOL_GPL(tegra_bpmp_transfer_atomic); -int tegra_bpmp_transfer(struct tegra_bpmp *bpmp, - struct tegra_bpmp_message *msg) +static int tegra_bpmp_transfer_channel(struct tegra_bpmp *bpmp, + struct tegra_bpmp_message *msg) { struct tegra_bpmp_channel *channel; unsigned long timeout; @@ -394,6 +500,18 @@ int tegra_bpmp_transfer(struct tegra_bpmp *bpmp, return tegra_bpmp_channel_read(channel, msg->rx.data, msg->rx.size, &msg->rx.ret); } + +int tegra_bpmp_transfer(struct tegra_bpmp *bpmp, + struct tegra_bpmp_message *msg) +{ + if (WARN_ON(irqs_disabled())) + return -EPERM; + + if (ACPI_HANDLE(bpmp->dev)) + return tegra_bpmp_transfer_acpi(bpmp, msg); + else + return tegra_bpmp_transfer_channel(bpmp, msg); +} EXPORT_SYMBOL_GPL(tegra_bpmp_transfer); static struct tegra_bpmp_mrq *tegra_bpmp_find_mrq(struct tegra_bpmp *bpmp, @@ -472,6 +590,9 @@ int tegra_bpmp_request_mrq(struct tegra_bpmp *bpmp, unsigned int mrq, struct tegra_bpmp_mrq *entry; unsigned long flags; + if (ACPI_HANDLE(bpmp->dev)) + return -EOPNOTSUPP; + if (!handler) return -EINVAL; @@ -497,6 +618,9 @@ void tegra_bpmp_free_mrq(struct tegra_bpmp *bpmp, unsigned int mrq, void *data) struct tegra_bpmp_mrq *entry; unsigned long flags; + if (ACPI_HANDLE(bpmp->dev)) + return; + spin_lock_irqsave(&bpmp->lock, flags); entry = tegra_bpmp_find_mrq(bpmp, mrq); @@ -572,11 +696,17 @@ static int tegra_bpmp_ping(struct tegra_bpmp *bpmp) msg.rx.data = &response; msg.rx.size = sizeof(response); - local_irq_save(flags); - start = ktime_get(); - err = tegra_bpmp_transfer_atomic(bpmp, &msg); - end = ktime_get(); - local_irq_restore(flags); + if (ACPI_HANDLE(bpmp->dev)) { + start = ktime_get(); + err = tegra_bpmp_transfer_acpi(bpmp, &msg); + end = ktime_get(); + } else { + local_irq_save(flags); + start = ktime_get(); + err = tegra_bpmp_transfer_atomic(bpmp, &msg); + end = ktime_get(); + local_irq_restore(flags); + } if (!err) dev_dbg(bpmp->dev, @@ -614,10 +744,13 @@ static int tegra_bpmp_get_firmware_tag_old(struct tegra_bpmp *bpmp, char *tag, msg.tx.data = &request; msg.tx.size = sizeof(request); - local_irq_save(flags); - err = tegra_bpmp_transfer_atomic(bpmp, &msg); - local_irq_restore(flags); - + if (ACPI_HANDLE(bpmp->dev)) { + err = tegra_bpmp_transfer_acpi(bpmp, &msg); + } else { + local_irq_save(flags); + err = tegra_bpmp_transfer_atomic(bpmp, &msg); + local_irq_restore(flags); + } if (err == 0) memcpy(tag, virt, TAG_SZ); @@ -703,6 +836,9 @@ static int tegra_bpmp_init_channels(struct tegra_bpmp *bpmp) { size_t size; + if (ACPI_HANDLE(bpmp->dev)) + return 0; + INIT_LIST_HEAD(&bpmp->mrqs); spin_lock_init(&bpmp->lock); @@ -743,6 +879,9 @@ static int tegra_bpmp_init_resources(struct tegra_bpmp *bpmp) { int err; + if (!bpmp->dev->of_node) + return 0; + err = of_platform_default_populate(bpmp->dev->of_node, NULL, bpmp->dev); if (err < 0) return err; @@ -784,16 +923,18 @@ static int tegra_bpmp_probe(struct platform_device *pdev) platform_set_drvdata(pdev, bpmp); - if (bpmp->soc->ops->init) { + if (!ACPI_HANDLE(bpmp->dev) && bpmp->soc->ops->init) { err = bpmp->soc->ops->init(bpmp); if (err < 0) return err; } - err = tegra_bpmp_request_mrq(bpmp, MRQ_PING, - tegra_bpmp_mrq_handle_ping, bpmp); - if (err < 0) - goto deinit; + if (!ACPI_HANDLE(bpmp->dev)) { + err = tegra_bpmp_request_mrq(bpmp, MRQ_PING, + tegra_bpmp_mrq_handle_ping, bpmp); + if (err < 0) + goto deinit; + } err = tegra_bpmp_ping(bpmp); if (err < 0) { @@ -822,7 +963,7 @@ static int tegra_bpmp_probe(struct platform_device *pdev) free_mrq: tegra_bpmp_free_mrq(bpmp, MRQ_PING, bpmp); deinit: - if (bpmp->soc->ops->deinit) + if (bpmp->soc && bpmp->soc->ops->deinit) bpmp->soc->ops->deinit(bpmp); return err; @@ -843,7 +984,7 @@ static int __maybe_unused tegra_bpmp_resume(struct device *dev) bpmp->suspended = false; - if (bpmp->soc->ops->resume) + if (bpmp->soc && bpmp->soc->ops->resume) return bpmp->soc->ops->resume(bpmp); else return 0; From dc8bf049ea021e1d1bcd87b9ebcb42b24ded06f5 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Mon, 18 May 2026 16:05:21 +0800 Subject: [PATCH 276/464] NVIDIA: VR: SAUCE: tegra: bpmp: Add sysfs for memory bandwidth QoS BugLink: https://bugs.launchpad.net/bugs/2147212 Tegra410 exposes memory bandwidth QoS for PCIe and GPU UPHY traffic on the path to DRAM. Each bandwidth group can cap PCIe read, PCIe write, or combined GPU UPHY read and write traffic, with target limits. The memory bandwidth QoS is not exposed as ordinary host MMIO and cannot be controlled from the kernel. The bandwidth limits can be programmed by sending the corresponding requests (MBWT MRQ) to the BPMP. On Tegra410, an ACPI-based platform, Linux BPMP driver does not use the device-tree mailbox path for communicating with the BPMP firmware. As a result, there is no existing client driver or interface that can be used to send the memory bandwidth requests to the BPMP. This patch exposes a sysfs directory mbwt_control on the tegra-bpmp platform device with pcie_instance_id, vc_type, and bandwidth. Writing bandwidth issues an MBWT_SET for the selected group (pcie_instance_id) and traffic class (vc_type). A read issues MBWT_GET and returns the bandwidth value reported by firmware. These attributes are exposed only if MBWT QUERY probe reports both MBWT_SET and MBWT_GET commands as supported. ABI documented in Documentation/ABI/testing/sys-platform-tegra-bpmp Signed-off-by: Aniruddha Rao (backported from V4 internal mail <20260423140823.2848045-4-anrao@nvidia.com>) [kobak: Keep functional MRQ_SOCHUB_MBWT ABI definitions and sysfs interface from V4; condense verbose per-field ABI comments while preserving enum/struct layout and Documentation/ABI coverage; use refcounted kobject allocation for mbwt_control; validate pcie_instance_id/vc_type before staging; return only the bandwidth value from bandwidth reads; report BPMP SET rejections to userspace; fail BPMP probe on mbwt_control sysfs registration failure so 7.0-bos does not silently boot without the requested MBWT interface.] Signed-off-by: Koba Ko Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- .../ABI/testing/sys-platform-tegra-bpmp | 49 +++ drivers/firmware/tegra/Makefile | 1 + drivers/firmware/tegra/bpmp-tegra-sysfs.c | 345 ++++++++++++++++++ drivers/firmware/tegra/bpmp.c | 14 + include/soc/tegra/bpmp-abi.h | 64 +++- include/soc/tegra/bpmp.h | 7 + 6 files changed, 479 insertions(+), 1 deletion(-) create mode 100644 Documentation/ABI/testing/sys-platform-tegra-bpmp create mode 100644 drivers/firmware/tegra/bpmp-tegra-sysfs.c diff --git a/Documentation/ABI/testing/sys-platform-tegra-bpmp b/Documentation/ABI/testing/sys-platform-tegra-bpmp new file mode 100644 index 0000000000000..53cee3ff1951e --- /dev/null +++ b/Documentation/ABI/testing/sys-platform-tegra-bpmp @@ -0,0 +1,49 @@ +What: /sys/bus/platform/devices//mbwt_control/pcie_instance_id +What: /sys/bus/platform/devices//mbwt_control/vc_type +What: /sys/bus/platform/devices//mbwt_control/bandwidth +Date: March 2026 +KernelVersion: 6.18 +Contact: Aniruddha TVS Rao +Description: + On ACPI-based Tegra systems the BPMP driver does not use the + device-tree mailbox path; firmware interaction is via AML. This sysfs + interface is a userspace tuning knob for memory bandwidth throttler + (MBWT) settings. + + On Tegra410, the PCIe bandwidth control path exposes QoS that caps + aggregate bandwidth for PCIe and for GPU traffic over UPHY. Each PCIe + bandwidth group has a single shared cap for all traffic in that group. + A group may contain only PCIe devices, only a GPU on UPHY, or PCIe and + GPU together in a bifurcated topology. + + Following attributes appear under a kobject named mbwt_control on the + tegra-bpmp platform device (the same struct device as the driver + binds to), only when that device has an ACPI companion and BPMP + firmware reports support for both MBWT_GET and MBWT_SET via the MBWT + QUERY probe. + + pcie_instance_id (RW): + PCIe bandwidth group index: 0 = pcie0, 1 = pcie1, ..., 5 = pcie5. + + vc_type (RW): + Traffic type to cap for that group: + 0 = PCIe read + 1 = PCIe write + 2 = GPU UPHY read plus write (combined) + + bandwidth (RW): + Target bandwidth cap in GB/s for the pcie_instance_id and vc_type + currently stored in the other two attributes. Intended range is + 1-110 GB/s; firmware validates the request (via MRQ MBWT_SET). + + When read, issues a MBWT_GET for that same pcie_instance_id and vc_type + and returns the bandwidth value in GB/s returned by firmware. + + Example: + echo 0 > .../mbwt_control/pcie_instance_id + echo 1 > .../mbwt_control/vc_type + echo 100 > .../mbwt_control/bandwidth + cat .../mbwt_control/bandwidth + +Users: Customer tuning of PCIe and GPU UPHY bandwidth caps on + ACPI-based Tegra410 systems. diff --git a/drivers/firmware/tegra/Makefile b/drivers/firmware/tegra/Makefile index 41e2e4dc31d63..6c577e59b3ee7 100644 --- a/drivers/firmware/tegra/Makefile +++ b/drivers/firmware/tegra/Makefile @@ -7,4 +7,5 @@ tegra-bpmp-$(CONFIG_ARCH_TEGRA_234_SOC) += bpmp-tegra186.o tegra-bpmp-$(CONFIG_ARCH_TEGRA_264_SOC) += bpmp-tegra186.o tegra-bpmp-$(CONFIG_DEBUG_FS) += bpmp-debugfs.o obj-$(CONFIG_TEGRA_BPMP) += tegra-bpmp.o +obj-$(CONFIG_TEGRA_BPMP) += bpmp-tegra-sysfs.o obj-$(CONFIG_TEGRA_IVC) += ivc.o diff --git a/drivers/firmware/tegra/bpmp-tegra-sysfs.c b/drivers/firmware/tegra/bpmp-tegra-sysfs.c new file mode 100644 index 0000000000000..065d660d5cc09 --- /dev/null +++ b/drivers/firmware/tegra/bpmp-tegra-sysfs.c @@ -0,0 +1,345 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2026, NVIDIA CORPORATION. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "bpmp-private.h" + +#define TEGRA_BPMP_MBWT_MAX_PCIE_INSTANCE 5U +#define TEGRA_BPMP_MBWT_MAX_VC_TYPE 2U + +struct tegra_bpmp_mbwt_sysfs { + struct kobject kobj; + /* Serializes pcie_instance_id and vc_type stores and bandwidth I/O. */ + struct mutex lock; + struct tegra_bpmp *bpmp; + unsigned int pcie_instance_id; + unsigned int vc_type; +}; + +#define to_mbwt_sysfs(k) container_of((k), struct tegra_bpmp_mbwt_sysfs, kobj) + +static void tegra_bpmp_mbwt_kobj_release(struct kobject *kobj) +{ + kfree(to_mbwt_sysfs(kobj)); +} + +static const struct kobj_type tegra_bpmp_mbwt_ktype = { + .release = tegra_bpmp_mbwt_kobj_release, + .sysfs_ops = &kobj_sysfs_ops, +}; + +/** + * tegra_sochub_mbwt_query_abi() - Ask BPMP whether an MBWT sub-command is supported. + * @bpmp: BPMP handle + * @cmd_code: Sub-command to probe (e.g. CMD_SOCHUB_MBWT_SET_BW) + * + * Returns 0 if the firmware reports the sub-command is supported (MRQ error 0). + * Returns a negative errno if the transfer fails, or %-EOPNOTSUPP if the + * firmware reports the sub-command is not supported. + */ +static int tegra_sochub_mbwt_query_abi(struct tegra_bpmp *bpmp, + unsigned int cmd_code) +{ + struct mrq_sochub_mbwt_request request; + struct tegra_bpmp_message msg; + int err; + + memset(&request, 0, sizeof(request)); + request.cmd = CMD_SOCHUB_MBWT_QUERY_ABI; + request.query_abi.cmd_code = cmd_code; + + memset(&msg, 0, sizeof(msg)); + msg.mrq = MRQ_SOCHUB_MBWT; + msg.tx.data = &request; + msg.tx.size = sizeof(request); + + err = tegra_bpmp_transfer(bpmp, &msg); + if (err) + return err; + + if (msg.rx.ret) + return -EOPNOTSUPP; + + return 0; +} + +static int tegra_sochub_set_mbwt(struct tegra_bpmp *bpmp, + unsigned int instance, + unsigned int vc_type, + unsigned int bandwidth) +{ + struct mrq_sochub_mbwt_request request; + struct tegra_bpmp_message msg; + int err; + + memset(&request, 0, sizeof(request)); + request.cmd = CMD_SOCHUB_MBWT_SET_BW; + request.set_bw.instance = instance; + request.set_bw.vc_type = vc_type; + request.set_bw.bw = bandwidth; + + memset(&msg, 0, sizeof(msg)); + msg.mrq = MRQ_SOCHUB_MBWT; + msg.tx.data = &request; + msg.tx.size = sizeof(request); + + err = tegra_bpmp_transfer(bpmp, &msg); + if (err) { + dev_err(bpmp->dev, + "Failed setting up the SocHub MBWT with error %d\n", + err); + return err; + } + + if (msg.rx.ret < 0) + return -EINVAL; + + return 0; +} + +static int tegra_sochub_get_mbwt(struct tegra_bpmp *bpmp, + unsigned int instance, + unsigned int vc_type, + unsigned int *bandwidth_out) +{ + struct mrq_sochub_mbwt_request request; + struct mrq_sochub_mbwt_response response; + struct cmd_sochub_mbwt_get_bw_resp mbwt; + struct tegra_bpmp_message msg; + int err; + + memset(&request, 0, sizeof(request)); + request.cmd = CMD_SOCHUB_MBWT_GET_BW; + request.get_bw.instance = instance; + request.get_bw.vc_type = vc_type; + + memset(&response, 0, sizeof(response)); + + memset(&msg, 0, sizeof(msg)); + msg.mrq = MRQ_SOCHUB_MBWT; + msg.tx.data = &request; + msg.tx.size = sizeof(request); + msg.rx.data = &response; + msg.rx.size = sizeof(response); + + err = tegra_bpmp_transfer(bpmp, &msg); + if (err) { + dev_err(bpmp->dev, + "Failed reading the SocHub MBWT with error %d\n", + err); + return err; + } + if (msg.rx.ret < 0) + return -EINVAL; + + memcpy(&mbwt, &response.get_bw, sizeof(response.get_bw)); + if (bandwidth_out) + *bandwidth_out = mbwt.bw; + + return 0; +} + +static ssize_t pcie_instance_id_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int id; + + mutex_lock(&mbwt->lock); + id = mbwt->pcie_instance_id; + mutex_unlock(&mbwt->lock); + + return sysfs_emit(buf, "%u\n", id); +} + +static ssize_t pcie_instance_id_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int val; + int err; + + err = kstrtou32(buf, 0, &val); + if (err) + return err; + if (val > TEGRA_BPMP_MBWT_MAX_PCIE_INSTANCE) + return -EINVAL; + + mutex_lock(&mbwt->lock); + mbwt->pcie_instance_id = val; + mutex_unlock(&mbwt->lock); + + return count; +} + +static ssize_t vc_type_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int vt; + + mutex_lock(&mbwt->lock); + vt = mbwt->vc_type; + mutex_unlock(&mbwt->lock); + + return sysfs_emit(buf, "%u\n", vt); +} + +static ssize_t vc_type_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int val; + int err; + + err = kstrtou32(buf, 0, &val); + if (err) + return err; + if (val > TEGRA_BPMP_MBWT_MAX_VC_TYPE) + return -EINVAL; + + mutex_lock(&mbwt->lock); + mbwt->vc_type = val; + mutex_unlock(&mbwt->lock); + + return count; +} + +static ssize_t bandwidth_show(struct kobject *kobj, struct kobj_attribute *attr, + char *buf) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int inst, vt, bw; + ssize_t ret; + int err; + + mutex_lock(&mbwt->lock); + inst = mbwt->pcie_instance_id; + vt = mbwt->vc_type; + + err = tegra_sochub_get_mbwt(mbwt->bpmp, inst, vt, &bw); + if (err) { + mutex_unlock(&mbwt->lock); + return err; + } + + ret = sysfs_emit(buf, "%u\n", bw); + mutex_unlock(&mbwt->lock); + + return ret; +} + +static ssize_t bandwidth_store(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = to_mbwt_sysfs(kobj); + unsigned int bw; + unsigned int inst, vt; + int err; + + err = kstrtou32(buf, 0, &bw); + if (err) + return err; + + mutex_lock(&mbwt->lock); + inst = mbwt->pcie_instance_id; + vt = mbwt->vc_type; + + err = tegra_sochub_set_mbwt(mbwt->bpmp, inst, vt, bw); + if (err) { + mutex_unlock(&mbwt->lock); + return err; + } + + mutex_unlock(&mbwt->lock); + + return count; +} + +static struct kobj_attribute pcie_instance_id_attr = + __ATTR(pcie_instance_id, 0644, pcie_instance_id_show, pcie_instance_id_store); +static struct kobj_attribute vc_type_attr = + __ATTR(vc_type, 0644, vc_type_show, vc_type_store); +static struct kobj_attribute bandwidth_attr = + __ATTR(bandwidth, 0644, bandwidth_show, bandwidth_store); + +static struct attribute *mbwt_attrs[] = { + &pcie_instance_id_attr.attr, + &vc_type_attr.attr, + &bandwidth_attr.attr, + NULL, +}; + +static const struct attribute_group mbwt_attr_group = { + .attrs = mbwt_attrs, +}; + +static void tegra_bpmp_mbwt_sysfs_teardown(void *data) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt = data; + + sysfs_remove_group(&mbwt->kobj, &mbwt_attr_group); + kobject_del(&mbwt->kobj); + kobject_put(&mbwt->kobj); +} + +int tegra_bpmp_sysfs_register(struct tegra_bpmp *bpmp) +{ + struct tegra_bpmp_mbwt_sysfs *mbwt; + int err; + + if (!ACPI_HANDLE(bpmp->dev)) + return 0; + + err = tegra_sochub_mbwt_query_abi(bpmp, CMD_SOCHUB_MBWT_SET_BW); + if (err) + return 0; + + err = tegra_sochub_mbwt_query_abi(bpmp, CMD_SOCHUB_MBWT_GET_BW); + if (err) + return 0; + + mbwt = kzalloc(sizeof(*mbwt), GFP_KERNEL); + if (!mbwt) + return -ENOMEM; + + mbwt->bpmp = bpmp; + mutex_init(&mbwt->lock); + + kobject_init(&mbwt->kobj, &tegra_bpmp_mbwt_ktype); + err = kobject_add(&mbwt->kobj, &bpmp->dev->kobj, "mbwt_control"); + if (err) { + kobject_put(&mbwt->kobj); + return err; + } + + err = sysfs_create_group(&mbwt->kobj, &mbwt_attr_group); + if (err) + goto err_put; + + err = devm_add_action(bpmp->dev, tegra_bpmp_mbwt_sysfs_teardown, mbwt); + if (err) { + sysfs_remove_group(&mbwt->kobj, &mbwt_attr_group); + goto err_put; + } + + return 0; + +err_put: + kobject_del(&mbwt->kobj); + kobject_put(&mbwt->kobj); + return err; +} diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c index 77b1fcae3d31b..d3132bc54b39b 100644 --- a/drivers/firmware/tegra/bpmp.c +++ b/drivers/firmware/tegra/bpmp.c @@ -954,6 +954,14 @@ static int tegra_bpmp_probe(struct platform_device *pdev) if (err < 0) goto free_mrq; + err = tegra_bpmp_sysfs_register(bpmp); + if (err < 0) { + dev_err(&pdev->dev, + "Failed registering sysfs attribute to the BPMP platform device: %d\n", + err); + goto free_mrq; + } + err = tegra_bpmp_init_debugfs(bpmp); if (err < 0) dev_err(&pdev->dev, "debugfs initialization failed: %d\n", err); @@ -1056,10 +1064,16 @@ static const struct of_device_id tegra_bpmp_match[] = { { } }; +static const struct acpi_device_id tegra_bpmp_acpi_match[] = { + {.id = "NVDA3001", .driver_data = 0}, + { } +}; + static struct platform_driver tegra_bpmp_driver = { .driver = { .name = "tegra-bpmp", .of_match_table = tegra_bpmp_match, + .acpi_match_table = tegra_bpmp_acpi_match, .pm = &tegra_bpmp_pm_ops, .suppress_bind_attrs = true, }, diff --git a/include/soc/tegra/bpmp-abi.h b/include/soc/tegra/bpmp-abi.h index dc0789c20333a..0b8d51556f124 100644 --- a/include/soc/tegra/bpmp-abi.h +++ b/include/soc/tegra/bpmp-abi.h @@ -341,6 +341,7 @@ struct mrq_response { #define MRQ_GEARS 82U #define MRQ_BWMGR_INT 83U #define MRQ_OC_STATUS 84U +#define MRQ_SOCHUB_MBWT 96U /** @cond DEPRECATED */ #define MRQ_RESERVED_2 2U @@ -374,7 +375,7 @@ struct mrq_response { * @brief Maximum MRQ code to be sent by CPU software to * BPMP. Subject to change in future */ -#define MAX_CPU_MRQ_ID 84U +#define MAX_CPU_MRQ_ID 96U /** * @addtogroup MRQ_Payloads @@ -3911,6 +3912,67 @@ struct mrq_gears_response { /** @} Gears */ /** @endcond bpmp_th500 */ +/** @cond (bpmp_tb500) + * @ingroup MRQ_Codes + * @def MRQ_SOCHUB_MBWT + * @brief Configure per-virtual-channel bandwidth caps for a SoC Hub instance using + * Memory Bandwidth Throttler (MBWT). + * + * * Initiators: Any + * * Targets: BPMP + * * Request Payload: @ref mrq_sochub_mbwt_request + * * Response Payload: @ref mrq_sochub_mbwt_response + * + * @addtogroup SOCHUB_MBWT + * @{ + */ + +/** + * @brief Sub-command identifiers for #MRQ_SOCHUB_MBWT. + */ +enum mrq_sochub_mbwt_cmd { + CMD_SOCHUB_MBWT_QUERY_ABI = 0, + CMD_SOCHUB_MBWT_GET_BW = 1, + CMD_SOCHUB_MBWT_SET_BW = 2, +}; + +struct cmd_sochub_mbwt_query_abi_req { + uint32_t cmd_code; +} BPMP_ABI_PACKED; + +struct cmd_sochub_mbwt_get_bw_req { + uint32_t instance; + uint32_t vc_type; +} BPMP_ABI_PACKED; + +struct cmd_sochub_mbwt_set_bw_req { + uint32_t instance; + uint32_t vc_type; + uint32_t bw; +} BPMP_ABI_PACKED; + +struct cmd_sochub_mbwt_get_bw_resp { + uint32_t bw; +} BPMP_ABI_PACKED; + +struct mrq_sochub_mbwt_request { + uint32_t cmd; + union { + struct cmd_sochub_mbwt_query_abi_req query_abi; + struct cmd_sochub_mbwt_get_bw_req get_bw; + struct cmd_sochub_mbwt_set_bw_req set_bw; + } BPMP_UNION_ANON; +} BPMP_ABI_PACKED; + +struct mrq_sochub_mbwt_response { + union { + struct cmd_sochub_mbwt_get_bw_resp get_bw; + } BPMP_UNION_ANON; +} BPMP_ABI_PACKED; + +/** @} SOCHUB_MBWT */ +/** @endcond */ + /** * @addtogroup Error_Codes * Negative values for mrq_response::err generally indicate some diff --git a/include/soc/tegra/bpmp.h b/include/soc/tegra/bpmp.h index f5e4ac5b8cce8..bae8ad985cbcb 100644 --- a/include/soc/tegra/bpmp.h +++ b/include/soc/tegra/bpmp.h @@ -140,6 +140,7 @@ int tegra_bpmp_request_mrq(struct tegra_bpmp *bpmp, unsigned int mrq, void tegra_bpmp_free_mrq(struct tegra_bpmp *bpmp, unsigned int mrq, void *data); bool tegra_bpmp_mrq_is_supported(struct tegra_bpmp *bpmp, unsigned int mrq); +int tegra_bpmp_sysfs_register(struct tegra_bpmp *bpmp); #else static inline struct tegra_bpmp *tegra_bpmp_get(struct device *dev) { @@ -181,6 +182,12 @@ static inline bool tegra_bpmp_mrq_is_supported(struct tegra_bpmp *bpmp, { return false; } + +static inline int tegra_bpmp_sysfs_register(struct tegra_bpmp *bpmp) +{ + return -EOPNOTSUPP; +} + #endif void tegra_bpmp_handle_rx(struct tegra_bpmp *bpmp); From 5400a8f9d2ecf784e15299f1872085a6d309f383 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Mon, 18 May 2026 10:06:30 -0700 Subject: [PATCH 277/464] firmware: arm_ffa: Honor partition info descriptor size BugLink: https://bugs.launchpad.net/bugs/2154045 FFA_PARTITION_INFO_GET_REGS reports the size of each partition information descriptor in x2[63:48]. However, __ffa_partition_info_get_regs() walks the returned register payload with a hardcoded 24-byte stride (regs += 3), even though the size is already read into buf_sz. That works for the FF-A v1.1/v1.2 24-byte descriptor layout, where each descriptor consumes three registers. Newer FF-A revisions can extend the descriptor while keeping the existing fields at the front. For example, a 48-byte descriptor consumes six registers, so advancing by only three registers desynchronises the parser and can make it read subsequent entries from the middle of a descriptor. Use the advertised descriptor size to derive the register stride. Validate that the size is register-aligned, large enough for the fields parsed by the driver, and that the requested number of descriptors fits in the returned x3..x17 register window. The driver still copies only the fields it understands, but now skips over any trailing descriptor fields correctly. Fixes: ba85c644ac8d ("firmware: arm_ffa: Add support for FFA_PARTITION_INFO_GET_REGS") Suggested-by: Sudeep Holla Signed-off-by: Jamie Nguyen Link: https://patch.msgid.link/20260518203116.42624-1-jamien@nvidia.com (sudeep.holla: Minor rewordng of the commit message and subject) Signed-off-by: Sudeep Holla (backported from commit 01b9cae706161a39452a2cce0f281d4369344c51 linux-next) Signed-off-by: Jamie Nguyen Acked-by: Nirmoy Das Acked-by: Seth Forshee Acked-by: Carol L Soto Acked-by: Lee Trager Signed-off-by: Brad Figg --- drivers/firmware/arm_ffa/driver.c | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c index e0263c3fad70b..e2e37200a32b1 100644 --- a/drivers/firmware/arm_ffa/driver.c +++ b/drivers/firmware/arm_ffa/driver.c @@ -324,11 +324,9 @@ __ffa_partition_info_get(u32 uuid0, u32 uuid1, u32 uuid2, u32 uuid3, #define PART_INFO_EXEC_CXT_MASK GENMASK(31, 16) #define PART_INFO_PROPS_MASK GENMASK(63, 32) #define FFA_PART_INFO_GET_REGS_FIRST_REG 3 -#define FFA_PART_INFO_GET_REGS_REGS_PER_DESC 3 -#define FFA_PART_INFO_GET_REGS_MAX_DESC \ - (((sizeof(ffa_value_t) / sizeof_field(ffa_value_t, a0)) - \ - FFA_PART_INFO_GET_REGS_FIRST_REG) / \ - FFA_PART_INFO_GET_REGS_REGS_PER_DESC) +#define FFA_PART_INFO_GET_REGS_MIN_REGS_PER_DESC 3 +#define FFA_PART_INFO_GET_REGS_NUM_REGS \ + (sizeof(ffa_value_t) / sizeof_field(ffa_value_t, a0)) #define PART_INFO_ID(x) ((u16)(FIELD_GET(PART_INFO_ID_MASK, (x)))) #define PART_INFO_EXEC_CXT(x) ((u16)(FIELD_GET(PART_INFO_EXEC_CXT_MASK, (x)))) #define PART_INFO_PROPERTIES(x) ((u32)(FIELD_GET(PART_INFO_PROPS_MASK, (x)))) @@ -342,7 +340,7 @@ __ffa_partition_info_get_regs(u32 uuid0, u32 uuid1, u32 uuid2, u32 uuid3, do { __le64 *regs; - int idx, nr_desc, buf_idx; + int idx, nr_desc, buf_idx, regs_per_desc, max_desc; invoke_ffa_fn((ffa_value_t){ .a0 = FFA_PARTITION_INFO_GET_REGS, @@ -365,8 +363,18 @@ __ffa_partition_info_get_regs(u32 uuid0, u32 uuid1, u32 uuid2, u32 uuid3, if (cur_idx < start_idx || cur_idx >= count) return -EINVAL; + buf_sz = PARTITION_INFO_SZ(partition_info.a2); + if (buf_sz % sizeof(*regs)) + return -EINVAL; + + regs_per_desc = buf_sz / sizeof(*regs); + if (regs_per_desc < FFA_PART_INFO_GET_REGS_MIN_REGS_PER_DESC) + return -EINVAL; + nr_desc = cur_idx - start_idx + 1; - if (nr_desc > FFA_PART_INFO_GET_REGS_MAX_DESC) + max_desc = (FFA_PART_INFO_GET_REGS_NUM_REGS - + FFA_PART_INFO_GET_REGS_FIRST_REG) / regs_per_desc; + if (nr_desc > max_desc) return -EINVAL; buf_idx = buf - buffer; @@ -374,9 +382,6 @@ __ffa_partition_info_get_regs(u32 uuid0, u32 uuid1, u32 uuid2, u32 uuid3, return -EINVAL; tag = UUID_INFO_TAG(partition_info.a2); - buf_sz = PARTITION_INFO_SZ(partition_info.a2); - if (buf_sz > sizeof(*buffer)) - buf_sz = sizeof(*buffer); regs = (void *)&partition_info.a3; for (idx = 0; idx < nr_desc; idx++, buf++) { @@ -395,7 +400,7 @@ __ffa_partition_info_get_regs(u32 uuid0, u32 uuid1, u32 uuid2, u32 uuid3, buf->exec_ctxt = PART_INFO_EXEC_CXT(val); buf->properties = PART_INFO_PROPERTIES(val); uuid_copy(&buf->uuid, &uuid_regs.uuid); - regs += 3; + regs += regs_per_desc; } start_idx = cur_idx + 1; From 8aa66a105cedfeb3c74198ecad8a168e14aa9b57 Mon Sep 17 00:00:00 2001 From: Vidya Sagar Date: Wed, 25 Feb 2026 19:08:01 +0530 Subject: [PATCH 278/464] PCI/CXL: Hide SBR from reset_methods if masked by CXL BugLink: https://bugs.launchpad.net/bugs/2154302 Per CXL r3.1, sec 8.1.5.2, the Secondary Bus Reset (SBR) bit in the Bridge Control register of a CXL port has no effect unless the "Unmask SBR" bit in the Port Control Extensions Register is set. After b1956e2d0713 ("PCI/CXL: Fail bus reset if upstream CXL Port has SBR masked"), Linux checks the "Unmask SBR" bit in pci_reset_bus_function(). But when probe==true, it previously returned 0, incorrectly indicating that SBR is a viable reset method for the device. As a result, "bus" is listed in the device's "reset_method" attribute even though the hardware is incapable of performing it. If a user writes "bus" to "reset_method" or triggers a reset that falls back to SBR, the operation fails with "write error: Inappropriate ioctl for device". If the link is operating in CXL mode (pcie_is_cxl()), return -ENOTTY immediately unless "Unmask SBR" is set, regardless of the probe argument. This ensures that "bus" is not advertised in "reset_methods" when the hardware prevents it, improving clarity for users and aligning the sysfs capability report with actual hardware behavior. Signed-off-by: Vidya Sagar [bhelgaas: commit log, use pcie_is_cxl()] Signed-off-by: Bjorn Helgaas Reviewed-by: Jonathan Cameron Reviewed-by: Dave Jiang Link: https://patch.msgid.link/20260225133801.30231-1-vidyas@nvidia.com (cherry picked from commit 702c1d56c7177a0481abd2814bab9495f1150967) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/pci/pci.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 2f8e7dfd391eb..6597fbb538537 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4935,12 +4935,8 @@ static int pci_reset_bus_function(struct pci_dev *dev, bool probe) * If "dev" is below a CXL port that has SBR control masked, SBR * won't do anything, so return error. */ - if (bridge && cxl_sbr_masked(bridge)) { - if (probe) - return 0; - + if (bridge && pcie_is_cxl(bridge) && cxl_sbr_masked(bridge)) return -ENOTTY; - } rc = pci_dev_reset_iommu_prepare(dev); if (rc) { From 73875ebe5e6b8c3b45567884ac67ff64335f8798 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:40 +0000 Subject: [PATCH 279/464] arm64/sysreg: Add MPAMSM_EL1 register BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAMSM_EL1 register determines the MPAM configuration for an SMCU. Add the register definition. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Acked-by: Catalin Marinas Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 29fa1be82b83f87e603ed4c21fe86c6e05fd0282) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/tools/sysreg | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index 9d1c211080571..1287cb1de6f3c 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -5172,6 +5172,14 @@ Field 31:16 PARTID_D Field 15:0 PARTID_I EndSysreg +Sysreg MPAMSM_EL1 3 0 10 5 3 +Res0 63:48 +Field 47:40 PMG_D +Res0 39:32 +Field 31:16 PARTID_D +Res0 15:0 +EndSysreg + Sysreg ISR_EL1 3 0 12 1 0 Res0 63:11 Field 10 IS From a054f51464b68e79c8b1efc07c1beddb0fdae83a Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:41 +0000 Subject: [PATCH 280/464] KVM: arm64: Preserve host MPAM configuration when changing traps BugLink: https://bugs.launchpad.net/bugs/2154527 When KVM enables or disables MPAM traps to EL2 it clears all other bits in MPAM2_EL2. Notably, it clears the partition ids (PARTIDs) and performance monitoring groups (PMGs). Avoid changing these bits in anticipation of adding support for MPAM in the kernel. Otherwise, on a VHE system with the host running at EL2 where MPAM2_EL2 and MPAM1_EL1 access the same register, any attempt to use MPAM to monitor or partition resources for kernel space would be foiled by running a KVM guest. Additionally, MPAM2_EL2.EnMPAMSM is always set to 0 which causes MPAMSM_EL1 to always trap. Keep EnMPAMSM set to 1 when not in a guest so that the kernel can use MPAMSM_EL1. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Acked-by: Marc Zyngier Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit eda1cd1f9d29b382a07d757cf8b29f9ee636355f) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/kvm/hyp/include/hyp/switch.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 4ca856f6f5fb0..7a5222065d481 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -267,7 +267,8 @@ static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu) static inline void __activate_traps_mpam(struct kvm_vcpu *vcpu) { - u64 r = MPAM2_EL2_TRAPMPAM0EL1 | MPAM2_EL2_TRAPMPAM1EL1; + u64 clr = MPAM2_EL2_EnMPAMSM; + u64 set = MPAM2_EL2_TRAPMPAM0EL1 | MPAM2_EL2_TRAPMPAM1EL1; if (!system_supports_mpam()) return; @@ -277,18 +278,21 @@ static inline void __activate_traps_mpam(struct kvm_vcpu *vcpu) write_sysreg_s(MPAMHCR_EL2_TRAP_MPAMIDR_EL1, SYS_MPAMHCR_EL2); } else { /* From v1.1 TIDR can trap MPAMIDR, set it unconditionally */ - r |= MPAM2_EL2_TIDR; + set |= MPAM2_EL2_TIDR; } - write_sysreg_s(r, SYS_MPAM2_EL2); + sysreg_clear_set_s(SYS_MPAM2_EL2, clr, set); } static inline void __deactivate_traps_mpam(void) { + u64 clr = MPAM2_EL2_TRAPMPAM0EL1 | MPAM2_EL2_TRAPMPAM1EL1 | MPAM2_EL2_TIDR; + u64 set = MPAM2_EL2_EnMPAMSM; + if (!system_supports_mpam()) return; - write_sysreg_s(0, SYS_MPAM2_EL2); + sysreg_clear_set_s(SYS_MPAM2_EL2, clr, set); if (system_supports_mpam_hcr()) write_sysreg_s(MPAMHCR_HOST_FLAGS, SYS_MPAMHCR_EL2); From eeec77d0fe8f82e741622e13ab9177244f93f29e Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:42 +0000 Subject: [PATCH 281/464] KVM: arm64: Make MPAMSM_EL1 accesses UNDEF BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAMSM_EL1 register controls the MPAM labeling for an SMCU, Streaming Mode Compute Unit. As there is no MPAM support in KVM, make sure MPAMSM_EL1 accesses trigger an UNDEF. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Acked-by: Marc Zyngier Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 2e7c684bdb50cfaf98da80ebaab4a961fdcd1aa2) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/kvm/sys_regs.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9d980ad6e7c59..ab09127f411d2 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3371,6 +3371,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_MPAM1_EL1), undef_access }, { SYS_DESC(SYS_MPAM0_EL1), undef_access }, + { SYS_DESC(SYS_MPAMSM_EL1), undef_access }, + { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, From 915f0bec07c43173cca8c1c514da35d008a63340 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:43 +0000 Subject: [PATCH 282/464] arm64: mpam: Context switch the MPAM registers BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM allows traffic in the SoC to be labeled by the OS, these labels are used to apply policy in caches and bandwidth regulators, and to monitor traffic in the SoC. The label is made up of a PARTID and PMG value. The x86 equivalent calls these CLOSID and RMID, but they don't map precisely. MPAM has two CPU system registers that is used to hold the PARTID and PMG values that traffic generated at each exception level will use. These can be set per-task by the resctrl file system. (resctrl is the defacto interface for controlling this stuff). Add a helper to switch this. struct task_struct's separate CLOSID and RMID fields are insufficient to implement resctrl using MPAM, as resctrl can change the PARTID (CLOSID) and PMG (sort of like the RMID) separately. On x86, the rmid is an independent number, so a race that writes a mismatched closid and rmid into hardware is benign. On arm64, the pmg bits extend the partid. (i.e. partid-5 has a pmg-0 that is not the same as partid-6's pmg-0). In this case, mismatching the values will 'dirty' a pmg value that resctrl believes is clean, and is not tracking with its 'limbo' code. To avoid this, the partid and pmg are always read and written as a pair. This requires a new u64 field. In struct task_struct there are two u32, rmid and closid for the x86 case, but as we can't use them here do something else. Add this new field, mpam_partid_pmg, to struct thread_info to avoid adding more architecture specific code to struct task_struct. Always use READ_ONCE()/WRITE_ONCE() when accessing this field. Resctrl allows a per-cpu 'default' value to be set, this overrides the values when scheduling a task in the default control-group, which has PARTID 0. The way 'code data prioritisation' gets emulated means the register value for the default group needs to be a variable. The current system register value is kept in a per-cpu variable to avoid writing to the system register if the value isn't going to change. Writes to this register may reset the hardware state for regulating bandwidth. Finally, there is no reason to context switch these registers unless there is a driver changing the values in struct task_struct. Hide the whole thing behind a static key. This also allows the driver to disable MPAM in response to errors reported by hardware. Move the existing static key to belong to the arch code, as in the future the MPAM driver may become a loadable module. All this should depend on whether there is an MPAM driver, hide it behind CONFIG_ARM64_MPAM. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick CC: Amit Singh Tomar Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Reviewed-by: Catalin Marinas Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 8e06d04ff1cf764066c62e5677bfb0b0c1d1fbbc) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/Kconfig | 2 + arch/arm64/include/asm/mpam.h | 67 ++++++++++++++++++++++++++++ arch/arm64/include/asm/thread_info.h | 3 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/mpam.c | 13 ++++++ arch/arm64/kernel/process.c | 7 +++ drivers/resctrl/mpam_devices.c | 2 - drivers/resctrl/mpam_internal.h | 4 +- 8 files changed, 95 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/include/asm/mpam.h create mode 100644 arch/arm64/kernel/mpam.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7d181d4d6aebe..cc492c86543ff 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2070,6 +2070,8 @@ config ARM64_MPAM MPAM is exposed to user-space via the resctrl pseudo filesystem. + This option enables the extra context switch code. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features" diff --git a/arch/arm64/include/asm/mpam.h b/arch/arm64/include/asm/mpam.h new file mode 100644 index 0000000000000..0747e0526927d --- /dev/null +++ b/arch/arm64/include/asm/mpam.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (C) 2025 Arm Ltd. */ + +#ifndef __ASM__MPAM_H +#define __ASM__MPAM_H + +#include +#include +#include + +#include + +DECLARE_STATIC_KEY_FALSE(mpam_enabled); +DECLARE_PER_CPU(u64, arm64_mpam_default); +DECLARE_PER_CPU(u64, arm64_mpam_current); + +/* + * The value of the MPAM0_EL1 sysreg when a task is in resctrl's default group. + * This is used by the context switch code to use the resctrl CPU property + * instead. The value is modified when CDP is enabled/disabled by mounting + * the resctrl filesystem. + */ +extern u64 arm64_mpam_global_default; + +/* + * The resctrl filesystem writes to the partid/pmg values for threads and CPUs, + * which may race with reads in mpam_thread_switch(). Ensure only one of the old + * or new values are used. Particular care should be taken with the pmg field as + * mpam_thread_switch() may read a partid and pmg that don't match, causing this + * value to be stored with cache allocations, despite being considered 'free' by + * resctrl. + */ +#ifdef CONFIG_ARM64_MPAM +static inline u64 mpam_get_regval(struct task_struct *tsk) +{ + return READ_ONCE(task_thread_info(tsk)->mpam_partid_pmg); +} + +static inline void mpam_thread_switch(struct task_struct *tsk) +{ + u64 oldregval; + int cpu = smp_processor_id(); + u64 regval = mpam_get_regval(tsk); + + if (!static_branch_likely(&mpam_enabled)) + return; + + if (regval == READ_ONCE(arm64_mpam_global_default)) + regval = READ_ONCE(per_cpu(arm64_mpam_default, cpu)); + + oldregval = READ_ONCE(per_cpu(arm64_mpam_current, cpu)); + if (oldregval == regval) + return; + + write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + isb(); + + /* Synchronising the EL0 write is left until the ERET to EL0 */ + write_sysreg_s(regval, SYS_MPAM0_EL1); + + WRITE_ONCE(per_cpu(arm64_mpam_current, cpu), regval); +} +#else +static inline void mpam_thread_switch(struct task_struct *tsk) {} +#endif /* CONFIG_ARM64_MPAM */ + +#endif /* __ASM__MPAM_H */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 7942478e40658..5d7fe3e153c85 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -41,6 +41,9 @@ struct thread_info { #ifdef CONFIG_SHADOW_CALL_STACK void *scs_base; void *scs_sp; +#endif +#ifdef CONFIG_ARM64_MPAM + u64 mpam_partid_pmg; #endif u32 cpu; }; diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index fe627100d1990..74b76bb704523 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -68,6 +68,7 @@ obj-$(CONFIG_CRASH_DUMP) += crash_dump.o obj-$(CONFIG_VMCORE_INFO) += vmcore_info.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_ARM64_MPAM) += mpam.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o diff --git a/arch/arm64/kernel/mpam.c b/arch/arm64/kernel/mpam.c new file mode 100644 index 0000000000000..9866d2ca0faa9 --- /dev/null +++ b/arch/arm64/kernel/mpam.c @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (C) 2025 Arm Ltd. */ + +#include + +#include +#include + +DEFINE_STATIC_KEY_FALSE(mpam_enabled); +DEFINE_PER_CPU(u64, arm64_mpam_default); +DEFINE_PER_CPU(u64, arm64_mpam_current); + +u64 arm64_mpam_global_default; diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 489554931231e..47698955fa1e4 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include #include @@ -738,6 +739,12 @@ struct task_struct *__switch_to(struct task_struct *prev, if (prev->thread.sctlr_user != next->thread.sctlr_user) update_sctlr_el1(next->thread.sctlr_user); + /* + * MPAM thread switch happens after the DSB to ensure prev's accesses + * use prev's MPAM settings. + */ + mpam_thread_switch(next); + /* the actual thread switch */ last = cpu_switch_to(prev, next); diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index fd4f107bd00e3..8590397033b9c 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -29,8 +29,6 @@ #include "mpam_internal.h" -DEFINE_STATIC_KEY_FALSE(mpam_enabled); /* This moves to arch code */ - /* * mpam_list_lock protects the SRCU lists when writing. Once the * mpam_enabled key is enabled these lists are read-only, diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 5f1fca64ce8b9..84e26c0c7050f 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -16,12 +16,12 @@ #include #include +#include + #define MPAM_MSC_MAX_NUM_RIS 16 struct platform_device; -DECLARE_STATIC_KEY_FALSE(mpam_enabled); - #ifdef CONFIG_MPAM_KUNIT_TEST #define PACKED_FOR_KUNIT __packed #else From b7eaff40a30d9d5e38e3c0c1df7d01b278ae8fc8 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:44 +0000 Subject: [PATCH 283/464] arm64: mpam: Re-initialise MPAM regs when CPU comes online BugLink: https://bugs.launchpad.net/bugs/2154527 Now that the MPAM system registers are expected to have values that change, reprogram them based on the previous value when a CPU is brought online. Previously MPAM's 'default PARTID' of 0 was always used for MPAM in kernel-space as this is the PARTID that hardware guarantees to reset. Because there are a limited number of PARTID, this value is exposed to user-space, meaning resctrl changes to the resctrl default group would also affect kernel threads. Instead, use the task's PARTID value for kernel work on behalf of user-space too. The default of 0 is kept for both user-space and kernel-space when MPAM is not enabled. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Reviewed-by: Catalin Marinas Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 87b78a5d70e83d4dbe31e1afda2be736a3330b31) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/kernel/cpufeature.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 348197d9b6017..175aa9d5c0a01 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -86,6 +86,7 @@ #include #include #include +#include #include #include #include @@ -2501,13 +2502,17 @@ test_has_mpam(const struct arm64_cpu_capabilities *entry, int scope) static void cpu_enable_mpam(const struct arm64_cpu_capabilities *entry) { - /* - * Access by the kernel (at EL1) should use the reserved PARTID - * which is configured unrestricted. This avoids priority-inversion - * where latency sensitive tasks have to wait for a task that has - * been throttled to release the lock. - */ - write_sysreg_s(0, SYS_MPAM1_EL1); + int cpu = smp_processor_id(); + u64 regval = 0; + + if (IS_ENABLED(CONFIG_ARM64_MPAM) && static_branch_likely(&mpam_enabled)) + regval = READ_ONCE(per_cpu(arm64_mpam_current, cpu)); + + write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + isb(); + + /* Synchronising the EL0 write is left until the ERET to EL0 */ + write_sysreg_s(regval, SYS_MPAM0_EL1); } static bool From 22f3863f679e81b2ac16559416eb928f95aa97b3 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:45 +0000 Subject: [PATCH 284/464] arm64: mpam: Drop the CONFIG_EXPERT restriction BugLink: https://bugs.launchpad.net/bugs/2154527 In anticipation of MPAM being useful remove the CONFIG_EXPERT restriction. This was done to prevent the driver being enabled before the user-space interface was wired up. Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Reviewed-by: James Morse Acked-by: Catalin Marinas Signed-off-by: Ben Horgan [ morse: Added second paragraph ] Signed-off-by: James Morse (cherry picked from commit c544f00a473239835d22e7109b403314d8b85974) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/Kconfig | 2 +- drivers/resctrl/Kconfig | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cc492c86543ff..a34eb53c563a2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2047,7 +2047,7 @@ config ARM64_TLB_RANGE config ARM64_MPAM bool "Enable support for MPAM" - select ARM64_MPAM_DRIVER if EXPERT # does nothing yet + select ARM64_MPAM_DRIVER select ACPI_MPAM if ACPI help Memory System Resource Partitioning and Monitoring (MPAM) is an diff --git a/drivers/resctrl/Kconfig b/drivers/resctrl/Kconfig index c808e04703946..c34e059c6e41f 100644 --- a/drivers/resctrl/Kconfig +++ b/drivers/resctrl/Kconfig @@ -1,6 +1,6 @@ menuconfig ARM64_MPAM_DRIVER bool "MPAM driver" - depends on ARM64 && ARM64_MPAM && EXPERT + depends on ARM64 && ARM64_MPAM help Memory System Resource Partitioning and Monitoring (MPAM) driver for System IP, e.g. caches and memory controllers. From b9e060d1f3e41b5e9e255ebd2fa55c35b7a7159e Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:46 +0000 Subject: [PATCH 285/464] arm64: mpam: Advertise the CPUs MPAM limits to the driver BugLink: https://bugs.launchpad.net/bugs/2154527 Requesters need to populate the MPAM fields for any traffic they send on the interconnect. For the CPUs these values are taken from the corresponding MPAMy_ELx register. Each requester may have a limit on the largest PARTID or PMG value that can be used. The MPAM driver has to determine the system-wide minimum supported PARTID and PMG values. To do this, the driver needs to be told what each requestor's limit is. CPUs are special, but this infrastructure is also needed for the SMMU and GIC ITS. Call the helper to tell the MPAM driver what the CPUs can do. The return value can be ignored by the arch code as it runs well before the MPAM driver starts probing. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Catalin Marinas Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan [ morse: requestor->requester as argued by ispell ] Signed-off-by: James Morse (cherry picked from commit 831a7f16728c5ceef04ab99a699c3d9e519dc4b8) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/kernel/mpam.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/arm64/kernel/mpam.c b/arch/arm64/kernel/mpam.c index 9866d2ca0faa9..e6feff2324acb 100644 --- a/arch/arm64/kernel/mpam.c +++ b/arch/arm64/kernel/mpam.c @@ -3,6 +3,7 @@ #include +#include #include #include @@ -11,3 +12,14 @@ DEFINE_PER_CPU(u64, arm64_mpam_default); DEFINE_PER_CPU(u64, arm64_mpam_current); u64 arm64_mpam_global_default; + +static int __init arm64_mpam_register_cpus(void) +{ + u64 mpamidr = read_sanitised_ftr_reg(SYS_MPAMIDR_EL1); + u16 partid_max = FIELD_GET(MPAMIDR_EL1_PARTID_MAX, mpamidr); + u8 pmg_max = FIELD_GET(MPAMIDR_EL1_PMG_MAX, mpamidr); + + return mpam_register_requestor(partid_max, pmg_max); +} +/* Must occur before mpam_msc_driver_init() from subsys_initcall() */ +arch_initcall(arm64_mpam_register_cpus) From a65b8a8d204801fa828841a85db74ea4068f3e30 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:47 +0000 Subject: [PATCH 286/464] arm64: mpam: Add cpu_pm notifier to restore MPAM sysregs BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAM system registers will be lost if the CPU is reset during PSCI's CPU_SUSPEND. Add a PM notifier to restore them. mpam_thread_switch(current) can't be used as this won't make any changes if the in-memory copy says the register already has the correct value. In reality the system register is UNKNOWN out of reset. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Reviewed-by: Catalin Marinas Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 735dad999905dfd246be1994bb8d203063aeb0d6) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/kernel/mpam.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/arm64/kernel/mpam.c b/arch/arm64/kernel/mpam.c index e6feff2324acb..48ec0ffd59997 100644 --- a/arch/arm64/kernel/mpam.c +++ b/arch/arm64/kernel/mpam.c @@ -4,6 +4,7 @@ #include #include +#include #include #include @@ -13,12 +14,44 @@ DEFINE_PER_CPU(u64, arm64_mpam_current); u64 arm64_mpam_global_default; +static int mpam_pm_notifier(struct notifier_block *self, + unsigned long cmd, void *v) +{ + u64 regval; + int cpu = smp_processor_id(); + + switch (cmd) { + case CPU_PM_EXIT: + /* + * Don't use mpam_thread_switch() as the system register + * value has changed under our feet. + */ + regval = READ_ONCE(per_cpu(arm64_mpam_current, cpu)); + write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + isb(); + + write_sysreg_s(regval, SYS_MPAM0_EL1); + + return NOTIFY_OK; + default: + return NOTIFY_DONE; + } +} + +static struct notifier_block mpam_pm_nb = { + .notifier_call = mpam_pm_notifier, +}; + static int __init arm64_mpam_register_cpus(void) { u64 mpamidr = read_sanitised_ftr_reg(SYS_MPAMIDR_EL1); u16 partid_max = FIELD_GET(MPAMIDR_EL1_PARTID_MAX, mpamidr); u8 pmg_max = FIELD_GET(MPAMIDR_EL1_PMG_MAX, mpamidr); + if (!system_supports_mpam()) + return 0; + + cpu_pm_register_notifier(&mpam_pm_nb); return mpam_register_requestor(partid_max, pmg_max); } /* Must occur before mpam_msc_driver_init() from subsys_initcall() */ From cdd2ad1693979d69ccf3d26ec32d262b7643c819 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:48 +0000 Subject: [PATCH 287/464] arm64: mpam: Initialise and context switch the MPAMSM_EL1 register BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAMSM_EL1 sets the MPAM labels, PMG and PARTID, for loads and stores generated by a shared SMCU. Disable the traps so the kernel can use it and set it to the same configuration as the per-EL cpu MPAM configuration. If an SMCU is not shared with other cpus then it is implementation defined whether the configuration from MPAMSM_EL1 is used or that from the appropriate MPAMy_ELx. As we set the same, PMG_D and PARTID_D, configuration for MPAM0_EL1, MPAM1_EL1 and MPAMSM_EL1 the resulting configuration is the same regardless. The range of valid configurations for the PARTID and PMG in MPAMSM_EL1 is not currently specified in Arm Architectural Reference Manual but the architect has confirmed that it is intended to be the same as that for the cpu configuration in the MPAMy_ELx registers. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Reviewed-by: Catalin Marinas Reviewed-by: James Morse Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 37fe0f984d9ca60e8d95fc9a85d37f4300159625) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/include/asm/el2_setup.h | 3 ++- arch/arm64/include/asm/mpam.h | 2 ++ arch/arm64/kernel/cpufeature.c | 2 ++ arch/arm64/kernel/mpam.c | 4 ++++ 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h index 85f4c1615472d..4d15071a4f3fc 100644 --- a/arch/arm64/include/asm/el2_setup.h +++ b/arch/arm64/include/asm/el2_setup.h @@ -513,7 +513,8 @@ check_override id_aa64pfr0, ID_AA64PFR0_EL1_MPAM_SHIFT, .Linit_mpam_\@, .Lskip_mpam_\@, x1, x2 .Linit_mpam_\@: - msr_s SYS_MPAM2_EL2, xzr // use the default partition + mov x0, #MPAM2_EL2_EnMPAMSM_MASK + msr_s SYS_MPAM2_EL2, x0 // use the default partition, // and disable lower traps mrs_s x0, SYS_MPAMIDR_EL1 tbz x0, #MPAMIDR_EL1_HAS_HCR_SHIFT, .Lskip_mpam_\@ // skip if no MPAMHCR reg diff --git a/arch/arm64/include/asm/mpam.h b/arch/arm64/include/asm/mpam.h index 0747e0526927d..6bccbfdccb87e 100644 --- a/arch/arm64/include/asm/mpam.h +++ b/arch/arm64/include/asm/mpam.h @@ -53,6 +53,8 @@ static inline void mpam_thread_switch(struct task_struct *tsk) return; write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + if (system_supports_sme()) + write_sysreg_s(regval & (MPAMSM_EL1_PARTID_D | MPAMSM_EL1_PMG_D), SYS_MPAMSM_EL1); isb(); /* Synchronising the EL0 write is left until the ERET to EL0 */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 175aa9d5c0a01..f5de18a987544 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2509,6 +2509,8 @@ cpu_enable_mpam(const struct arm64_cpu_capabilities *entry) regval = READ_ONCE(per_cpu(arm64_mpam_current, cpu)); write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + if (cpus_have_cap(ARM64_SME)) + write_sysreg_s(regval & (MPAMSM_EL1_PARTID_D | MPAMSM_EL1_PMG_D), SYS_MPAMSM_EL1); isb(); /* Synchronising the EL0 write is left until the ERET to EL0 */ diff --git a/arch/arm64/kernel/mpam.c b/arch/arm64/kernel/mpam.c index 48ec0ffd59997..3a490de4fa125 100644 --- a/arch/arm64/kernel/mpam.c +++ b/arch/arm64/kernel/mpam.c @@ -28,6 +28,10 @@ static int mpam_pm_notifier(struct notifier_block *self, */ regval = READ_ONCE(per_cpu(arm64_mpam_current, cpu)); write_sysreg_s(regval | MPAM1_EL1_MPAMEN, SYS_MPAM1_EL1); + if (system_supports_sme()) { + write_sysreg_s(regval & (MPAMSM_EL1_PARTID_D | MPAMSM_EL1_PMG_D), + SYS_MPAMSM_EL1); + } isb(); write_sysreg_s(regval, SYS_MPAM0_EL1); From 992acf20a03eb04fae41cc97d7060b3a9257b397 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:49 +0000 Subject: [PATCH 288/464] arm64: mpam: Add helpers to change a task or cpu's MPAM PARTID/PMG values BugLink: https://bugs.launchpad.net/bugs/2154527 Care must be taken when modifying the PARTID and PMG of a task in any per-task structure as writing these values may race with the task being scheduled in, and reading the modified values. Add helpers to set the task properties, and the CPU default value. These use WRITE_ONCE() that pairs with the READ_ONCE() in mpam_get_regval() to avoid causing torn values. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Cc: Dave Martin Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Catalin Marinas Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 2cf9ca3fae38b7894e7f1435cec92f9a679b42f9) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/include/asm/mpam.h | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/mpam.h b/arch/arm64/include/asm/mpam.h index 6bccbfdccb87e..05aa71200f61a 100644 --- a/arch/arm64/include/asm/mpam.h +++ b/arch/arm64/include/asm/mpam.h @@ -4,6 +4,7 @@ #ifndef __ASM__MPAM_H #define __ASM__MPAM_H +#include #include #include #include @@ -22,6 +23,23 @@ DECLARE_PER_CPU(u64, arm64_mpam_current); */ extern u64 arm64_mpam_global_default; +#ifdef CONFIG_ARM64_MPAM +static inline u64 __mpam_regval(u16 partid_d, u16 partid_i, u8 pmg_d, u8 pmg_i) +{ + return FIELD_PREP(MPAM0_EL1_PARTID_D, partid_d) | + FIELD_PREP(MPAM0_EL1_PARTID_I, partid_i) | + FIELD_PREP(MPAM0_EL1_PMG_D, pmg_d) | + FIELD_PREP(MPAM0_EL1_PMG_I, pmg_i); +} + +static inline void mpam_set_cpu_defaults(int cpu, u16 partid_d, u16 partid_i, + u8 pmg_d, u8 pmg_i) +{ + u64 default_val = __mpam_regval(partid_d, partid_i, pmg_d, pmg_i); + + WRITE_ONCE(per_cpu(arm64_mpam_default, cpu), default_val); +} + /* * The resctrl filesystem writes to the partid/pmg values for threads and CPUs, * which may race with reads in mpam_thread_switch(). Ensure only one of the old @@ -30,12 +48,20 @@ extern u64 arm64_mpam_global_default; * value to be stored with cache allocations, despite being considered 'free' by * resctrl. */ -#ifdef CONFIG_ARM64_MPAM static inline u64 mpam_get_regval(struct task_struct *tsk) { return READ_ONCE(task_thread_info(tsk)->mpam_partid_pmg); } +static inline void mpam_set_task_partid_pmg(struct task_struct *tsk, + u16 partid_d, u16 partid_i, + u8 pmg_d, u8 pmg_i) +{ + u64 regval = __mpam_regval(partid_d, partid_i, pmg_d, pmg_i); + + WRITE_ONCE(task_thread_info(tsk)->mpam_partid_pmg, regval); +} + static inline void mpam_thread_switch(struct task_struct *tsk) { u64 oldregval; From 763bd0b12a1a6d0e0c65e9541b141fb249f0f58e Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:51 +0000 Subject: [PATCH 289/464] arm_mpam: resctrl: Add boilerplate cpuhp and domain allocation BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl has its own data structures to describe its resources. We can't use these directly as we play tricks with the 'MBA' resource, picking the MPAM controls or monitors that best apply. We may export the same component as both L3 and MBA. Add mpam_resctrl_res[] as the array of class->resctrl mappings we are exporting, and add the cpuhp hooks that allocated and free the resctrl domain structures. Only the mpam control feature are considered here and monitor support will be added later. While we're here, plumb in a few other obvious things. CONFIG_ARM_CPU_RESCTRL is used to allow this code to be built even though it can't yet be linked against resctrl. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 09e61daf8e96b9bdb04dd112bdecf9382fd3f919) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/Makefile | 1 + drivers/resctrl/mpam_devices.c | 12 ++ drivers/resctrl/mpam_internal.h | 21 +++ drivers/resctrl/mpam_resctrl.c | 324 ++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 3 + 5 files changed, 361 insertions(+) create mode 100644 drivers/resctrl/mpam_resctrl.c diff --git a/drivers/resctrl/Makefile b/drivers/resctrl/Makefile index 898199dcf80d5..40beaf999582c 100644 --- a/drivers/resctrl/Makefile +++ b/drivers/resctrl/Makefile @@ -1,4 +1,5 @@ obj-$(CONFIG_ARM64_MPAM_DRIVER) += mpam.o mpam-y += mpam_devices.o +mpam-$(CONFIG_ARM_CPU_RESCTRL) += mpam_resctrl.o ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG) += -DDEBUG diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 8590397033b9c..699b3361a5584 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -1610,6 +1610,9 @@ static int mpam_cpu_online(unsigned int cpu) mpam_reprogram_msc(msc); } + if (mpam_is_enabled()) + return mpam_resctrl_online_cpu(cpu); + return 0; } @@ -1653,6 +1656,9 @@ static int mpam_cpu_offline(unsigned int cpu) { struct mpam_msc *msc; + if (mpam_is_enabled()) + mpam_resctrl_offline_cpu(cpu); + guard(srcu)(&mpam_srcu); list_for_each_entry_srcu(msc, &mpam_all_msc, all_msc_list, srcu_read_lock_held(&mpam_srcu)) { @@ -2501,6 +2507,12 @@ static void mpam_enable_once(void) mutex_unlock(&mpam_list_lock); cpus_read_unlock(); + if (!err) { + err = mpam_resctrl_setup(); + if (err) + pr_err("Failed to initialise resctrl: %d\n", err); + } + if (err) { mpam_disable_reason = "Failed to enable."; schedule_work(&mpam_broken_work); diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 84e26c0c7050f..11071d005d85d 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -331,6 +332,16 @@ struct mpam_msc_ris { struct mpam_garbage garbage; }; +struct mpam_resctrl_dom { + struct mpam_component *ctrl_comp; + struct rdt_ctrl_domain resctrl_ctrl_dom; +}; + +struct mpam_resctrl_res { + struct mpam_class *class; + struct rdt_resource resctrl_res; +}; + static inline int mpam_alloc_csu_mon(struct mpam_class *class) { struct mpam_props *cprops = &class->props; @@ -385,6 +396,16 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx); int mpam_get_cpumask_from_cache_id(unsigned long cache_id, u32 cache_level, cpumask_t *affinity); +#ifdef CONFIG_RESCTRL_FS +int mpam_resctrl_setup(void); +int mpam_resctrl_online_cpu(unsigned int cpu); +void mpam_resctrl_offline_cpu(unsigned int cpu); +#else +static inline int mpam_resctrl_setup(void) { return 0; } +static inline int mpam_resctrl_online_cpu(unsigned int cpu) { return 0; } +static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { } +#endif /* CONFIG_RESCTRL_FS */ + /* * MPAM MSCs have the following register layout. See: * Arm Memory System Resource Partitioning and Monitoring (MPAM) System diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c new file mode 100644 index 0000000000000..9a30709704142 --- /dev/null +++ b/drivers/resctrl/mpam_resctrl.c @@ -0,0 +1,324 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2025 Arm Ltd. + +#define pr_fmt(fmt) "%s:%s: " fmt, KBUILD_MODNAME, __func__ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "mpam_internal.h" + +/* + * The classes we've picked to map to resctrl resources, wrapped + * in with their resctrl structure. + * Class pointer may be NULL. + */ +static struct mpam_resctrl_res mpam_resctrl_controls[RDT_NUM_RESOURCES]; + +#define for_each_mpam_resctrl_control(res, rid) \ + for (rid = 0, res = &mpam_resctrl_controls[rid]; \ + rid < RDT_NUM_RESOURCES; \ + rid++, res = &mpam_resctrl_controls[rid]) + +/* The lock for modifying resctrl's domain lists from cpuhp callbacks. */ +static DEFINE_MUTEX(domain_list_lock); + +bool resctrl_arch_alloc_capable(void) +{ + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + + for_each_mpam_resctrl_control(res, rid) { + if (res->resctrl_res.alloc_capable) + return true; + } + + return false; +} + +/* + * MSC may raise an error interrupt if it sees an out or range partid/pmg, + * and go on to truncate the value. Regardless of what the hardware supports, + * only the system wide safe value is safe to use. + */ +u32 resctrl_arch_get_num_closid(struct rdt_resource *ignored) +{ + return mpam_partid_max + 1; +} + +struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) +{ + if (l >= RDT_NUM_RESOURCES) + return NULL; + + return &mpam_resctrl_controls[l].resctrl_res; +} + +static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) +{ + /* TODO: initialise the resctrl resources */ + + return 0; +} + +static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) +{ + struct mpam_class *class = comp->class; + + if (class->type == MPAM_CLASS_CACHE) + return comp->comp_id; + + /* TODO: repaint domain ids to match the L3 domain ids */ + /* Otherwise, expose the ID used by the firmware table code. */ + return comp->comp_id; +} + +static void mpam_resctrl_domain_hdr_init(int cpu, struct mpam_component *comp, + enum resctrl_res_level rid, + struct rdt_domain_hdr *hdr) +{ + lockdep_assert_cpus_held(); + + INIT_LIST_HEAD(&hdr->list); + hdr->id = mpam_resctrl_pick_domain_id(cpu, comp); + hdr->rid = rid; + cpumask_set_cpu(cpu, &hdr->cpu_mask); +} + +static void mpam_resctrl_online_domain_hdr(unsigned int cpu, + struct rdt_domain_hdr *hdr) +{ + lockdep_assert_cpus_held(); + + cpumask_set_cpu(cpu, &hdr->cpu_mask); +} + +/** + * mpam_resctrl_offline_domain_hdr() - Update the domain header to remove a CPU. + * @cpu: The CPU to remove from the domain. + * @hdr: The domain's header. + * + * Removes @cpu from the header mask. If this was the last CPU in the domain, + * the domain header is removed from its parent list and true is returned, + * indicating the parent structure can be freed. + * If there are other CPUs in the domain, returns false. + */ +static bool mpam_resctrl_offline_domain_hdr(unsigned int cpu, + struct rdt_domain_hdr *hdr) +{ + lockdep_assert_held(&domain_list_lock); + + cpumask_clear_cpu(cpu, &hdr->cpu_mask); + if (cpumask_empty(&hdr->cpu_mask)) { + list_del_rcu(&hdr->list); + synchronize_rcu(); + return true; + } + + return false; +} + +static void mpam_resctrl_domain_insert(struct list_head *list, + struct rdt_domain_hdr *new) +{ + struct rdt_domain_hdr *err; + struct list_head *pos = NULL; + + lockdep_assert_held(&domain_list_lock); + + err = resctrl_find_domain(list, new->id, &pos); + if (WARN_ON_ONCE(err)) + return; + + list_add_tail_rcu(&new->list, pos); +} + +static struct mpam_resctrl_dom * +mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) +{ + int err; + struct mpam_resctrl_dom *dom; + struct rdt_ctrl_domain *ctrl_d; + struct mpam_class *class = res->class; + struct mpam_component *comp_iter, *ctrl_comp; + struct rdt_resource *r = &res->resctrl_res; + + lockdep_assert_held(&domain_list_lock); + + ctrl_comp = NULL; + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(comp_iter, &class->components, class_list, + srcu_read_lock_held(&mpam_srcu)) { + if (cpumask_test_cpu(cpu, &comp_iter->affinity)) { + ctrl_comp = comp_iter; + break; + } + } + + /* class has no component for this CPU */ + if (WARN_ON_ONCE(!ctrl_comp)) + return ERR_PTR(-EINVAL); + + dom = kzalloc_node(sizeof(*dom), GFP_KERNEL, cpu_to_node(cpu)); + if (!dom) + return ERR_PTR(-ENOMEM); + + if (r->alloc_capable) { + dom->ctrl_comp = ctrl_comp; + + ctrl_d = &dom->resctrl_ctrl_dom; + mpam_resctrl_domain_hdr_init(cpu, ctrl_comp, r->rid, &ctrl_d->hdr); + ctrl_d->hdr.type = RESCTRL_CTRL_DOMAIN; + err = resctrl_online_ctrl_domain(r, ctrl_d); + if (err) + goto free_domain; + + mpam_resctrl_domain_insert(&r->ctrl_domains, &ctrl_d->hdr); + } else { + pr_debug("Skipped control domain online - no controls\n"); + } + return dom; + +free_domain: + kfree(dom); + dom = ERR_PTR(err); + + return dom; +} + +static struct mpam_resctrl_dom * +mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) +{ + struct mpam_resctrl_dom *dom; + struct rdt_resource *r = &res->resctrl_res; + + lockdep_assert_cpus_held(); + + list_for_each_entry_rcu(dom, &r->ctrl_domains, resctrl_ctrl_dom.hdr.list) { + if (cpumask_test_cpu(cpu, &dom->ctrl_comp->affinity)) + return dom; + } + + return NULL; +} + +int mpam_resctrl_online_cpu(unsigned int cpu) +{ + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + + guard(mutex)(&domain_list_lock); + for_each_mpam_resctrl_control(res, rid) { + struct mpam_resctrl_dom *dom; + struct rdt_resource *r = &res->resctrl_res; + + if (!res->class) + continue; // dummy_resource; + + dom = mpam_resctrl_get_domain_from_cpu(cpu, res); + if (!dom) { + dom = mpam_resctrl_alloc_domain(cpu, res); + if (IS_ERR(dom)) + return PTR_ERR(dom); + } else { + if (r->alloc_capable) { + struct rdt_ctrl_domain *ctrl_d = &dom->resctrl_ctrl_dom; + + mpam_resctrl_online_domain_hdr(cpu, &ctrl_d->hdr); + } + } + } + + resctrl_online_cpu(cpu); + + return 0; +} + +void mpam_resctrl_offline_cpu(unsigned int cpu) +{ + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + + resctrl_offline_cpu(cpu); + + guard(mutex)(&domain_list_lock); + for_each_mpam_resctrl_control(res, rid) { + struct mpam_resctrl_dom *dom; + struct rdt_ctrl_domain *ctrl_d; + bool ctrl_dom_empty; + struct rdt_resource *r = &res->resctrl_res; + + if (!res->class) + continue; // dummy resource + + dom = mpam_resctrl_get_domain_from_cpu(cpu, res); + if (WARN_ON_ONCE(!dom)) + continue; + + if (r->alloc_capable) { + ctrl_d = &dom->resctrl_ctrl_dom; + ctrl_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &ctrl_d->hdr); + if (ctrl_dom_empty) + resctrl_offline_ctrl_domain(&res->resctrl_res, ctrl_d); + } else { + ctrl_dom_empty = true; + } + + if (ctrl_dom_empty) + kfree(dom); + } +} + +int mpam_resctrl_setup(void) +{ + int err = 0; + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + + cpus_read_lock(); + for_each_mpam_resctrl_control(res, rid) { + INIT_LIST_HEAD_RCU(&res->resctrl_res.ctrl_domains); + res->resctrl_res.rid = rid; + } + + /* TODO: pick MPAM classes to map to resctrl resources */ + + /* Initialise the resctrl structures from the classes */ + for_each_mpam_resctrl_control(res, rid) { + if (!res->class) + continue; // dummy resource + + err = mpam_resctrl_control_init(res); + if (err) { + pr_debug("Failed to initialise rid %u\n", rid); + break; + } + } + cpus_read_unlock(); + + if (err) { + pr_debug("Internal error %d - resctrl not supported\n", err); + return err; + } + + if (!resctrl_arch_alloc_capable()) { + pr_debug("No alloc(%u) found - resctrl not supported\n", + resctrl_arch_alloc_capable()); + return -EOPNOTSUPP; + } + + /* TODO: call resctrl_init() */ + + return 0; +} diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 7f00c5285a326..2c7d1413a401f 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -49,6 +49,9 @@ static inline int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, } #endif +bool resctrl_arch_alloc_capable(void); +bool resctrl_arch_mon_capable(void); + /** * mpam_register_requestor() - Register a requestor with the MPAM driver * @partid_max: The maximum PARTID value the requestor can generate. From ff05c9fa8b00a1829c712c353a02af131dab4b81 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:52 +0000 Subject: [PATCH 290/464] arm_mpam: resctrl: Pick the caches we will use as resctrl resources BugLink: https://bugs.launchpad.net/bugs/2154527 Systems with MPAM support may have a variety of control types at any point of their system layout. We can only expose certain types of control, and only if they exist at particular locations. Start with the well-known caches. These have to be depth 2 or 3 and support MPAM's cache portion bitmap controls, with a number of portions fewer than resctrl's limit. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 52a4edb16121d07734e4e392767d26d286f08c35) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 91 +++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 2 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 9a30709704142..65bb670dc3fb1 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -65,9 +65,95 @@ struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) return &mpam_resctrl_controls[l].resctrl_res; } +static bool cache_has_usable_cpor(struct mpam_class *class) +{ + struct mpam_props *cprops = &class->props; + + if (!mpam_has_feature(mpam_feat_cpor_part, cprops)) + return false; + + /* resctrl uses u32 for all bitmap configurations */ + return class->props.cpbm_wd <= 32; +} + +/* Test whether we can export MPAM_CLASS_CACHE:{2,3}? */ +static void mpam_resctrl_pick_caches(void) +{ + struct mpam_class *class; + struct mpam_resctrl_res *res; + + lockdep_assert_cpus_held(); + + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(class, &mpam_classes, classes_list, + srcu_read_lock_held(&mpam_srcu)) { + if (class->type != MPAM_CLASS_CACHE) { + pr_debug("class %u is not a cache\n", class->level); + continue; + } + + if (class->level != 2 && class->level != 3) { + pr_debug("class %u is not L2 or L3\n", class->level); + continue; + } + + if (!cache_has_usable_cpor(class)) { + pr_debug("class %u cache misses CPOR\n", class->level); + continue; + } + + if (!cpumask_equal(&class->affinity, cpu_possible_mask)) { + pr_debug("class %u has missing CPUs, mask %*pb != %*pb\n", class->level, + cpumask_pr_args(&class->affinity), + cpumask_pr_args(cpu_possible_mask)); + continue; + } + + if (class->level == 2) + res = &mpam_resctrl_controls[RDT_RESOURCE_L2]; + else + res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + res->class = class; + } +} + static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) { - /* TODO: initialise the resctrl resources */ + struct mpam_class *class = res->class; + struct rdt_resource *r = &res->resctrl_res; + + switch (r->rid) { + case RDT_RESOURCE_L2: + case RDT_RESOURCE_L3: + r->schema_fmt = RESCTRL_SCHEMA_BITMAP; + r->cache.arch_has_sparse_bitmasks = true; + + r->cache.cbm_len = class->props.cpbm_wd; + /* mpam_devices will reject empty bitmaps */ + r->cache.min_cbm_bits = 1; + + if (r->rid == RDT_RESOURCE_L2) { + r->name = "L2"; + r->ctrl_scope = RESCTRL_L2_CACHE; + r->cdp_capable = true; + } else { + r->name = "L3"; + r->ctrl_scope = RESCTRL_L3_CACHE; + r->cdp_capable = true; + } + + /* + * Which bits are shared with other ...things... Unknown + * devices use partid-0 which uses all the bitmap fields. Until + * we have configured the SMMU and GIC not to do this 'all the + * bits' is the correct answer here. + */ + r->cache.shareable_bits = resctrl_get_default_ctrl(r); + r->alloc_capable = true; + break; + default: + return -EINVAL; + } return 0; } @@ -292,7 +378,8 @@ int mpam_resctrl_setup(void) res->resctrl_res.rid = rid; } - /* TODO: pick MPAM classes to map to resctrl resources */ + /* Find some classes to use for controls */ + mpam_resctrl_pick_caches(); /* Initialise the resctrl structures from the classes */ for_each_mpam_resctrl_control(res, rid) { From eb2bfa639633f491c9e7eaa35d473217d020556b Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:53 +0000 Subject: [PATCH 291/464] arm_mpam: resctrl: Implement resctrl_arch_reset_all_ctrls() BugLink: https://bugs.launchpad.net/bugs/2154527 We already have a helper for resetting an mpam class and component. Hook it up to resctrl_arch_reset_all_ctrls() and the domain offline path. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Shaopeng Tan Reviewed-by: Zeng Heng Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 370d166d878d0c0aa06568d67387a1151a200501) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 2 +- drivers/resctrl/mpam_internal.h | 3 +++ drivers/resctrl/mpam_resctrl.c | 13 +++++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 699b3361a5584..f47d1354d10b1 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -2552,7 +2552,7 @@ static void mpam_reset_component_locked(struct mpam_component *comp) } } -static void mpam_reset_class_locked(struct mpam_class *class) +void mpam_reset_class_locked(struct mpam_class *class) { struct mpam_component *comp; diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 11071d005d85d..3ddd921d34051 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -386,6 +386,9 @@ extern u8 mpam_pmg_max; void mpam_enable(struct work_struct *work); void mpam_disable(struct work_struct *work); +/* Reset all the RIS in a class under cpus_read_lock() */ +void mpam_reset_class_locked(struct mpam_class *class); + int mpam_apply_config(struct mpam_component *comp, u16 partid, struct mpam_config *cfg); diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 65bb670dc3fb1..b2217d11561d8 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -170,6 +170,19 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) return comp->comp_id; } +void resctrl_arch_reset_all_ctrls(struct rdt_resource *r) +{ + struct mpam_resctrl_res *res; + + lockdep_assert_cpus_held(); + + if (!mpam_is_enabled()) + return; + + res = container_of(r, struct mpam_resctrl_res, resctrl_res); + mpam_reset_class_locked(res->class); +} + static void mpam_resctrl_domain_hdr_init(int cpu, struct mpam_component *comp, enum resctrl_res_level rid, struct rdt_domain_hdr *hdr) From 5ed5425bf8762d6653a1124797b625fafdfbdf3f Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:54 +0000 Subject: [PATCH 292/464] arm_mpam: resctrl: Add resctrl_arch_get_config() BugLink: https://bugs.launchpad.net/bugs/2154527 Implement resctrl_arch_get_config() by testing the live configuration for a CPOR bitmap. For any other configuration type return the default. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 02cc661687886563a0e08ecee51c5ef7d1737237) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 43 ++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index b2217d11561d8..3af57b6f2c1b5 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -170,6 +170,49 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) return comp->comp_id; } +u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, + u32 closid, enum resctrl_conf_type type) +{ + u32 partid; + struct mpam_config *cfg; + struct mpam_props *cprops; + struct mpam_resctrl_res *res; + struct mpam_resctrl_dom *dom; + enum mpam_device_features configured_by; + + lockdep_assert_cpus_held(); + + if (!mpam_is_enabled()) + return resctrl_get_default_ctrl(r); + + res = container_of(r, struct mpam_resctrl_res, resctrl_res); + dom = container_of(d, struct mpam_resctrl_dom, resctrl_ctrl_dom); + cprops = &res->class->props; + + partid = resctrl_get_config_index(closid, type); + cfg = &dom->ctrl_comp->cfg[partid]; + + switch (r->rid) { + case RDT_RESOURCE_L2: + case RDT_RESOURCE_L3: + configured_by = mpam_feat_cpor_part; + break; + default: + return resctrl_get_default_ctrl(r); + } + + if (!r->alloc_capable || partid >= resctrl_arch_get_num_closid(r) || + !mpam_has_feature(configured_by, cfg)) + return resctrl_get_default_ctrl(r); + + switch (configured_by) { + case mpam_feat_cpor_part: + return cfg->cpbm; + default: + return resctrl_get_default_ctrl(r); + } +} + void resctrl_arch_reset_all_ctrls(struct rdt_resource *r) { struct mpam_resctrl_res *res; From f70299dd0b56cb3c9674266f6575e5d62aa238e1 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:55 +0000 Subject: [PATCH 293/464] arm_mpam: resctrl: Implement helpers to update configuration BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl has two helpers for updating the configuration. resctrl_arch_update_one() updates a single value, and is used by the software-controller to apply feedback to the bandwidth controls, it has to be called on one of the CPUs in the resctrl:domain. resctrl_arch_update_domains() copies multiple staged configurations, it can be called from anywhere. Both helpers should update any changes to the underlying hardware. Implement resctrl_arch_update_domains() to use resctrl_arch_update_one(). Neither need to be called on a specific CPU as the mpam driver will send IPIs as needed. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 9cd2b522be2cc64fab179d75537d2e8df38d26a6) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 70 ++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 3af57b6f2c1b5..ea60777934ffd 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -213,6 +213,76 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, } } +int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, + u32 closid, enum resctrl_conf_type t, u32 cfg_val) +{ + u32 partid; + struct mpam_config cfg; + struct mpam_props *cprops; + struct mpam_resctrl_res *res; + struct mpam_resctrl_dom *dom; + + lockdep_assert_cpus_held(); + lockdep_assert_irqs_enabled(); + + /* + * No need to check the CPU as mpam_apply_config() doesn't care, and + * resctrl_arch_update_domains() relies on this. + */ + res = container_of(r, struct mpam_resctrl_res, resctrl_res); + dom = container_of(d, struct mpam_resctrl_dom, resctrl_ctrl_dom); + cprops = &res->class->props; + + partid = resctrl_get_config_index(closid, t); + if (!r->alloc_capable || partid >= resctrl_arch_get_num_closid(r)) { + pr_debug("Not alloc capable or computed PARTID out of range\n"); + return -EINVAL; + } + + /* + * Copy the current config to avoid clearing other resources when the + * same component is exposed multiple times through resctrl. + */ + cfg = dom->ctrl_comp->cfg[partid]; + + switch (r->rid) { + case RDT_RESOURCE_L2: + case RDT_RESOURCE_L3: + cfg.cpbm = cfg_val; + mpam_set_feature(mpam_feat_cpor_part, &cfg); + break; + default: + return -EINVAL; + } + + return mpam_apply_config(dom->ctrl_comp, partid, &cfg); +} + +int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid) +{ + int err; + struct rdt_ctrl_domain *d; + + lockdep_assert_cpus_held(); + lockdep_assert_irqs_enabled(); + + list_for_each_entry_rcu(d, &r->ctrl_domains, hdr.list) { + for (enum resctrl_conf_type t = 0; t < CDP_NUM_TYPES; t++) { + struct resctrl_staged_config *cfg = &d->staged_config[t]; + + if (!cfg->have_new_ctrl) + continue; + + err = resctrl_arch_update_one(r, d, closid, t, + cfg->new_ctrl); + if (err) + return err; + } + } + + return 0; +} + void resctrl_arch_reset_all_ctrls(struct rdt_resource *r) { struct mpam_resctrl_res *res; From 32119d10b5ddc4eb489bfe0fb8876553f0193564 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:56 +0000 Subject: [PATCH 294/464] arm_mpam: resctrl: Add plumbing against arm64 task and cpu hooks BugLink: https://bugs.launchpad.net/bugs/2154527 arm64 provides helpers for changing a task's and a cpu's mpam partid/pmg values. These are used to back a number of resctrl_arch_ functions. Connect them up. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 9d2e1a99fae58ce992f147bdf83b5d9089f70b27) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 58 ++++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 5 +++ 2 files changed, 63 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index ea60777934ffd..9cde5b7e644cc 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,8 @@ static struct mpam_resctrl_res mpam_resctrl_controls[RDT_NUM_RESOURCES]; /* The lock for modifying resctrl's domain lists from cpuhp callbacks. */ static DEFINE_MUTEX(domain_list_lock); +static bool cdp_enabled; + bool resctrl_arch_alloc_capable(void) { struct mpam_resctrl_res *res; @@ -57,6 +60,61 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *ignored) return mpam_partid_max + 1; } +void resctrl_arch_sched_in(struct task_struct *tsk) +{ + lockdep_assert_preemption_disabled(); + + mpam_thread_switch(tsk); +} + +void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid, u32 rmid) +{ + WARN_ON_ONCE(closid > U16_MAX); + WARN_ON_ONCE(rmid > U8_MAX); + + if (!cdp_enabled) { + mpam_set_cpu_defaults(cpu, closid, closid, rmid, rmid); + } else { + /* + * When CDP is enabled, resctrl halves the closid range and we + * use odd/even partid for one closid. + */ + u32 partid_d = resctrl_get_config_index(closid, CDP_DATA); + u32 partid_i = resctrl_get_config_index(closid, CDP_CODE); + + mpam_set_cpu_defaults(cpu, partid_d, partid_i, rmid, rmid); + } +} + +void resctrl_arch_sync_cpu_closid_rmid(void *info) +{ + struct resctrl_cpu_defaults *r = info; + + lockdep_assert_preemption_disabled(); + + if (r) { + resctrl_arch_set_cpu_default_closid_rmid(smp_processor_id(), + r->closid, r->rmid); + } + + resctrl_arch_sched_in(current); +} + +void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid) +{ + WARN_ON_ONCE(closid > U16_MAX); + WARN_ON_ONCE(rmid > U8_MAX); + + if (!cdp_enabled) { + mpam_set_task_partid_pmg(tsk, closid, closid, rmid, rmid); + } else { + u32 partid_d = resctrl_get_config_index(closid, CDP_DATA); + u32 partid_i = resctrl_get_config_index(closid, CDP_CODE); + + mpam_set_task_partid_pmg(tsk, partid_d, partid_i, rmid, rmid); + } +} + struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) { if (l >= RDT_NUM_RESOURCES) diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 2c7d1413a401f..5a78299ec464b 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -52,6 +52,11 @@ static inline int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, bool resctrl_arch_alloc_capable(void); bool resctrl_arch_mon_capable(void); +void resctrl_arch_set_cpu_default_closid(int cpu, u32 closid); +void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid); +void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid, u32 rmid); +void resctrl_arch_sched_in(struct task_struct *tsk); + /** * mpam_register_requestor() - Register a requestor with the MPAM driver * @partid_max: The maximum PARTID value the requestor can generate. From 29a87f3b6dc604218990fbb95cdbc3d60aec9557 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:45:57 +0000 Subject: [PATCH 295/464] arm_mpam: resctrl: Add CDP emulation BugLink: https://bugs.launchpad.net/bugs/2154527 Intel RDT's CDP feature allows the cache to use a different control value depending on whether the accesses was for instruction fetch or a data access. MPAM's equivalent feature is the other way up: the CPU assigns a different partid label to traffic depending on whether it was instruction fetch or a data access, which causes the cache to use a different control value based solely on the partid. MPAM can emulate CDP, with the side effect that the alternative partid is seen by all MSC, it can't be enabled per-MSC. Add the resctrl hooks to turn this on or off. Add the helpers that match a closid against a task, which need to be aware that the value written to hardware is not the same as the one resctrl is using. Update the 'arm64_mpam_global_default' variable the arch code uses during context switch to know when the per-cpu value should be used instead. Also, update these per-cpu values and sync the resulting mpam partid/pmg configuration to hardware. resctrl can enable CDP for L2 caches, L3 caches or both. When it is enabled by one and not the other MPAM globally enabled CDP but hides the effect on the other cache resource. This hiding is possible as CPOR is the only supported cache control and that uses a resource bitmap; two partids with the same bitmap act as one. Awkwardly, the MB controls don't implement CDP and CDP can't be hidden as the memory bandwidth control is a maximum per partid which can't be modelled with more partids. If the total maximum is used for both the data and instruction partids then then the maximum may be exceeded and if it is split in two then the one using more bandwidth will hit a lower limit. Hence, hide the MB controls completely if CDP is enabled for any resource. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Cc: Dave Martin Cc: Amit Singh Tomar Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 6789fb99282c0a8e8e84701b7edf456f4a9e71e2) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/include/asm/mpam.h | 1 + drivers/resctrl/mpam_internal.h | 1 + drivers/resctrl/mpam_resctrl.c | 122 ++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 2 + 4 files changed, 126 insertions(+) diff --git a/arch/arm64/include/asm/mpam.h b/arch/arm64/include/asm/mpam.h index 05aa71200f61a..70d396e7b6da8 100644 --- a/arch/arm64/include/asm/mpam.h +++ b/arch/arm64/include/asm/mpam.h @@ -4,6 +4,7 @@ #ifndef __ASM__MPAM_H #define __ASM__MPAM_H +#include #include #include #include diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 3ddd921d34051..42ee024701650 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -340,6 +340,7 @@ struct mpam_resctrl_dom { struct mpam_resctrl_res { struct mpam_class *class; struct rdt_resource resctrl_res; + bool cdp_enabled; }; static inline int mpam_alloc_csu_mon(struct mpam_class *class) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 9cde5b7e644cc..2111542f485e1 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -35,6 +35,10 @@ static struct mpam_resctrl_res mpam_resctrl_controls[RDT_NUM_RESOURCES]; /* The lock for modifying resctrl's domain lists from cpuhp callbacks. */ static DEFINE_MUTEX(domain_list_lock); +/* + * MPAM emulates CDP by setting different PARTID in the I/D fields of MPAM0_EL1. + * This applies globally to all traffic the CPU generates. + */ static bool cdp_enabled; bool resctrl_arch_alloc_capable(void) @@ -50,6 +54,74 @@ bool resctrl_arch_alloc_capable(void) return false; } +bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level rid) +{ + return mpam_resctrl_controls[rid].cdp_enabled; +} + +/** + * resctrl_reset_task_closids() - Reset the PARTID/PMG values for all tasks. + * + * At boot, all existing tasks use partid zero for D and I. + * To enable/disable CDP emulation, all these tasks need relabelling. + */ +static void resctrl_reset_task_closids(void) +{ + struct task_struct *p, *t; + + read_lock(&tasklist_lock); + for_each_process_thread(p, t) { + resctrl_arch_set_closid_rmid(t, RESCTRL_RESERVED_CLOSID, + RESCTRL_RESERVED_RMID); + } + read_unlock(&tasklist_lock); +} + +int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) +{ + u32 partid_i = RESCTRL_RESERVED_CLOSID, partid_d = RESCTRL_RESERVED_CLOSID; + int cpu; + + /* + * resctrl_arch_set_cdp_enabled() is only called with enable set to + * false on error and unmount. + */ + cdp_enabled = enable; + mpam_resctrl_controls[rid].cdp_enabled = enable; + + /* The mbw_max feature can't hide cdp as it's a per-partid maximum. */ + if (cdp_enabled && !mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled) + mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = false; + + if (mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled && + mpam_resctrl_controls[RDT_RESOURCE_MBA].class) + mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = true; + + if (enable) { + if (mpam_partid_max < 1) + return -EINVAL; + + partid_d = resctrl_get_config_index(RESCTRL_RESERVED_CLOSID, CDP_DATA); + partid_i = resctrl_get_config_index(RESCTRL_RESERVED_CLOSID, CDP_CODE); + } + + mpam_set_task_partid_pmg(current, partid_d, partid_i, 0, 0); + WRITE_ONCE(arm64_mpam_global_default, mpam_get_regval(current)); + + resctrl_reset_task_closids(); + + for_each_possible_cpu(cpu) + mpam_set_cpu_defaults(cpu, partid_d, partid_i, 0, 0); + on_each_cpu(resctrl_arch_sync_cpu_closid_rmid, NULL, 1); + + return 0; +} + +static bool mpam_resctrl_hide_cdp(enum resctrl_res_level rid) +{ + return cdp_enabled && !resctrl_arch_get_cdp_enabled(rid); +} + /* * MSC may raise an error interrupt if it sees an out or range partid/pmg, * and go on to truncate the value. Regardless of what the hardware supports, @@ -115,6 +187,30 @@ void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid) } } +bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid) +{ + u64 regval = mpam_get_regval(tsk); + u32 tsk_closid = FIELD_GET(MPAM0_EL1_PARTID_D, regval); + + if (cdp_enabled) + tsk_closid >>= 1; + + return tsk_closid == closid; +} + +/* The task's pmg is not unique, the partid must be considered too */ +bool resctrl_arch_match_rmid(struct task_struct *tsk, u32 closid, u32 rmid) +{ + u64 regval = mpam_get_regval(tsk); + u32 tsk_closid = FIELD_GET(MPAM0_EL1_PARTID_D, regval); + u32 tsk_rmid = FIELD_GET(MPAM0_EL1_PMG_D, regval); + + if (cdp_enabled) + tsk_closid >>= 1; + + return (tsk_closid == closid) && (tsk_rmid == rmid); +} + struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) { if (l >= RDT_NUM_RESOURCES) @@ -247,6 +343,14 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, dom = container_of(d, struct mpam_resctrl_dom, resctrl_ctrl_dom); cprops = &res->class->props; + /* + * When CDP is enabled, but the resource doesn't support it, + * the control is cloned across both partids. + * Pick one at random to read: + */ + if (mpam_resctrl_hide_cdp(r->rid)) + type = CDP_DATA; + partid = resctrl_get_config_index(closid, type); cfg = &dom->ctrl_comp->cfg[partid]; @@ -274,6 +378,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, u32 closid, enum resctrl_conf_type t, u32 cfg_val) { + int err; u32 partid; struct mpam_config cfg; struct mpam_props *cprops; @@ -291,6 +396,9 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, dom = container_of(d, struct mpam_resctrl_dom, resctrl_ctrl_dom); cprops = &res->class->props; + if (mpam_resctrl_hide_cdp(r->rid)) + t = CDP_DATA; + partid = resctrl_get_config_index(closid, t); if (!r->alloc_capable || partid >= resctrl_arch_get_num_closid(r)) { pr_debug("Not alloc capable or computed PARTID out of range\n"); @@ -313,6 +421,20 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, return -EINVAL; } + /* + * When CDP is enabled, but the resource doesn't support it, we need to + * apply the same configuration to the other partid. + */ + if (mpam_resctrl_hide_cdp(r->rid)) { + partid = resctrl_get_config_index(closid, CDP_CODE); + err = mpam_apply_config(dom->ctrl_comp, partid, &cfg); + if (err) + return err; + + partid = resctrl_get_config_index(closid, CDP_DATA); + return mpam_apply_config(dom->ctrl_comp, partid, &cfg); + } + return mpam_apply_config(dom->ctrl_comp, partid, &cfg); } diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 5a78299ec464b..d329b1dc148ba 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -56,6 +56,8 @@ void resctrl_arch_set_cpu_default_closid(int cpu, u32 closid); void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid); void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid, u32 rmid); void resctrl_arch_sched_in(struct task_struct *tsk); +bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid); +bool resctrl_arch_match_rmid(struct task_struct *tsk, u32 closid, u32 rmid); /** * mpam_register_requestor() - Register a requestor with the MPAM driver From 8de86df2417628f2f377bb6a985e01da60f93e67 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:45:58 +0000 Subject: [PATCH 296/464] arm_mpam: resctrl: Hide CDP emulation behind CONFIG_EXPERT BugLink: https://bugs.launchpad.net/bugs/2154527 When CDP is not enabled, the 'rmid_entry's in the limbo list, rmid_busy_llc, map directly to a (PARTID,PMG) pair and when CDP is enabled the mapping is to two different pairs. As the limbo list is reused between mounts and CDP disabled on unmount this can lead to stale mapping and the limbo handler will then make monitor reads with potentially out of range PARTID. This may then cause an MPAM error interrupt and the driver will disable MPAM. No problems are expected if you just mount the resctrl file system once with CDP enabled and never unmount it. Hide CDP emulation behind CONFIG_EXPERT to protect the unwary. Signed-off-by: Ben Horgan Reviewed-by: Gavin Shan Reviewed-by: Zeng Heng Reviewed-by: James Morse Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Signed-off-by: James Morse (cherry picked from commit 01a0021f6c39557037bfc41ede7230a0696677ff) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 2111542f485e1..2331e6ddb814b 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -82,6 +82,18 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) u32 partid_i = RESCTRL_RESERVED_CLOSID, partid_d = RESCTRL_RESERVED_CLOSID; int cpu; + if (!IS_ENABLED(CONFIG_EXPERT) && enable) { + /* + * If the resctrl fs is mounted more than once, sequentially, + * then CDP can lead to the use of out of range PARTIDs. + */ + pr_warn("CDP not supported\n"); + return -EOPNOTSUPP; + } + + if (enable) + pr_warn("CDP is an expert feature and may cause MPAM to malfunction.\n"); + /* * resctrl_arch_set_cdp_enabled() is only called with enable set to * false on error and unmount. From f4695db5e0b524f6fda63f25ea6dd81c07ff41f9 Mon Sep 17 00:00:00 2001 From: Dave Martin Date: Fri, 13 Mar 2026 14:45:59 +0000 Subject: [PATCH 297/464] arm_mpam: resctrl: Convert to/from MPAMs fixed-point formats BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM uses a fixed-point formats for some hardware controls. Resctrl provides the bandwidth controls as a percentage. Add helpers to convert between these. Ensure bwa_wd is at most 16 to make it clear higher values have no meaning. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Signed-off-by: Dave Martin Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 80d147d293130ee3c8a395cbbea1813e26ab9a1b) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 7 +++++ drivers/resctrl/mpam_resctrl.c | 51 ++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index f47d1354d10b1..e4a52b79d0bf1 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -725,6 +725,13 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) mpam_set_feature(mpam_feat_mbw_part, props); props->bwa_wd = FIELD_GET(MPAMF_MBW_IDR_BWA_WD, mbw_features); + + /* + * The BWA_WD field can represent 0-63, but the control fields it + * describes have a maximum of 16 bits. + */ + props->bwa_wd = min(props->bwa_wd, 16); + if (props->bwa_wd && FIELD_GET(MPAMF_MBW_IDR_HAS_MAX, mbw_features)) mpam_set_feature(mpam_feat_mbw_max, props); diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 2331e6ddb814b..240a06df2f079 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -242,6 +243,56 @@ static bool cache_has_usable_cpor(struct mpam_class *class) return class->props.cpbm_wd <= 32; } +/* + * Each fixed-point hardware value architecturally represents a range + * of values: the full range 0% - 100% is split contiguously into + * (1 << cprops->bwa_wd) equal bands. + * + * Although the bwa_bwd fields have 6 bits the maximum valid value is 16 + * as it reports the width of fields that are at most 16 bits. When + * fewer than 16 bits are valid the least significant bits are + * ignored. The implied binary point is kept between bits 15 and 16 and + * so the valid bits are leftmost. + * + * See ARM IHI0099B.a "MPAM system component specification", Section 9.3, + * "The fixed-point fractional format" for more information. + * + * Find the nearest percentage value to the upper bound of the selected band: + */ +static u32 mbw_max_to_percent(u16 mbw_max, struct mpam_props *cprops) +{ + u32 val = mbw_max; + + val >>= 16 - cprops->bwa_wd; + val += 1; + val *= MAX_MBA_BW; + val = DIV_ROUND_CLOSEST(val, 1 << cprops->bwa_wd); + + return val; +} + +/* + * Find the band whose upper bound is closest to the specified percentage. + * + * A round-to-nearest policy is followed here as a balanced compromise + * between unexpected under-commit of the resource (where the total of + * a set of resource allocations after conversion is less than the + * expected total, due to rounding of the individual converted + * percentages) and over-commit (where the total of the converted + * allocations is greater than expected). + */ +static u16 percent_to_mbw_max(u8 pc, struct mpam_props *cprops) +{ + u32 val = pc; + + val <<= cprops->bwa_wd; + val = DIV_ROUND_CLOSEST(val, MAX_MBA_BW); + val = max(val, 1) - 1; + val <<= 16 - cprops->bwa_wd; + + return val; +} + /* Test whether we can export MPAM_CLASS_CACHE:{2,3}? */ static void mpam_resctrl_pick_caches(void) { From 9c011b552f1c10f311888a253041b0bf209cc564 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:46:00 +0000 Subject: [PATCH 298/464] arm_mpam: resctrl: Add rmid index helpers BugLink: https://bugs.launchpad.net/bugs/2154527 Because MPAM's pmg aren't identical to RDT's rmid, resctrl handles some data structures by index. This allows x86 to map indexes to RMID, and MPAM to map them to partid-and-pmg. Add the helpers to do this. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Suggested-by: James Morse Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 3e9b35823aabcb85cc039960256426e50f1fd601) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 16 ++++++++++++++++ include/linux/arm_mpam.h | 3 +++ 2 files changed, 19 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 240a06df2f079..370830ab11197 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -145,6 +145,22 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *ignored) return mpam_partid_max + 1; } +u32 resctrl_arch_system_num_rmid_idx(void) +{ + return (mpam_pmg_max + 1) * (mpam_partid_max + 1); +} + +u32 resctrl_arch_rmid_idx_encode(u32 closid, u32 rmid) +{ + return closid * (mpam_pmg_max + 1) + rmid; +} + +void resctrl_arch_rmid_idx_decode(u32 idx, u32 *closid, u32 *rmid) +{ + *closid = idx / (mpam_pmg_max + 1); + *rmid = idx % (mpam_pmg_max + 1); +} + void resctrl_arch_sched_in(struct task_struct *tsk) { lockdep_assert_preemption_disabled(); diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index d329b1dc148ba..7d23c90f077dc 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -58,6 +58,9 @@ void resctrl_arch_set_cpu_default_closid_rmid(int cpu, u32 closid, u32 rmid); void resctrl_arch_sched_in(struct task_struct *tsk); bool resctrl_arch_match_closid(struct task_struct *tsk, u32 closid); bool resctrl_arch_match_rmid(struct task_struct *tsk, u32 closid, u32 rmid); +u32 resctrl_arch_rmid_idx_encode(u32 closid, u32 rmid); +void resctrl_arch_rmid_idx_decode(u32 idx, u32 *closid, u32 *rmid); +u32 resctrl_arch_system_num_rmid_idx(void); /** * mpam_register_requestor() - Register a requestor with the MPAM driver From 61692c79c2104e7f6089af73bcf63e736bc83a80 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:46:01 +0000 Subject: [PATCH 299/464] arm_mpam: resctrl: Wait for cacheinfo to be ready BugLink: https://bugs.launchpad.net/bugs/2154527 In order to calculate the rmid realloc threshold the size of the cache needs to be known. Cache domains will also be named after the cache id. So that this information can be extracted from cacheinfo we need to wait for it to be ready. The cacheinfo information is populated in device_initcall() so we wait for that. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 1c1e2968a860c5af9fca67f1c0e88aab83ace0b3) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 370830ab11197..bf91cff05daf7 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -16,6 +16,7 @@ #include #include #include +#include #include @@ -42,6 +43,13 @@ static DEFINE_MUTEX(domain_list_lock); */ static bool cdp_enabled; +/* + * We use cacheinfo to discover the size of the caches and their id. cacheinfo + * populates this from a device_initcall(). mpam_resctrl_setup() must wait. + */ +static bool cacheinfo_ready; +static DECLARE_WAIT_QUEUE_HEAD(wait_cacheinfo_ready); + bool resctrl_arch_alloc_capable(void) { struct mpam_resctrl_res *res; @@ -757,6 +765,8 @@ int mpam_resctrl_setup(void) struct mpam_resctrl_res *res; enum resctrl_res_level rid; + wait_event(wait_cacheinfo_ready, cacheinfo_ready); + cpus_read_lock(); for_each_mpam_resctrl_control(res, rid) { INIT_LIST_HEAD_RCU(&res->resctrl_res.ctrl_domains); @@ -794,3 +804,12 @@ int mpam_resctrl_setup(void) return 0; } + +static int __init __cacheinfo_ready(void) +{ + cacheinfo_ready = true; + wake_up(&wait_cacheinfo_ready); + + return 0; +} +device_initcall_sync(__cacheinfo_ready); From 3592ea2037bdd7d337025da36a264fbccf45130b Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:02 +0000 Subject: [PATCH 300/464] arm_mpam: resctrl: Add support for 'MB' resource BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl supports 'MB', as a percentage throttling of traffic from the L3. This is the control that mba_sc uses, so ideally the class chosen should be as close as possible to the counters used for mbm_total. If there is a single L3, it's the last cache, and the topology of the memory matches then the traffic at the memory controller will be equivalent to that at egress of the L3. If these conditions are met allow the memory class to back MB. MB's percentage control should be backed either with the fixed point fraction MBW_MAX or bandwidth portion bitmaps. The bandwidth portion bitmaps is not used as its tricky to pick which bits to use to avoid contention, and may be possible to expose this as something other than a percentage in the future. Tested-by: Shaopeng Tan Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Gavin Shan Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Dave Martin Signed-off-by: Dave Martin Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 36528c7681b8093f5f9270d2af7c4326d771f181) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 281 ++++++++++++++++++++++++++++++++- 1 file changed, 280 insertions(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index bf91cff05daf7..60d111f7abfd5 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -267,6 +267,33 @@ static bool cache_has_usable_cpor(struct mpam_class *class) return class->props.cpbm_wd <= 32; } +static bool mba_class_use_mbw_max(struct mpam_props *cprops) +{ + return (mpam_has_feature(mpam_feat_mbw_max, cprops) && + cprops->bwa_wd); +} + +static bool class_has_usable_mba(struct mpam_props *cprops) +{ + return mba_class_use_mbw_max(cprops); +} + +/* + * Calculate the worst-case percentage change from each implemented step + * in the control. + */ +static u32 get_mba_granularity(struct mpam_props *cprops) +{ + if (!mba_class_use_mbw_max(cprops)) + return 0; + + /* + * bwa_wd is the number of bits implemented in the 0.xxx + * fixed point fraction. 1 bit is 50%, 2 is 25% etc. + */ + return DIV_ROUND_UP(MAX_MBA_BW, 1 << cprops->bwa_wd); +} + /* * Each fixed-point hardware value architecturally represents a range * of values: the full range 0% - 100% is split contiguously into @@ -317,6 +344,160 @@ static u16 percent_to_mbw_max(u8 pc, struct mpam_props *cprops) return val; } +static u32 get_mba_min(struct mpam_props *cprops) +{ + if (!mba_class_use_mbw_max(cprops)) { + WARN_ON_ONCE(1); + return 0; + } + + return mbw_max_to_percent(0, cprops); +} + +/* Find the L3 cache that has affinity with this CPU */ +static int find_l3_equivalent_bitmask(int cpu, cpumask_var_t tmp_cpumask) +{ + u32 cache_id = get_cpu_cacheinfo_id(cpu, 3); + + lockdep_assert_cpus_held(); + + return mpam_get_cpumask_from_cache_id(cache_id, 3, tmp_cpumask); +} + +/* + * topology_matches_l3() - Is the provided class the same shape as L3 + * @victim: The class we'd like to pretend is L3. + * + * resctrl expects all the world's a Xeon, and all counters are on the + * L3. We allow some mapping counters on other classes. This requires + * that the CPU->domain mapping is the same kind of shape. + * + * Using cacheinfo directly would make this work even if resctrl can't + * use the L3 - but cacheinfo can't tell us anything about offline CPUs. + * Using the L3 resctrl domain list also depends on CPUs being online. + * Using the mpam_class we picked for L3 so we can use its domain list + * assumes that there are MPAM controls on the L3. + * Instead, this path eventually uses the mpam_get_cpumask_from_cache_id() + * helper which can tell us about offline CPUs ... but getting the cache_id + * to start with relies on at least one CPU per L3 cache being online at + * boot. + * + * Walk the victim component list and compare the affinity mask with the + * corresponding L3. The topology matches if each victim:component's affinity + * mask is the same as the CPU's corresponding L3's. These lists/masks are + * computed from firmware tables so don't change at runtime. + */ +static bool topology_matches_l3(struct mpam_class *victim) +{ + int cpu, err; + struct mpam_component *victim_iter; + + lockdep_assert_cpus_held(); + + cpumask_var_t __free(free_cpumask_var) tmp_cpumask = CPUMASK_VAR_NULL; + if (!alloc_cpumask_var(&tmp_cpumask, GFP_KERNEL)) + return false; + + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(victim_iter, &victim->components, class_list, + srcu_read_lock_held(&mpam_srcu)) { + if (cpumask_empty(&victim_iter->affinity)) { + pr_debug("class %u has CPU-less component %u - can't match L3!\n", + victim->level, victim_iter->comp_id); + return false; + } + + cpu = cpumask_any_and(&victim_iter->affinity, cpu_online_mask); + if (WARN_ON_ONCE(cpu >= nr_cpu_ids)) + return false; + + cpumask_clear(tmp_cpumask); + err = find_l3_equivalent_bitmask(cpu, tmp_cpumask); + if (err) { + pr_debug("Failed to find L3's equivalent component to class %u component %u\n", + victim->level, victim_iter->comp_id); + return false; + } + + /* Any differing bits in the affinity mask? */ + if (!cpumask_equal(tmp_cpumask, &victim_iter->affinity)) { + pr_debug("class %u component %u has Mismatched CPU mask with L3 equivalent\n" + "L3:%*pbl != victim:%*pbl\n", + victim->level, victim_iter->comp_id, + cpumask_pr_args(tmp_cpumask), + cpumask_pr_args(&victim_iter->affinity)); + + return false; + } + } + + return true; +} + +/* + * Test if the traffic for a class matches that at egress from the L3. For + * MSC at memory controllers this is only possible if there is a single L3 + * as otherwise the counters at the memory can include bandwidth from the + * non-local L3. + */ +static bool traffic_matches_l3(struct mpam_class *class) +{ + int err, cpu; + + lockdep_assert_cpus_held(); + + if (class->type == MPAM_CLASS_CACHE && class->level == 3) + return true; + + if (class->type == MPAM_CLASS_CACHE && class->level != 3) { + pr_debug("class %u is a different cache from L3\n", class->level); + return false; + } + + if (class->type != MPAM_CLASS_MEMORY) { + pr_debug("class %u is neither of type cache or memory\n", class->level); + return false; + } + + cpumask_var_t __free(free_cpumask_var) tmp_cpumask = CPUMASK_VAR_NULL; + if (!alloc_cpumask_var(&tmp_cpumask, GFP_KERNEL)) { + pr_debug("cpumask allocation failed\n"); + return false; + } + + cpu = cpumask_any_and(&class->affinity, cpu_online_mask); + err = find_l3_equivalent_bitmask(cpu, tmp_cpumask); + if (err) { + pr_debug("Failed to find L3 downstream to cpu %d\n", cpu); + return false; + } + + if (!cpumask_equal(tmp_cpumask, cpu_possible_mask)) { + pr_debug("There is more than one L3\n"); + return false; + } + + /* Be strict; the traffic might stop in the intermediate cache. */ + if (get_cpu_cacheinfo_id(cpu, 4) != -1) { + pr_debug("L3 isn't the last level of cache\n"); + return false; + } + + if (num_possible_nodes() > 1) { + pr_debug("There is more than one numa node\n"); + return false; + } + +#ifdef CONFIG_HMEM_REPORTING + if (node_devices[cpu_to_node(cpu)]->cache_dev) { + pr_debug("There is a memory side cache\n"); + return false; + } +#endif + + return true; +} + /* Test whether we can export MPAM_CLASS_CACHE:{2,3}? */ static void mpam_resctrl_pick_caches(void) { @@ -358,9 +539,68 @@ static void mpam_resctrl_pick_caches(void) } } +static void mpam_resctrl_pick_mba(void) +{ + struct mpam_class *class, *candidate_class = NULL; + struct mpam_resctrl_res *res; + + lockdep_assert_cpus_held(); + + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(class, &mpam_classes, classes_list, + srcu_read_lock_held(&mpam_srcu)) { + struct mpam_props *cprops = &class->props; + + if (class->level != 3 && class->type == MPAM_CLASS_CACHE) { + pr_debug("class %u is a cache but not the L3\n", class->level); + continue; + } + + if (!class_has_usable_mba(cprops)) { + pr_debug("class %u has no bandwidth control\n", + class->level); + continue; + } + + if (!cpumask_equal(&class->affinity, cpu_possible_mask)) { + pr_debug("class %u has missing CPUs\n", class->level); + continue; + } + + if (!topology_matches_l3(class)) { + pr_debug("class %u topology doesn't match L3\n", + class->level); + continue; + } + + if (!traffic_matches_l3(class)) { + pr_debug("class %u traffic doesn't match L3 egress\n", + class->level); + continue; + } + + /* + * Pick a resource to be MBA that as close as possible to + * the L3. mbm_total counts the bandwidth leaving the L3 + * cache and MBA should correspond as closely as possible + * for proper operation of mba_sc. + */ + if (!candidate_class || class->level < candidate_class->level) + candidate_class = class; + } + + if (candidate_class) { + pr_debug("selected class %u to back MBA\n", + candidate_class->level); + res = &mpam_resctrl_controls[RDT_RESOURCE_MBA]; + res->class = candidate_class; + } +} + static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) { struct mpam_class *class = res->class; + struct mpam_props *cprops = &class->props; struct rdt_resource *r = &res->resctrl_res; switch (r->rid) { @@ -392,6 +632,19 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) r->cache.shareable_bits = resctrl_get_default_ctrl(r); r->alloc_capable = true; break; + case RDT_RESOURCE_MBA: + r->schema_fmt = RESCTRL_SCHEMA_RANGE; + r->ctrl_scope = RESCTRL_L3_CACHE; + + r->membw.delay_linear = true; + r->membw.throttle_mode = THREAD_THROTTLE_UNDEFINED; + r->membw.min_bw = get_mba_min(cprops); + r->membw.max_bw = MAX_MBA_BW; + r->membw.bw_gran = get_mba_granularity(cprops); + + r->name = "MB"; + r->alloc_capable = true; + break; default: return -EINVAL; } @@ -406,7 +659,17 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) if (class->type == MPAM_CLASS_CACHE) return comp->comp_id; - /* TODO: repaint domain ids to match the L3 domain ids */ + if (topology_matches_l3(class)) { + /* Use the corresponding L3 component ID as the domain ID */ + int id = get_cpu_cacheinfo_id(cpu, 3); + + /* Implies topology_matches_l3() made a mistake */ + if (WARN_ON_ONCE(id == -1)) + return comp->comp_id; + + return id; + } + /* Otherwise, expose the ID used by the firmware table code. */ return comp->comp_id; } @@ -446,6 +709,12 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, case RDT_RESOURCE_L3: configured_by = mpam_feat_cpor_part; break; + case RDT_RESOURCE_MBA: + if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { + configured_by = mpam_feat_mbw_max; + break; + } + fallthrough; default: return resctrl_get_default_ctrl(r); } @@ -457,6 +726,8 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, switch (configured_by) { case mpam_feat_cpor_part: return cfg->cpbm; + case mpam_feat_mbw_max: + return mbw_max_to_percent(cfg->mbw_max, cprops); default: return resctrl_get_default_ctrl(r); } @@ -504,6 +775,13 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, cfg.cpbm = cfg_val; mpam_set_feature(mpam_feat_cpor_part, &cfg); break; + case RDT_RESOURCE_MBA: + if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { + cfg.mbw_max = percent_to_mbw_max(cfg_val, cprops); + mpam_set_feature(mpam_feat_mbw_max, &cfg); + break; + } + fallthrough; default: return -EINVAL; } @@ -775,6 +1053,7 @@ int mpam_resctrl_setup(void) /* Find some classes to use for controls */ mpam_resctrl_pick_caches(); + mpam_resctrl_pick_mba(); /* Initialise the resctrl structures from the classes */ for_each_mpam_resctrl_control(res, rid) { From 4957781463a9d96d4ecde1b12ef5dbd8ec773841 Mon Sep 17 00:00:00 2001 From: Dave Martin Date: Fri, 13 Mar 2026 14:46:03 +0000 Subject: [PATCH 301/464] arm_mpam: resctrl: Add kunit test for control format conversions BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl specifies the format of the control schemes, and these don't match the hardware. Some of the conversions are a bit hairy - add some kunit tests. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Signed-off-by: Dave Martin [morse: squashed enough of Dave's fixes in here that it's his patch now!] Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 5dc8f73eaa5dfccb229b9a25c797720e6379f8e0) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 4 + drivers/resctrl/test_mpam_resctrl.c | 315 ++++++++++++++++++++++++++++ 2 files changed, 319 insertions(+) create mode 100644 drivers/resctrl/test_mpam_resctrl.c diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 60d111f7abfd5..f8d4666fbaa85 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1092,3 +1092,7 @@ static int __init __cacheinfo_ready(void) return 0; } device_initcall_sync(__cacheinfo_ready); + +#ifdef CONFIG_MPAM_KUNIT_TEST +#include "test_mpam_resctrl.c" +#endif diff --git a/drivers/resctrl/test_mpam_resctrl.c b/drivers/resctrl/test_mpam_resctrl.c new file mode 100644 index 0000000000000..b93d6ad87e43f --- /dev/null +++ b/drivers/resctrl/test_mpam_resctrl.c @@ -0,0 +1,315 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2025 Arm Ltd. +/* This file is intended to be included into mpam_resctrl.c */ + +#include +#include +#include +#include +#include + +struct percent_value_case { + u8 pc; + u8 width; + u16 value; +}; + +/* + * Mysterious inscriptions taken from the union of ARM DDI 0598D.b, + * "Arm Architecture Reference Manual Supplement - Memory System + * Resource Partitioning and Monitoring (MPAM), for A-profile + * architecture", Section 9.8, "About the fixed-point fractional + * format" (exact percentage entries only) and ARM IHI0099B.a + * "MPAM system component specification", Section 9.3, + * "The fixed-point fractional format": + */ +static const struct percent_value_case percent_value_cases[] = { + /* Architectural cases: */ + { 1, 8, 1 }, { 1, 12, 0x27 }, { 1, 16, 0x28e }, + { 25, 8, 0x3f }, { 25, 12, 0x3ff }, { 25, 16, 0x3fff }, + { 33, 8, 0x53 }, { 33, 12, 0x546 }, { 33, 16, 0x5479 }, + { 35, 8, 0x58 }, { 35, 12, 0x598 }, { 35, 16, 0x5998 }, + { 45, 8, 0x72 }, { 45, 12, 0x732 }, { 45, 16, 0x7332 }, + { 50, 8, 0x7f }, { 50, 12, 0x7ff }, { 50, 16, 0x7fff }, + { 52, 8, 0x84 }, { 52, 12, 0x850 }, { 52, 16, 0x851d }, + { 55, 8, 0x8b }, { 55, 12, 0x8cb }, { 55, 16, 0x8ccb }, + { 58, 8, 0x93 }, { 58, 12, 0x946 }, { 58, 16, 0x9479 }, + { 75, 8, 0xbf }, { 75, 12, 0xbff }, { 75, 16, 0xbfff }, + { 80, 8, 0xcb }, { 80, 12, 0xccb }, { 80, 16, 0xcccb }, + { 88, 8, 0xe0 }, { 88, 12, 0xe13 }, { 88, 16, 0xe146 }, + { 95, 8, 0xf2 }, { 95, 12, 0xf32 }, { 95, 16, 0xf332 }, + { 100, 8, 0xff }, { 100, 12, 0xfff }, { 100, 16, 0xffff }, +}; + +static void test_percent_value_desc(const struct percent_value_case *param, + char *desc) +{ + snprintf(desc, KUNIT_PARAM_DESC_SIZE, + "pc=%d, width=%d, value=0x%.*x\n", + param->pc, param->width, + DIV_ROUND_UP(param->width, 4), param->value); +} + +KUNIT_ARRAY_PARAM(test_percent_value, percent_value_cases, + test_percent_value_desc); + +struct percent_value_test_info { + u32 pc; /* result of value-to-percent conversion */ + u32 value; /* result of percent-to-value conversion */ + u32 max_value; /* maximum raw value allowed by test params */ + unsigned int shift; /* promotes raw testcase value to 16 bits */ +}; + +/* + * Convert a reference percentage to a fixed-point MAX value and + * vice-versa, based on param (not test->param_value!) + */ +static void __prepare_percent_value_test(struct kunit *test, + struct percent_value_test_info *res, + const struct percent_value_case *param) +{ + struct mpam_props fake_props = { }; + + /* Reject bogus test parameters that would break the tests: */ + KUNIT_ASSERT_GE(test, param->width, 1); + KUNIT_ASSERT_LE(test, param->width, 16); + KUNIT_ASSERT_LT(test, param->value, 1 << param->width); + + mpam_set_feature(mpam_feat_mbw_max, &fake_props); + fake_props.bwa_wd = param->width; + + res->shift = 16 - param->width; + res->max_value = GENMASK_U32(param->width - 1, 0); + res->value = percent_to_mbw_max(param->pc, &fake_props); + res->pc = mbw_max_to_percent(param->value << res->shift, &fake_props); +} + +static void test_get_mba_granularity(struct kunit *test) +{ + int ret; + struct mpam_props fake_props = { }; + + /* Use MBW_MAX */ + mpam_set_feature(mpam_feat_mbw_max, &fake_props); + + fake_props.bwa_wd = 0; + KUNIT_EXPECT_FALSE(test, mba_class_use_mbw_max(&fake_props)); + + fake_props.bwa_wd = 1; + KUNIT_EXPECT_TRUE(test, mba_class_use_mbw_max(&fake_props)); + + /* Architectural maximum: */ + fake_props.bwa_wd = 16; + KUNIT_EXPECT_TRUE(test, mba_class_use_mbw_max(&fake_props)); + + /* No usable control... */ + fake_props.bwa_wd = 0; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 0); + + fake_props.bwa_wd = 1; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 50); /* DIV_ROUND_UP(100, 1 << 1)% = 50% */ + + fake_props.bwa_wd = 2; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 25); /* DIV_ROUND_UP(100, 1 << 2)% = 25% */ + + fake_props.bwa_wd = 3; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 13); /* DIV_ROUND_UP(100, 1 << 3)% = 13% */ + + fake_props.bwa_wd = 6; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 2); /* DIV_ROUND_UP(100, 1 << 6)% = 2% */ + + fake_props.bwa_wd = 7; + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 1); /* DIV_ROUND_UP(100, 1 << 7)% = 1% */ + + /* Granularity saturates at 1% */ + fake_props.bwa_wd = 16; /* architectural maximum */ + ret = get_mba_granularity(&fake_props); + KUNIT_EXPECT_EQ(test, ret, 1); /* DIV_ROUND_UP(100, 1 << 16)% = 1% */ +} + +static void test_mbw_max_to_percent(struct kunit *test) +{ + const struct percent_value_case *param = test->param_value; + struct percent_value_test_info res; + + /* + * Since the reference values in percent_value_cases[] all + * correspond to exact percentages, round-to-nearest will + * always give the exact percentage back when the MPAM max + * value has precision of 0.5% or finer. (Always true for the + * reference data, since they all specify 8 bits or more of + * precision. + * + * So, keep it simple and demand an exact match: + */ + __prepare_percent_value_test(test, &res, param); + KUNIT_EXPECT_EQ(test, res.pc, param->pc); +} + +static void test_percent_to_mbw_max(struct kunit *test) +{ + const struct percent_value_case *param = test->param_value; + struct percent_value_test_info res; + + __prepare_percent_value_test(test, &res, param); + + KUNIT_EXPECT_GE(test, res.value, param->value << res.shift); + KUNIT_EXPECT_LE(test, res.value, (param->value + 1) << res.shift); + KUNIT_EXPECT_LE(test, res.value, res.max_value << res.shift); + + /* No flexibility allowed for 0% and 100%! */ + + if (param->pc == 0) + KUNIT_EXPECT_EQ(test, res.value, 0); + + if (param->pc == 100) + KUNIT_EXPECT_EQ(test, res.value, res.max_value << res.shift); +} + +static const void *test_all_bwa_wd_gen_params(struct kunit *test, const void *prev, + char *desc) +{ + uintptr_t param = (uintptr_t)prev; + + if (param > 15) + return NULL; + + param++; + + snprintf(desc, KUNIT_PARAM_DESC_SIZE, "wd=%u\n", (unsigned int)param); + + return (void *)param; +} + +static unsigned int test_get_bwa_wd(struct kunit *test) +{ + uintptr_t param = (uintptr_t)test->param_value; + + KUNIT_ASSERT_GE(test, param, 1); + KUNIT_ASSERT_LE(test, param, 16); + + return param; +} + +static void test_mbw_max_to_percent_limits(struct kunit *test) +{ + struct mpam_props fake_props = {0}; + u32 max_value; + + mpam_set_feature(mpam_feat_mbw_max, &fake_props); + fake_props.bwa_wd = test_get_bwa_wd(test); + max_value = GENMASK(15, 16 - fake_props.bwa_wd); + + KUNIT_EXPECT_EQ(test, mbw_max_to_percent(max_value, &fake_props), + MAX_MBA_BW); + KUNIT_EXPECT_EQ(test, mbw_max_to_percent(0, &fake_props), + get_mba_min(&fake_props)); + + /* + * Rounding policy dependent 0% sanity-check: + * With round-to-nearest, the minimum mbw_max value really + * should map to 0% if there are at least 200 steps. + * (100 steps may be enough for some other rounding policies.) + */ + if (fake_props.bwa_wd >= 8) + KUNIT_EXPECT_EQ(test, mbw_max_to_percent(0, &fake_props), 0); + + if (fake_props.bwa_wd < 8 && + mbw_max_to_percent(0, &fake_props) == 0) + kunit_warn(test, "wd=%d: Testsuite/driver Rounding policy mismatch?", + fake_props.bwa_wd); +} + +/* + * Check that converting a percentage to mbw_max and back again (or, as + * appropriate, vice-versa) always restores the original value: + */ +static void test_percent_max_roundtrip_stability(struct kunit *test) +{ + struct mpam_props fake_props = {0}; + unsigned int shift; + u32 pc, max, pc2, max2; + + mpam_set_feature(mpam_feat_mbw_max, &fake_props); + fake_props.bwa_wd = test_get_bwa_wd(test); + shift = 16 - fake_props.bwa_wd; + + /* + * Converting a valid value from the coarser scale to the finer + * scale and back again must yield the original value: + */ + if (fake_props.bwa_wd >= 7) { + /* More than 100 steps: only test exact pc values: */ + for (pc = get_mba_min(&fake_props); pc <= MAX_MBA_BW; pc++) { + max = percent_to_mbw_max(pc, &fake_props); + pc2 = mbw_max_to_percent(max, &fake_props); + KUNIT_EXPECT_EQ(test, pc2, pc); + } + } else { + /* Fewer than 100 steps: only test exact mbw_max values: */ + for (max = 0; max < 1 << 16; max += 1 << shift) { + pc = mbw_max_to_percent(max, &fake_props); + max2 = percent_to_mbw_max(pc, &fake_props); + KUNIT_EXPECT_EQ(test, max2, max); + } + } +} + +static void test_percent_to_max_rounding(struct kunit *test) +{ + const struct percent_value_case *param = test->param_value; + unsigned int num_rounded_up = 0, total = 0; + struct percent_value_test_info res; + + for (param = percent_value_cases, total = 0; + param < &percent_value_cases[ARRAY_SIZE(percent_value_cases)]; + param++, total++) { + __prepare_percent_value_test(test, &res, param); + if (res.value > param->value << res.shift) + num_rounded_up++; + } + + /* + * The MPAM driver applies a round-to-nearest policy, whereas a + * round-down policy seems to have been applied in the + * reference table from which the test vectors were selected. + * + * For a large and well-distributed suite of test vectors, + * about half should be rounded up and half down compared with + * the reference table. The actual test vectors are few in + * number and probably not very well distributed however, so + * tolerate a round-up rate of between 1/4 and 3/4 before + * crying foul: + */ + + kunit_info(test, "Round-up rate: %u%% (%u/%u)\n", + DIV_ROUND_CLOSEST(num_rounded_up * 100, total), + num_rounded_up, total); + + KUNIT_EXPECT_GE(test, 4 * num_rounded_up, 1 * total); + KUNIT_EXPECT_LE(test, 4 * num_rounded_up, 3 * total); +} + +static struct kunit_case mpam_resctrl_test_cases[] = { + KUNIT_CASE(test_get_mba_granularity), + KUNIT_CASE_PARAM(test_mbw_max_to_percent, test_percent_value_gen_params), + KUNIT_CASE_PARAM(test_percent_to_mbw_max, test_percent_value_gen_params), + KUNIT_CASE_PARAM(test_mbw_max_to_percent_limits, test_all_bwa_wd_gen_params), + KUNIT_CASE(test_percent_to_max_rounding), + KUNIT_CASE_PARAM(test_percent_max_roundtrip_stability, + test_all_bwa_wd_gen_params), + {} +}; + +static struct kunit_suite mpam_resctrl_test_suite = { + .name = "mpam_resctrl_test_suite", + .test_cases = mpam_resctrl_test_cases, +}; + +kunit_test_suites(&mpam_resctrl_test_suite); From e6e505923c559ef5691e9f80a3de4392bc22c58f Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:46:04 +0000 Subject: [PATCH 302/464] arm_mpam: resctrl: Add monitor initialisation and domain boilerplate BugLink: https://bugs.launchpad.net/bugs/2154527 Add the boilerplate that tells resctrl about the mpam monitors that are available. resctrl expects all (non-telemetry) monitors to be on the L3 and so advertise them there and invent an L3 resctrl resource if required. The L3 cache itself has to exist as the cache ids are used as the domain ids. Bring the resctrl monitor domains online and offline based on the cpus they contain. Support for specific monitor types is left to later. Tested-by: Punit Agrawal Reviewed-by: Zeng Heng Reviewed-by: Jonathan Cameron Signed-off-by: Ben Horgan Reviewed-by: Gavin Shan Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Signed-off-by: James Morse (cherry picked from commit 264c285999fce128fc52743bce582468b26e9f65) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_internal.h | 15 +++ drivers/resctrl/mpam_resctrl.c | 231 ++++++++++++++++++++++++++++++-- 2 files changed, 235 insertions(+), 11 deletions(-) diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 42ee024701650..0ba7ec7fcb456 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -334,7 +334,16 @@ struct mpam_msc_ris { struct mpam_resctrl_dom { struct mpam_component *ctrl_comp; + + /* + * There is no single mon_comp because different events may be backed + * by different class/components. mon_comp is indexed by the event + * number. + */ + struct mpam_component *mon_comp[QOS_NUM_EVENTS]; + struct rdt_ctrl_domain resctrl_ctrl_dom; + struct rdt_l3_mon_domain resctrl_mon_dom; }; struct mpam_resctrl_res { @@ -343,6 +352,12 @@ struct mpam_resctrl_res { bool cdp_enabled; }; +struct mpam_resctrl_mon { + struct mpam_class *class; + + /* per-class data that resctrl needs will live here */ +}; + static inline int mpam_alloc_csu_mon(struct mpam_class *class) { struct mpam_props *cprops = &class->props; diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index f8d4666fbaa85..e03d0f400993c 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -34,6 +34,23 @@ static struct mpam_resctrl_res mpam_resctrl_controls[RDT_NUM_RESOURCES]; rid < RDT_NUM_RESOURCES; \ rid++, res = &mpam_resctrl_controls[rid]) +/* + * The classes we've picked to map to resctrl events. + * Resctrl believes all the worlds a Xeon, and these are all on the L3. This + * array lets us find the actual class backing the event counters. e.g. + * the only memory bandwidth counters may be on the memory controller, but to + * make use of them, we pretend they are on L3. Restrict the events considered + * to those supported by MPAM. + * Class pointer may be NULL. + */ +#define MPAM_MAX_EVENT QOS_L3_MBM_TOTAL_EVENT_ID +static struct mpam_resctrl_mon mpam_resctrl_counters[MPAM_MAX_EVENT + 1]; + +#define for_each_mpam_resctrl_mon(mon, eventid) \ + for (eventid = QOS_FIRST_EVENT, mon = &mpam_resctrl_counters[eventid]; \ + eventid <= MPAM_MAX_EVENT; \ + eventid++, mon = &mpam_resctrl_counters[eventid]) + /* The lock for modifying resctrl's domain lists from cpuhp callbacks. */ static DEFINE_MUTEX(domain_list_lock); @@ -63,6 +80,15 @@ bool resctrl_arch_alloc_capable(void) return false; } +bool resctrl_arch_mon_capable(void) +{ + struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + struct rdt_resource *l3 = &res->resctrl_res; + + /* All monitors are presented as being on the L3 cache */ + return l3->mon_capable; +} + bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level rid) { return mpam_resctrl_controls[rid].cdp_enabled; @@ -89,6 +115,8 @@ static void resctrl_reset_task_closids(void) int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) { u32 partid_i = RESCTRL_RESERVED_CLOSID, partid_d = RESCTRL_RESERVED_CLOSID; + struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + struct rdt_resource *l3 = &res->resctrl_res; int cpu; if (!IS_ENABLED(CONFIG_EXPERT) && enable) { @@ -110,6 +138,11 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) cdp_enabled = enable; mpam_resctrl_controls[rid].cdp_enabled = enable; + if (enable) + l3->mon.num_rmid = resctrl_arch_system_num_rmid_idx() / 2; + else + l3->mon.num_rmid = resctrl_arch_system_num_rmid_idx(); + /* The mbw_max feature can't hide cdp as it's a per-partid maximum. */ if (cdp_enabled && !mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled) mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = false; @@ -674,6 +707,56 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) return comp->comp_id; } +static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, + enum resctrl_event_id type) +{ + struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + struct rdt_resource *l3 = &res->resctrl_res; + + lockdep_assert_cpus_held(); + + /* + * There also needs to be an L3 cache present. + * The check just requires any online CPU and it can't go offline as we + * hold the cpu lock. + */ + if (get_cpu_cacheinfo_id(raw_smp_processor_id(), 3) == -1) + return 0; + + /* + * If there are no MPAM resources on L3, force it into existence. + * topology_matches_l3() already ensures this looks like the L3. + * The domain-ids will be fixed up by mpam_resctrl_domain_hdr_init(). + */ + if (!res->class) { + pr_warn_once("Faking L3 MSC to enable counters.\n"); + res->class = mpam_resctrl_counters[type].class; + } + + /* + * Called multiple times!, once per event type that has a + * monitoring class. + * Setting name is necessary on monitor only platforms. + */ + l3->name = "L3"; + l3->mon_scope = RESCTRL_L3_CACHE; + + /* + * num-rmid is the upper bound for the number of monitoring groups that + * can exist simultaneously, including the default monitoring group for + * each control group. Hence, advertise the whole rmid_idx space even + * though each control group has its own pmg/rmid space. Unfortunately, + * this does mean userspace needs to know the architecture to correctly + * interpret this value. + */ + l3->mon.num_rmid = resctrl_arch_system_num_rmid_idx(); + + if (resctrl_enable_mon_event(type, false, 0, NULL)) + l3->mon_capable = true; + + return 0; +} + u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, u32 closid, enum resctrl_conf_type type) { @@ -901,11 +984,26 @@ static void mpam_resctrl_domain_insert(struct list_head *list, list_add_tail_rcu(&new->list, pos); } +static struct mpam_component *find_component(struct mpam_class *class, int cpu) +{ + struct mpam_component *comp; + + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(comp, &class->components, class_list, + srcu_read_lock_held(&mpam_srcu)) { + if (cpumask_test_cpu(cpu, &comp->affinity)) + return comp; + } + + return NULL; +} + static struct mpam_resctrl_dom * mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) { int err; struct mpam_resctrl_dom *dom; + struct rdt_l3_mon_domain *mon_d; struct rdt_ctrl_domain *ctrl_d; struct mpam_class *class = res->class; struct mpam_component *comp_iter, *ctrl_comp; @@ -945,8 +1043,56 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) } else { pr_debug("Skipped control domain online - no controls\n"); } + + if (r->mon_capable) { + struct mpam_component *any_mon_comp; + struct mpam_resctrl_mon *mon; + enum resctrl_event_id eventid; + + /* + * Even if the monitor domain is backed by a different + * component, the L3 component IDs need to be used... only + * there may be no ctrl_comp for the L3. + * Search each event's class list for a component with + * overlapping CPUs and set up the dom->mon_comp array. + */ + + for_each_mpam_resctrl_mon(mon, eventid) { + struct mpam_component *mon_comp; + + if (!mon->class) + continue; // dummy resource + + mon_comp = find_component(mon->class, cpu); + dom->mon_comp[eventid] = mon_comp; + if (mon_comp) + any_mon_comp = mon_comp; + } + if (!any_mon_comp) { + WARN_ON_ONCE(0); + err = -EFAULT; + goto offline_ctrl_domain; + } + + mon_d = &dom->resctrl_mon_dom; + mpam_resctrl_domain_hdr_init(cpu, any_mon_comp, r->rid, &mon_d->hdr); + mon_d->hdr.type = RESCTRL_MON_DOMAIN; + err = resctrl_online_mon_domain(r, &mon_d->hdr); + if (err) + goto offline_ctrl_domain; + + mpam_resctrl_domain_insert(&r->mon_domains, &mon_d->hdr); + } else { + pr_debug("Skipped monitor domain online - no monitors\n"); + } + return dom; +offline_ctrl_domain: + if (r->alloc_capable) { + mpam_resctrl_offline_domain_hdr(cpu, &ctrl_d->hdr); + resctrl_offline_ctrl_domain(r, ctrl_d); + } free_domain: kfree(dom); dom = ERR_PTR(err); @@ -954,6 +1100,35 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) return dom; } +/* + * We know all the monitors are associated with the L3, even if there are no + * controls and therefore no control component. Find the cache-id for the CPU + * and use that to search for existing resctrl domains. + * This relies on mpam_resctrl_pick_domain_id() using the L3 cache-id + * for anything that is not a cache. + */ +static struct mpam_resctrl_dom *mpam_resctrl_get_mon_domain_from_cpu(int cpu) +{ + int cache_id; + struct mpam_resctrl_dom *dom; + struct mpam_resctrl_res *l3 = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + + lockdep_assert_cpus_held(); + + if (!l3->class) + return NULL; + cache_id = get_cpu_cacheinfo_id(cpu, 3); + if (cache_id < 0) + return NULL; + + list_for_each_entry_rcu(dom, &l3->resctrl_res.mon_domains, resctrl_mon_dom.hdr.list) { + if (dom->resctrl_mon_dom.hdr.id == cache_id) + return dom; + } + + return NULL; +} + static struct mpam_resctrl_dom * mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) { @@ -967,7 +1142,11 @@ mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) return dom; } - return NULL; + if (r->rid != RDT_RESOURCE_L3) + return NULL; + + /* Search the mon domain list too - needed on monitor only platforms. */ + return mpam_resctrl_get_mon_domain_from_cpu(cpu); } int mpam_resctrl_online_cpu(unsigned int cpu) @@ -994,6 +1173,11 @@ int mpam_resctrl_online_cpu(unsigned int cpu) mpam_resctrl_online_domain_hdr(cpu, &ctrl_d->hdr); } + if (r->mon_capable) { + struct rdt_l3_mon_domain *mon_d = &dom->resctrl_mon_dom; + + mpam_resctrl_online_domain_hdr(cpu, &mon_d->hdr); + } } } @@ -1012,8 +1196,9 @@ void mpam_resctrl_offline_cpu(unsigned int cpu) guard(mutex)(&domain_list_lock); for_each_mpam_resctrl_control(res, rid) { struct mpam_resctrl_dom *dom; + struct rdt_l3_mon_domain *mon_d; struct rdt_ctrl_domain *ctrl_d; - bool ctrl_dom_empty; + bool ctrl_dom_empty, mon_dom_empty; struct rdt_resource *r = &res->resctrl_res; if (!res->class) @@ -1032,7 +1217,16 @@ void mpam_resctrl_offline_cpu(unsigned int cpu) ctrl_dom_empty = true; } - if (ctrl_dom_empty) + if (r->mon_capable) { + mon_d = &dom->resctrl_mon_dom; + mon_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &mon_d->hdr); + if (mon_dom_empty) + resctrl_offline_mon_domain(&res->resctrl_res, &mon_d->hdr); + } else { + mon_dom_empty = true; + } + + if (ctrl_dom_empty && mon_dom_empty) kfree(dom); } } @@ -1042,12 +1236,15 @@ int mpam_resctrl_setup(void) int err = 0; struct mpam_resctrl_res *res; enum resctrl_res_level rid; + struct mpam_resctrl_mon *mon; + enum resctrl_event_id eventid; wait_event(wait_cacheinfo_ready, cacheinfo_ready); cpus_read_lock(); for_each_mpam_resctrl_control(res, rid) { INIT_LIST_HEAD_RCU(&res->resctrl_res.ctrl_domains); + INIT_LIST_HEAD_RCU(&res->resctrl_res.mon_domains); res->resctrl_res.rid = rid; } @@ -1063,25 +1260,37 @@ int mpam_resctrl_setup(void) err = mpam_resctrl_control_init(res); if (err) { pr_debug("Failed to initialise rid %u\n", rid); - break; + goto internal_error; } } - cpus_read_unlock(); - if (err) { - pr_debug("Internal error %d - resctrl not supported\n", err); - return err; + for_each_mpam_resctrl_mon(mon, eventid) { + if (!mon->class) + continue; // dummy resource + + err = mpam_resctrl_monitor_init(mon, eventid); + if (err) { + pr_debug("Failed to initialise event %u\n", eventid); + goto internal_error; + } } - if (!resctrl_arch_alloc_capable()) { - pr_debug("No alloc(%u) found - resctrl not supported\n", - resctrl_arch_alloc_capable()); + cpus_read_unlock(); + + if (!resctrl_arch_alloc_capable() && !resctrl_arch_mon_capable()) { + pr_debug("No alloc(%u) or monitor(%u) found - resctrl not supported\n", + resctrl_arch_alloc_capable(), resctrl_arch_mon_capable()); return -EOPNOTSUPP; } /* TODO: call resctrl_init() */ return 0; + +internal_error: + cpus_read_unlock(); + pr_debug("Internal error %d - resctrl not supported\n", err); + return err; } static int __init __cacheinfo_ready(void) From 00e8f9a8a697fd096cc953390b6f942d86bf17ca Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:05 +0000 Subject: [PATCH 303/464] arm_mpam: resctrl: Add support for csu counters BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl exposes a counter via a file named llc_occupancy. This isn't really a counter as its value goes up and down, this is a snapshot of the cache storage usage monitor. Add some picking code which will only find an L3. The resctrl counter file is called llc_occupancy but we don't check it is the last one as it is already identified as L3. Tested-by: Shaopeng Tan Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Gavin Shan Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Dave Martin Signed-off-by: Dave Martin Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 1458c4f053355f88cc5d190ca02243d2c60fa010) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index e03d0f400993c..07bb20a01b383 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -311,6 +311,28 @@ static bool class_has_usable_mba(struct mpam_props *cprops) return mba_class_use_mbw_max(cprops); } +static bool cache_has_usable_csu(struct mpam_class *class) +{ + struct mpam_props *cprops; + + if (!class) + return false; + + cprops = &class->props; + + if (!mpam_has_feature(mpam_feat_msmon_csu, cprops)) + return false; + + /* + * CSU counters settle on the value, so we can get away with + * having only one. + */ + if (!cprops->num_csu_mon) + return false; + + return true; +} + /* * Calculate the worst-case percentage change from each implemented step * in the control. @@ -630,6 +652,64 @@ static void mpam_resctrl_pick_mba(void) } } +static void counter_update_class(enum resctrl_event_id evt_id, + struct mpam_class *class) +{ + struct mpam_class *existing_class = mpam_resctrl_counters[evt_id].class; + + if (existing_class) { + if (class->level == 3) { + pr_debug("Existing class is L3 - L3 wins\n"); + return; + } + + if (existing_class->level < class->level) { + pr_debug("Existing class is closer to L3, %u versus %u - closer is better\n", + existing_class->level, class->level); + return; + } + } + + mpam_resctrl_counters[evt_id].class = class; +} + +static void mpam_resctrl_pick_counters(void) +{ + struct mpam_class *class; + + lockdep_assert_cpus_held(); + + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(class, &mpam_classes, classes_list, + srcu_read_lock_held(&mpam_srcu)) { + /* The name of the resource is L3... */ + if (class->type == MPAM_CLASS_CACHE && class->level != 3) { + pr_debug("class %u is a cache but not the L3", class->level); + continue; + } + + if (!cpumask_equal(&class->affinity, cpu_possible_mask)) { + pr_debug("class %u does not cover all CPUs", + class->level); + continue; + } + + if (cache_has_usable_csu(class)) { + pr_debug("class %u has usable CSU", + class->level); + + /* CSU counters only make sense on a cache. */ + switch (class->type) { + case MPAM_CLASS_CACHE: + counter_update_class(QOS_L3_OCCUP_EVENT_ID, class); + break; + default: + break; + } + } + } +} + static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) { struct mpam_class *class = res->class; @@ -1264,6 +1344,9 @@ int mpam_resctrl_setup(void) } } + /* Find some classes to use for monitors */ + mpam_resctrl_pick_counters(); + for_each_mpam_resctrl_mon(mon, eventid) { if (!mon->class) continue; // dummy resource From 69bd0c0d1b44033e4ff9a1389d260bf59aa58527 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:06 +0000 Subject: [PATCH 304/464] arm_mpam: resctrl: Allow resctrl to allocate monitors BugLink: https://bugs.launchpad.net/bugs/2154527 When resctrl wants to read a domain's 'QOS_L3_OCCUP', it needs to allocate a monitor on the corresponding resource. Monitors are allocated by class instead of component. Add helpers to allocate a CSU monitor. These helper return an out of range value for MBM counters. Allocating a montitor context is expected to block until hardware resources become available. This only makes sense for QOS_L3_OCCUP as unallocated MBM counters are losing data. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 2a3c79c61539779a09928893518c8286d7774b54) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_internal.h | 14 ++++++- drivers/resctrl/mpam_resctrl.c | 67 +++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 5 +++ 3 files changed, 85 insertions(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 0ba7ec7fcb456..72c209ce040fa 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -29,6 +29,14 @@ struct platform_device; #define PACKED_FOR_KUNIT #endif +/* + * This 'mon' values must not alias an actual monitor, so must be larger than + * U16_MAX, but not be confused with an errno value, so smaller than + * (u32)-SZ_4K. + * USE_PRE_ALLOCATED is used to avoid confusion with an actual monitor. + */ +#define USE_PRE_ALLOCATED (U16_MAX + 1) + static inline bool mpam_is_enabled(void) { return static_branch_likely(&mpam_enabled); @@ -214,7 +222,11 @@ enum mon_filter_options { }; struct mon_cfg { - u16 mon; + /* + * mon must be large enough to hold out of range values like + * USE_PRE_ALLOCATED + */ + u32 mon; u8 pmg; bool match_pmg; bool csu_exclude_clean; diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 07bb20a01b383..9682ffb151846 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -22,6 +22,8 @@ #include "mpam_internal.h" +DECLARE_WAIT_QUEUE_HEAD(resctrl_mon_ctx_waiters); + /* * The classes we've picked to map to resctrl resources, wrapped * in with their resctrl structure. @@ -289,6 +291,71 @@ struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) return &mpam_resctrl_controls[l].resctrl_res; } +static int resctrl_arch_mon_ctx_alloc_no_wait(enum resctrl_event_id evtid) +{ + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evtid]; + + if (!mon->class) + return -EINVAL; + + switch (evtid) { + case QOS_L3_OCCUP_EVENT_ID: + /* With CDP, one monitor gets used for both code/data reads */ + return mpam_alloc_csu_mon(mon->class); + case QOS_L3_MBM_LOCAL_EVENT_ID: + case QOS_L3_MBM_TOTAL_EVENT_ID: + return USE_PRE_ALLOCATED; + default: + return -EOPNOTSUPP; + } +} + +void *resctrl_arch_mon_ctx_alloc(struct rdt_resource *r, + enum resctrl_event_id evtid) +{ + DEFINE_WAIT(wait); + int *ret; + + ret = kmalloc_obj(*ret); + if (!ret) + return ERR_PTR(-ENOMEM); + + do { + prepare_to_wait(&resctrl_mon_ctx_waiters, &wait, + TASK_INTERRUPTIBLE); + *ret = resctrl_arch_mon_ctx_alloc_no_wait(evtid); + if (*ret == -ENOSPC) + schedule(); + } while (*ret == -ENOSPC && !signal_pending(current)); + finish_wait(&resctrl_mon_ctx_waiters, &wait); + + return ret; +} + +static void resctrl_arch_mon_ctx_free_no_wait(enum resctrl_event_id evtid, + u32 mon_idx) +{ + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evtid]; + + if (!mon->class) + return; + + if (evtid == QOS_L3_OCCUP_EVENT_ID) + mpam_free_csu_mon(mon->class, mon_idx); + + wake_up(&resctrl_mon_ctx_waiters); +} + +void resctrl_arch_mon_ctx_free(struct rdt_resource *r, + enum resctrl_event_id evtid, void *arch_mon_ctx) +{ + u32 mon_idx = *(u32 *)arch_mon_ctx; + + kfree(arch_mon_ctx); + + resctrl_arch_mon_ctx_free_no_wait(evtid, mon_idx); +} + static bool cache_has_usable_cpor(struct mpam_class *class) { struct mpam_props *cprops = &class->props; diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 7d23c90f077dc..e1461e32af756 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -5,6 +5,7 @@ #define __LINUX_ARM_MPAM_H #include +#include #include struct mpam_msc; @@ -62,6 +63,10 @@ u32 resctrl_arch_rmid_idx_encode(u32 closid, u32 rmid); void resctrl_arch_rmid_idx_decode(u32 idx, u32 *closid, u32 *rmid); u32 resctrl_arch_system_num_rmid_idx(void); +struct rdt_resource; +void *resctrl_arch_mon_ctx_alloc(struct rdt_resource *r, enum resctrl_event_id evtid); +void resctrl_arch_mon_ctx_free(struct rdt_resource *r, enum resctrl_event_id evtid, void *ctx); + /** * mpam_register_requestor() - Register a requestor with the MPAM driver * @partid_max: The maximum PARTID value the requestor can generate. From a7cfb3feaaade6a888172e6540401f810523c151 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:07 +0000 Subject: [PATCH 305/464] arm_mpam: resctrl: Add resctrl_arch_rmid_read() BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl uses resctrl_arch_rmid_read() to read counters. CDP emulation means the counter may need reading in three different ways. The helpers behind the resctrl_arch_ functions will be re-used for the ABMC equivalent functions. Add the rounding helper for checking monitor values while we're here. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Jesse Chick Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit fb56b29932ca276df268806ad52ed80f40f99a6e) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 82 ++++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 5 +++ 2 files changed, 87 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 9682ffb151846..9a15ddd340f73 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -356,6 +356,88 @@ void resctrl_arch_mon_ctx_free(struct rdt_resource *r, resctrl_arch_mon_ctx_free_no_wait(evtid, mon_idx); } +static int __read_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, + enum mpam_device_features mon_type, + int mon_idx, + enum resctrl_conf_type cdp_type, u32 closid, u32 rmid, u64 *val) +{ + struct mon_cfg cfg; + + if (!mpam_is_enabled()) + return -EINVAL; + + /* Shift closid to account for CDP */ + closid = resctrl_get_config_index(closid, cdp_type); + + if (irqs_disabled()) { + /* Check if we can access this domain without an IPI */ + return -EIO; + } + + cfg = (struct mon_cfg) { + .mon = mon_idx, + .match_pmg = true, + .partid = closid, + .pmg = rmid, + }; + + return mpam_msmon_read(mon_comp, &cfg, mon_type, val); +} + +static int read_mon_cdp_safe(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, + enum mpam_device_features mon_type, + int mon_idx, u32 closid, u32 rmid, u64 *val) +{ + if (cdp_enabled) { + u64 code_val = 0, data_val = 0; + int err; + + err = __read_mon(mon, mon_comp, mon_type, mon_idx, + CDP_CODE, closid, rmid, &code_val); + if (err) + return err; + + err = __read_mon(mon, mon_comp, mon_type, mon_idx, + CDP_DATA, closid, rmid, &data_val); + if (err) + return err; + + *val += code_val + data_val; + return 0; + } + + return __read_mon(mon, mon_comp, mon_type, mon_idx, + CDP_NONE, closid, rmid, val); +} + +/* MBWU when not in ABMC mode (not supported), and CSU counters. */ +int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, + u32 closid, u32 rmid, enum resctrl_event_id eventid, + void *arch_priv, u64 *val, void *arch_mon_ctx) +{ + struct mpam_resctrl_dom *l3_dom; + struct mpam_component *mon_comp; + u32 mon_idx = *(u32 *)arch_mon_ctx; + enum mpam_device_features mon_type; + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[eventid]; + + resctrl_arch_rmid_read_context_check(); + + if (eventid >= QOS_NUM_EVENTS || !mon->class) + return -EINVAL; + + l3_dom = container_of(hdr, struct mpam_resctrl_dom, resctrl_mon_dom.hdr); + mon_comp = l3_dom->mon_comp[eventid]; + + if (eventid != QOS_L3_OCCUP_EVENT_ID) + return -EINVAL; + + mon_type = mpam_feat_msmon_csu; + + return read_mon_cdp_safe(mon, mon_comp, mon_type, mon_idx, + closid, rmid, val); +} + static bool cache_has_usable_cpor(struct mpam_class *class) { struct mpam_props *cprops = &class->props; diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index e1461e32af756..86d5e326d2bd3 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -67,6 +67,11 @@ struct rdt_resource; void *resctrl_arch_mon_ctx_alloc(struct rdt_resource *r, enum resctrl_event_id evtid); void resctrl_arch_mon_ctx_free(struct rdt_resource *r, enum resctrl_event_id evtid, void *ctx); +static inline unsigned int resctrl_arch_round_mon_val(unsigned int val) +{ + return val; +} + /** * mpam_register_requestor() - Register a requestor with the MPAM driver * @partid_max: The maximum PARTID value the requestor can generate. From cc3654947c47055c9e6a7e1384e67db38bba8538 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:08 +0000 Subject: [PATCH 306/464] arm_mpam: resctrl: Update the rmid reallocation limit BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl's limbo code needs to be told when the data left in a cache is small enough for the partid+pmg value to be re-allocated. x86 uses the cache size divided by the number of rmid users the cache may have. Do the same, but for the smallest cache, and with the number of partid-and-pmg users. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 49b04e401825431529e866470d8d2dcd8e9ef058) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 39 ++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 9a15ddd340f73..f82fff3519df4 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -438,6 +438,42 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, closid, rmid, val); } +/* + * The rmid realloc threshold should be for the smallest cache exposed to + * resctrl. + */ +static int update_rmid_limits(struct mpam_class *class) +{ + u32 num_unique_pmg = resctrl_arch_system_num_rmid_idx(); + struct mpam_props *cprops = &class->props; + struct cacheinfo *ci; + + lockdep_assert_cpus_held(); + + if (!mpam_has_feature(mpam_feat_msmon_csu, cprops)) + return 0; + + /* + * Assume cache levels are the same size for all CPUs... + * The check just requires any online CPU and it can't go offline as we + * hold the cpu lock. + */ + ci = get_cpu_cacheinfo_level(raw_smp_processor_id(), class->level); + if (!ci || ci->size == 0) { + pr_debug("Could not read cache size for class %u\n", + class->level); + return -EINVAL; + } + + if (!resctrl_rmid_realloc_limit || + ci->size < resctrl_rmid_realloc_limit) { + resctrl_rmid_realloc_limit = ci->size; + resctrl_rmid_realloc_threshold = ci->size / num_unique_pmg; + } + + return 0; +} + static bool cache_has_usable_cpor(struct mpam_class *class) { struct mpam_props *cprops = &class->props; @@ -850,6 +886,9 @@ static void mpam_resctrl_pick_counters(void) /* CSU counters only make sense on a cache. */ switch (class->type) { case MPAM_CLASS_CACHE: + if (update_rmid_limits(class)) + break; + counter_update_class(QOS_L3_OCCUP_EVENT_ID, class); break; default: From f6199cdd090e15093624de20e602dd479b1c9a6a Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:09 +0000 Subject: [PATCH 307/464] arm_mpam: resctrl: Add empty definitions for assorted resctrl functions BugLink: https://bugs.launchpad.net/bugs/2154527 A few resctrl features and hooks need to be provided, but aren't needed or supported on MPAM platforms. resctrl has individual hooks to separately enable and disable the closid/partid and rmid/pmg context switching code. For MPAM this is all the same thing, as the value in struct task_struct is used to cache the value that should be written to hardware. arm64's context switching code is enabled once MPAM is usable, but doesn't touch the hardware unless the value has changed. For now event configuration is not supported, and can be turned off by returning 'false' from resctrl_arch_is_evt_configurable(). The new io_alloc feature is not supported either, always return false from the enable helper to indicate and fail the enable. Add this, and empty definitions for the other hooks. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit efc775eadce2c6e0921c21d9c29a7b6686022281) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 65 ++++++++++++++++++++++++++++++++++ include/linux/arm_mpam.h | 9 +++++ 2 files changed, 74 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index f82fff3519df4..777ecdc2d0f85 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -91,6 +91,71 @@ bool resctrl_arch_mon_capable(void) return l3->mon_capable; } +bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt) +{ + return false; +} + +void resctrl_arch_mon_event_config_read(void *info) +{ +} + +void resctrl_arch_mon_event_config_write(void *info) +{ +} + +void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domain *d) +{ +} + +void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 closid, u32 rmid, enum resctrl_event_id eventid) +{ +} + +void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 closid, u32 rmid, int cntr_id, + enum resctrl_event_id eventid) +{ +} + +void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + enum resctrl_event_id evtid, u32 rmid, u32 closid, + u32 cntr_id, bool assign) +{ +} + +int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 unused, u32 rmid, int cntr_id, + enum resctrl_event_id eventid, u64 *val) +{ + return -EOPNOTSUPP; +} + +bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) +{ + return false; +} + +int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) +{ + return -EINVAL; +} + +int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable) +{ + return -EOPNOTSUPP; +} + +bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r) +{ + return false; +} + +void resctrl_arch_pre_mount(void) +{ +} + bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level rid) { return mpam_resctrl_controls[rid].cdp_enabled; diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 86d5e326d2bd3..f92a36187a527 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -67,6 +67,15 @@ struct rdt_resource; void *resctrl_arch_mon_ctx_alloc(struct rdt_resource *r, enum resctrl_event_id evtid); void resctrl_arch_mon_ctx_free(struct rdt_resource *r, enum resctrl_event_id evtid, void *ctx); +/* + * The CPU configuration for MPAM is cheap to write, and is only written if it + * has changed. No need for fine grained enables. + */ +static inline void resctrl_arch_enable_mon(void) { } +static inline void resctrl_arch_disable_mon(void) { } +static inline void resctrl_arch_enable_alloc(void) { } +static inline void resctrl_arch_disable_alloc(void) { } + static inline unsigned int resctrl_arch_round_mon_val(unsigned int val) { return val; From 1c9a05124accf71e2cbafd6d91c8c003bc2e4835 Mon Sep 17 00:00:00 2001 From: Takashi Iwai Date: Fri, 27 Mar 2026 16:30:53 +0100 Subject: [PATCH 308/464] ALSA: usb-audio: Replace hard-coded number with MAX_CHANNELS BugLink: https://bugs.launchpad.net/bugs/2154527 One place in mixer.c still used a hard-coded number 16 instead of MAX_CHANNELS. Replace with it, so that we can extend the max number of channels gracefully. Link: https://lore.kernel.org/F1B104A5-CD6A-4A26-AB46-14BF233C0579@getmailspring.com Tested-by: Phil Willoughby Link: https://patch.msgid.link/20260327153056.691575-1-tiwai@suse.de Signed-off-by: Takashi Iwai (cherry picked from commit ec9a788620be1c11535fe99e9b2779f9eef2b099) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- sound/usb/mixer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sound/usb/mixer.c b/sound/usb/mixer.c index 21191da4c0496..bf6b6b8582ef9 100644 --- a/sound/usb/mixer.c +++ b/sound/usb/mixer.c @@ -1714,7 +1714,7 @@ static void __build_feature_ctl(struct usb_mixer_interface *mixer, cval->master_readonly = readonly_mask; } else { int i, c = 0; - for (i = 0; i < 16; i++) + for (i = 0; i < MAX_CHANNELS; i++) if (ctl_mask & BIT(i)) c++; cval->channels = c; From 69d34f1ae44ff16d0e664de08dab9e99d1777a1c Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:10 +0000 Subject: [PATCH 309/464] arm64: mpam: Select ARCH_HAS_CPU_RESCTRL BugLink: https://bugs.launchpad.net/bugs/2154527 Enough MPAM support is present to enable ARCH_HAS_CPU_RESCTRL. Let it rip^Wlink! ARCH_HAS_CPU_RESCTRL indicates resctrl can be enabled. It is enabled by the arch code simply because it has 'arch' in its name. This removes ARM_CPU_RESCTRL as a mimic of X86_CPU_RESCTRL. While here, move the ACPI dependency to the driver's Kconfig file. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Acked-by: Catalin Marinas Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 4aab135bda1661a795e4fe96418bf840833e1119) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/resctrl.h | 2 ++ drivers/resctrl/Kconfig | 7 +++++++ drivers/resctrl/Makefile | 2 +- 4 files changed, 11 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/resctrl.h diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a34eb53c563a2..bf706982e0335 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2048,7 +2048,7 @@ config ARM64_TLB_RANGE config ARM64_MPAM bool "Enable support for MPAM" select ARM64_MPAM_DRIVER - select ACPI_MPAM if ACPI + select ARCH_HAS_CPU_RESCTRL help Memory System Resource Partitioning and Monitoring (MPAM) is an optional extension to the Arm architecture that allows each diff --git a/arch/arm64/include/asm/resctrl.h b/arch/arm64/include/asm/resctrl.h new file mode 100644 index 0000000000000..b506e95cf6e37 --- /dev/null +++ b/arch/arm64/include/asm/resctrl.h @@ -0,0 +1,2 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include diff --git a/drivers/resctrl/Kconfig b/drivers/resctrl/Kconfig index c34e059c6e41f..672abea3b03cc 100644 --- a/drivers/resctrl/Kconfig +++ b/drivers/resctrl/Kconfig @@ -1,6 +1,7 @@ menuconfig ARM64_MPAM_DRIVER bool "MPAM driver" depends on ARM64 && ARM64_MPAM + select ACPI_MPAM if ACPI help Memory System Resource Partitioning and Monitoring (MPAM) driver for System IP, e.g. caches and memory controllers. @@ -22,3 +23,9 @@ config MPAM_KUNIT_TEST If unsure, say N. endif + +config ARM64_MPAM_RESCTRL_FS + bool + default y if ARM64_MPAM_DRIVER && RESCTRL_FS + select RESCTRL_RMID_DEPENDS_ON_CLOSID + select RESCTRL_ASSIGN_FIXED diff --git a/drivers/resctrl/Makefile b/drivers/resctrl/Makefile index 40beaf999582c..4f6d0e81f9b8f 100644 --- a/drivers/resctrl/Makefile +++ b/drivers/resctrl/Makefile @@ -1,5 +1,5 @@ obj-$(CONFIG_ARM64_MPAM_DRIVER) += mpam.o mpam-y += mpam_devices.o -mpam-$(CONFIG_ARM_CPU_RESCTRL) += mpam_resctrl.o +mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS) += mpam_resctrl.o ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG) += -DDEBUG From 4b1d84bd07633f1ba7a68656f0d324d00c32fa70 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:11 +0000 Subject: [PATCH 310/464] arm_mpam: resctrl: Call resctrl_init() on platforms that can support resctrl BugLink: https://bugs.launchpad.net/bugs/2154527 Now that MPAM links against resctrl, call resctrl_init() to register the filesystem and setup resctrl's structures. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Peter Newman Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit fb481ec08699e9daf08ab839a79ab37b1bcca94d) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 32 ++++++++++++++--- drivers/resctrl/mpam_internal.h | 4 +++ drivers/resctrl/mpam_resctrl.c | 63 ++++++++++++++++++++++++++++++++- 3 files changed, 94 insertions(+), 5 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index e4a52b79d0bf1..03f769f6864cc 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -73,6 +73,14 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable); /* When mpam is disabled, the printed reason to aid debugging */ static char *mpam_disable_reason; +/* + * Whether resctrl has been setup. Used by cpuhp in preference to + * mpam_is_enabled(). The disable call after an error interrupt makes + * mpam_is_enabled() false before the cpuhp callbacks are made. + * Reads/writes should hold mpam_cpuhp_state_lock, (or be cpuhp callbacks). + */ +static bool mpam_resctrl_enabled; + /* * An MSC is a physical container for controls and monitors, each identified by * their RIS index. These share a base-address, interrupts and some MMIO @@ -1617,7 +1625,7 @@ static int mpam_cpu_online(unsigned int cpu) mpam_reprogram_msc(msc); } - if (mpam_is_enabled()) + if (mpam_resctrl_enabled) return mpam_resctrl_online_cpu(cpu); return 0; @@ -1663,7 +1671,7 @@ static int mpam_cpu_offline(unsigned int cpu) { struct mpam_msc *msc; - if (mpam_is_enabled()) + if (mpam_resctrl_enabled) mpam_resctrl_offline_cpu(cpu); guard(srcu)(&mpam_srcu); @@ -2527,6 +2535,7 @@ static void mpam_enable_once(void) } static_branch_enable(&mpam_enabled); + mpam_resctrl_enabled = true; mpam_register_cpuhp_callbacks(mpam_cpu_online, mpam_cpu_offline, "mpam:online"); @@ -2586,24 +2595,39 @@ static void mpam_reset_class(struct mpam_class *class) void mpam_disable(struct work_struct *ignored) { int idx; + bool do_resctrl_exit; struct mpam_class *class; struct mpam_msc *msc, *tmp; + if (mpam_is_enabled()) + static_branch_disable(&mpam_enabled); + mutex_lock(&mpam_cpuhp_state_lock); if (mpam_cpuhp_state) { cpuhp_remove_state(mpam_cpuhp_state); mpam_cpuhp_state = 0; } + + /* + * Removing the cpuhp state called mpam_cpu_offline() and told resctrl + * all the CPUs are offline. + */ + do_resctrl_exit = mpam_resctrl_enabled; + mpam_resctrl_enabled = false; mutex_unlock(&mpam_cpuhp_state_lock); - static_branch_disable(&mpam_enabled); + if (do_resctrl_exit) + mpam_resctrl_exit(); mpam_unregister_irqs(); idx = srcu_read_lock(&mpam_srcu); list_for_each_entry_srcu(class, &mpam_classes, classes_list, - srcu_read_lock_held(&mpam_srcu)) + srcu_read_lock_held(&mpam_srcu)) { mpam_reset_class(class); + if (do_resctrl_exit) + mpam_resctrl_teardown_class(class); + } srcu_read_unlock(&mpam_srcu, idx); mutex_lock(&mpam_list_lock); diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 72c209ce040fa..3f6390649621a 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -429,12 +429,16 @@ int mpam_get_cpumask_from_cache_id(unsigned long cache_id, u32 cache_level, #ifdef CONFIG_RESCTRL_FS int mpam_resctrl_setup(void); +void mpam_resctrl_exit(void); int mpam_resctrl_online_cpu(unsigned int cpu); void mpam_resctrl_offline_cpu(unsigned int cpu); +void mpam_resctrl_teardown_class(struct mpam_class *class); #else static inline int mpam_resctrl_setup(void) { return 0; } +static inline void mpam_resctrl_exit(void) { } static inline int mpam_resctrl_online_cpu(unsigned int cpu) { return 0; } static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { } +static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { } #endif /* CONFIG_RESCTRL_FS */ /* diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 777ecdc2d0f85..a9938006d0e6e 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -69,6 +69,12 @@ static bool cdp_enabled; static bool cacheinfo_ready; static DECLARE_WAIT_QUEUE_HEAD(wait_cacheinfo_ready); +/* + * If resctrl_init() succeeded, resctrl_exit() can be used to remove support + * for the filesystem in the event of an error. + */ +static bool resctrl_enabled; + bool resctrl_arch_alloc_capable(void) { struct mpam_resctrl_res *res; @@ -360,6 +366,9 @@ static int resctrl_arch_mon_ctx_alloc_no_wait(enum resctrl_event_id evtid) { struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evtid]; + if (!mpam_is_enabled()) + return -EINVAL; + if (!mon->class) return -EINVAL; @@ -402,6 +411,9 @@ static void resctrl_arch_mon_ctx_free_no_wait(enum resctrl_event_id evtid, { struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evtid]; + if (!mpam_is_enabled()) + return; + if (!mon->class) return; @@ -488,6 +500,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, resctrl_arch_rmid_read_context_check(); + if (!mpam_is_enabled()) + return -EINVAL; + if (eventid >= QOS_NUM_EVENTS || !mon->class) return -EINVAL; @@ -1162,6 +1177,9 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, lockdep_assert_cpus_held(); lockdep_assert_irqs_enabled(); + if (!mpam_is_enabled()) + return -EINVAL; + /* * No need to check the CPU as mpam_apply_config() doesn't care, and * resctrl_arch_update_domains() relies on this. @@ -1227,6 +1245,9 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid) lockdep_assert_cpus_held(); lockdep_assert_irqs_enabled(); + if (!mpam_is_enabled()) + return -EINVAL; + list_for_each_entry_rcu(d, &r->ctrl_domains, hdr.list) { for (enum resctrl_conf_type t = 0; t < CDP_NUM_TYPES; t++) { struct resctrl_staged_config *cfg = &d->staged_config[t]; @@ -1619,7 +1640,11 @@ int mpam_resctrl_setup(void) return -EOPNOTSUPP; } - /* TODO: call resctrl_init() */ + err = resctrl_init(); + if (err) + return err; + + WRITE_ONCE(resctrl_enabled, true); return 0; @@ -1629,6 +1654,42 @@ int mpam_resctrl_setup(void) return err; } +void mpam_resctrl_exit(void) +{ + if (!READ_ONCE(resctrl_enabled)) + return; + + WRITE_ONCE(resctrl_enabled, false); + resctrl_exit(); +} + +/* + * The driver is detaching an MSC from this class, if resctrl was using it, + * pull on resctrl_exit(). + */ +void mpam_resctrl_teardown_class(struct mpam_class *class) +{ + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + struct mpam_resctrl_mon *mon; + enum resctrl_event_id eventid; + + might_sleep(); + + for_each_mpam_resctrl_control(res, rid) { + if (res->class == class) { + res->class = NULL; + break; + } + } + for_each_mpam_resctrl_mon(mon, eventid) { + if (mon->class == class) { + mon->class = NULL; + break; + } + } +} + static int __init __cacheinfo_ready(void) { cacheinfo_ready = true; From 0ed6f63fc267d51c704f1108a8e6967cdaa715ac Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Fri, 13 Mar 2026 14:46:12 +0000 Subject: [PATCH 311/464] arm_mpam: Add quirk framework BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAM specification includes the MPAMF_IIDR, which serves to uniquely identify the MSC implementation through a combination of implementer details, product ID, variant, and revision. Certain hardware issues/errata can be resolved using software workarounds. Introduce a quirk framework to allow workarounds to be enabled based on the MPAMF_IIDR value. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Zeng Heng Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Reviewed-by: Gavin Shan Signed-off-by: Shanker Donthineni Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Co-developed-by: James Morse Signed-off-by: James Morse (cherry picked from commit fa7745218c9828ac4849ef62bccad684aec0f422) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 32 ++++++++++++++++++++++++++++++++ drivers/resctrl/mpam_internal.h | 25 +++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 03f769f6864cc..adca8fda05770 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -636,6 +636,30 @@ static struct mpam_msc_ris *mpam_get_or_create_ris(struct mpam_msc *msc, return ERR_PTR(-ENOENT); } +static const struct mpam_quirk mpam_quirks[] = { + { NULL } /* Sentinel */ +}; + +static void mpam_enable_quirks(struct mpam_msc *msc) +{ + const struct mpam_quirk *quirk; + + for (quirk = &mpam_quirks[0]; quirk->iidr_mask; quirk++) { + int err = 0; + + if (quirk->iidr != (msc->iidr & quirk->iidr_mask)) + continue; + + if (quirk->init) + err = quirk->init(msc, quirk); + + if (err) + continue; + + mpam_set_quirk(quirk->workaround, msc); + } +} + /* * IHI009A.a has this nugget: "If a monitor does not support automatic behaviour * of NRDY, software can use this bit for any purpose" - so hardware might not @@ -864,8 +888,11 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc) /* Grab an IDR value to find out how many RIS there are */ mutex_lock(&msc->part_sel_lock); idr = mpam_msc_read_idr(msc); + msc->iidr = mpam_read_partsel_reg(msc, IIDR); mutex_unlock(&msc->part_sel_lock); + mpam_enable_quirks(msc); + msc->ris_max = FIELD_GET(MPAMF_IDR_RIS_MAX, idr); /* Use these values so partid/pmg always starts with a valid value */ @@ -1970,6 +1997,7 @@ static bool mpam_has_cmax_wd_feature(struct mpam_props *props) * resulting safe value must be compatible with both. When merging values in * the tree, all the aliasing resources must be handled first. * On mismatch, parent is modified. + * Quirks on an MSC will apply to all MSC in that class. */ static void __props_mismatch(struct mpam_props *parent, struct mpam_props *child, bool alias) @@ -2089,6 +2117,7 @@ static void __props_mismatch(struct mpam_props *parent, * nobble the class feature, as we can't configure all the resources. * e.g. The L3 cache is composed of two resources with 13 and 17 portion * bitmaps respectively. + * Quirks on an MSC will apply to all MSC in that class. */ static void __class_props_mismatch(struct mpam_class *class, struct mpam_vmsc *vmsc) @@ -2102,6 +2131,9 @@ __class_props_mismatch(struct mpam_class *class, struct mpam_vmsc *vmsc) dev_dbg(dev, "Merging features for class:0x%lx &= vmsc:0x%lx\n", (long)cprops->features, (long)vprops->features); + /* Merge quirks */ + class->quirks |= vmsc->msc->quirks; + /* Take the safe value for any common features */ __props_mismatch(cprops, vprops, false); } diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 3f6390649621a..40377f801f41d 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -85,6 +85,8 @@ struct mpam_msc { u8 pmg_max; unsigned long ris_idxs; u32 ris_max; + u32 iidr; + u16 quirks; /* * error_irq_lock is taken when registering/unregistering the error @@ -214,6 +216,28 @@ struct mpam_props { #define mpam_set_feature(_feat, x) __set_bit(_feat, (x)->features) #define mpam_clear_feature(_feat, x) __clear_bit(_feat, (x)->features) +/* Workaround bits for msc->quirks */ +enum mpam_device_quirks { + MPAM_QUIRK_LAST +}; + +#define mpam_has_quirk(_quirk, x) ((1 << (_quirk) & (x)->quirks)) +#define mpam_set_quirk(_quirk, x) ((x)->quirks |= (1 << (_quirk))) + +struct mpam_quirk { + int (*init)(struct mpam_msc *msc, const struct mpam_quirk *quirk); + + u32 iidr; + u32 iidr_mask; + + enum mpam_device_quirks workaround; +}; + +#define MPAM_IIDR_MATCH_ONE (FIELD_PREP_CONST(MPAMF_IIDR_PRODUCTID, 0xfff) | \ + FIELD_PREP_CONST(MPAMF_IIDR_VARIANT, 0xf) | \ + FIELD_PREP_CONST(MPAMF_IIDR_REVISION, 0xf) | \ + FIELD_PREP_CONST(MPAMF_IIDR_IMPLEMENTER, 0xfff)) + /* The values for MSMON_CFG_MBWU_FLT.RWBW */ enum mon_filter_options { COUNT_BOTH = 0, @@ -257,6 +281,7 @@ struct mpam_class { struct mpam_props props; u32 nrdy_usec; + u16 quirks; u8 level; enum mpam_class_types type; From e50d1b9d0d95090124b3bdcdb88af58426681bf2 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Fri, 13 Mar 2026 14:46:13 +0000 Subject: [PATCH 312/464] arm_mpam: Add workaround for T241-MPAM-1 BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAM bandwidth partitioning controls will not be correctly configured, and hardware will retain default configuration register values, meaning generally that bandwidth will remain unprovisioned. To address the issue, follow the below steps after updating the MBW_MIN and/or MBW_MAX registers. - Perform 64b reads from all 12 bridge MPAM shadow registers at offsets (0x360048 + slice*0x10000 + partid*8). These registers are read-only. - Continue iterating until all 12 shadow register values match in a loop. pr_warn_once if the values fail to match within the loop count 1000. - Perform 64b writes with the value 0x0 to the two spare registers at offsets 0x1b0000 and 0x1c0000. In the hardware, writes to the MPAMCFG_MBW_MAX MPAMCFG_MBW_MIN registers are transformed into broadcast writes to the 12 shadow registers. The final two writes to the spare registers cause a final rank of downstream micro-architectural MPAM registers to be updated from the shadow copies. The intervening loop to read the 12 shadow registers helps avoid a race condition where writes to the spare registers occur before all shadow registers have been updated. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Punit Agrawal Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Gavin Shan Signed-off-by: Shanker Donthineni Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 70e81fbedc6570b2397e07a645136af0a0eec907) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 2 + drivers/resctrl/mpam_devices.c | 88 +++++++++++++++++++++ drivers/resctrl/mpam_internal.h | 9 +++ 3 files changed, 99 insertions(+) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index 4c300caad9011..a65620f98e3aa 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -247,6 +247,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | T241 MPAM | T241-MPAM-1 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+ diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index adca8fda05770..4ccc468f34676 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -29,6 +29,16 @@ #include "mpam_internal.h" +/* Values for the T241 errata workaround */ +#define T241_CHIPS_MAX 4 +#define T241_CHIP_NSLICES 12 +#define T241_SPARE_REG0_OFF 0x1b0000 +#define T241_SPARE_REG1_OFF 0x1c0000 +#define T241_CHIP_ID(phys) FIELD_GET(GENMASK_ULL(44, 43), phys) +#define T241_SHADOW_REG_OFF(sidx, pid) (0x360048 + (sidx) * 0x10000 + (pid) * 8) +#define SMCCC_SOC_ID_T241 0x036b0241 +static void __iomem *t241_scratch_regs[T241_CHIPS_MAX]; + /* * mpam_list_lock protects the SRCU lists when writing. Once the * mpam_enabled key is enabled these lists are read-only, @@ -636,7 +646,45 @@ static struct mpam_msc_ris *mpam_get_or_create_ris(struct mpam_msc *msc, return ERR_PTR(-ENOENT); } +static int mpam_enable_quirk_nvidia_t241_1(struct mpam_msc *msc, + const struct mpam_quirk *quirk) +{ + s32 soc_id = arm_smccc_get_soc_id_version(); + struct resource *r; + phys_addr_t phys; + + /* + * A mapping to a device other than the MSC is needed, check + * SOC_ID is NVIDIA T241 chip (036b:0241) + */ + if (soc_id < 0 || soc_id != SMCCC_SOC_ID_T241) + return -EINVAL; + + r = platform_get_resource(msc->pdev, IORESOURCE_MEM, 0); + if (!r) + return -EINVAL; + + /* Find the internal registers base addr from the CHIP ID */ + msc->t241_id = T241_CHIP_ID(r->start); + phys = FIELD_PREP(GENMASK_ULL(45, 44), msc->t241_id) | 0x19000000ULL; + + t241_scratch_regs[msc->t241_id] = ioremap(phys, SZ_8M); + if (WARN_ON_ONCE(!t241_scratch_regs[msc->t241_id])) + return -EINVAL; + + pr_info_once("Enabled workaround for NVIDIA T241 erratum T241-MPAM-1\n"); + + return 0; +} + static const struct mpam_quirk mpam_quirks[] = { + { + /* NVIDIA t241 erratum T241-MPAM-1 */ + .init = mpam_enable_quirk_nvidia_t241_1, + .iidr = MPAM_IIDR_NVIDIA_T241, + .iidr_mask = MPAM_IIDR_MATCH_ONE, + .workaround = T241_SCRUB_SHADOW_REGS, + }, { NULL } /* Sentinel */ }; @@ -1374,6 +1422,44 @@ static void mpam_reset_msc_bitmap(struct mpam_msc *msc, u16 reg, u16 wd) __mpam_write_reg(msc, reg, bm); } +static void mpam_apply_t241_erratum(struct mpam_msc_ris *ris, u16 partid) +{ + int sidx, i, lcount = 1000; + void __iomem *regs; + u64 val0, val; + + regs = t241_scratch_regs[ris->vmsc->msc->t241_id]; + + for (i = 0; i < lcount; i++) { + /* Read the shadow register at index 0 */ + val0 = readq_relaxed(regs + T241_SHADOW_REG_OFF(0, partid)); + + /* Check if all the shadow registers have the same value */ + for (sidx = 1; sidx < T241_CHIP_NSLICES; sidx++) { + val = readq_relaxed(regs + + T241_SHADOW_REG_OFF(sidx, partid)); + if (val != val0) + break; + } + if (sidx == T241_CHIP_NSLICES) + break; + } + + if (i == lcount) + pr_warn_once("t241: inconsistent values in shadow regs"); + + /* Write a value zero to spare registers to take effect of MBW conf */ + writeq_relaxed(0, regs + T241_SPARE_REG0_OFF); + writeq_relaxed(0, regs + T241_SPARE_REG1_OFF); +} + +static void mpam_quirk_post_config_change(struct mpam_msc_ris *ris, u16 partid, + struct mpam_config *cfg) +{ + if (mpam_has_quirk(T241_SCRUB_SHADOW_REGS, ris->vmsc->msc)) + mpam_apply_t241_erratum(ris, partid); +} + /* Called via IPI. Call while holding an SRCU reference */ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, struct mpam_config *cfg) @@ -1453,6 +1539,8 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, mpam_write_partsel_reg(msc, PRI, pri_val); } + mpam_quirk_post_config_change(ris, partid, cfg); + mutex_unlock(&msc->part_sel_lock); } diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 40377f801f41d..413315ba96320 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -130,6 +130,9 @@ struct mpam_msc { void __iomem *mapped_hwpage; size_t mapped_hwpage_sz; + /* Values only used on some platforms for quirks */ + u32 t241_id; + struct mpam_garbage garbage; }; @@ -218,6 +221,7 @@ struct mpam_props { /* Workaround bits for msc->quirks */ enum mpam_device_quirks { + T241_SCRUB_SHADOW_REGS, MPAM_QUIRK_LAST }; @@ -238,6 +242,11 @@ struct mpam_quirk { FIELD_PREP_CONST(MPAMF_IIDR_REVISION, 0xf) | \ FIELD_PREP_CONST(MPAMF_IIDR_IMPLEMENTER, 0xfff)) +#define MPAM_IIDR_NVIDIA_T241 (FIELD_PREP_CONST(MPAMF_IIDR_PRODUCTID, 0x241) | \ + FIELD_PREP_CONST(MPAMF_IIDR_VARIANT, 0) | \ + FIELD_PREP_CONST(MPAMF_IIDR_REVISION, 0) | \ + FIELD_PREP_CONST(MPAMF_IIDR_IMPLEMENTER, 0x36b)) + /* The values for MSMON_CFG_MBWU_FLT.RWBW */ enum mon_filter_options { COUNT_BOTH = 0, From 84344f524b3cc071e4ca630f4c232e43bb8db420 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Fri, 13 Mar 2026 14:46:14 +0000 Subject: [PATCH 313/464] arm_mpam: Add workaround for T241-MPAM-4 BugLink: https://bugs.launchpad.net/bugs/2154527 In the T241 implementation of memory-bandwidth partitioning, in the absence of contention for bandwidth, the minimum bandwidth setting can affect the amount of achieved bandwidth. Specifically, the achieved bandwidth in the absence of contention can settle to any value between the values of MPAMCFG_MBW_MIN and MPAMCFG_MBW_MAX. Also, if MPAMCFG_MBW_MIN is set zero (below 0.78125%), once a core enters a throttled state, it will never leave that state. The first issue is not a concern if the MPAM software allows to program MPAMCFG_MBW_MIN through the sysfs interface. This patch ensures program MBW_MIN=1 (0.78125%) whenever MPAMCFG_MBW_MIN=0 is programmed. In the scenario where the resctrl doesn't support the MBW_MIN interface via sysfs, to achieve bandwidth closer to MBW_MAX in the absence of contention, software should configure a relatively narrow gap between MBW_MIN and MBW_MAX. The recommendation is to use a 5% gap to mitigate the problem. Clear the feature MBW_MIN feature from the class to ensure we don't accidentally change behaviour when resctrl adds support for a MBW_MIN interface. Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Fenghua Yu Reviewed-by: Gavin Shan Signed-off-by: Shanker Donthineni Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit a7efe23ed6dd08259ad1b238e9c33bb511666fd4) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 2 + drivers/resctrl/mpam_devices.c | 55 +++++++++++++++++++-- drivers/resctrl/mpam_internal.h | 1 + 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index a65620f98e3aa..a4b246655e37e 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -249,6 +249,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 MPAM | T241-MPAM-1 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | T241 MPAM | T241-MPAM-4 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+ diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 4ccc468f34676..809b31a8f6ef7 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -685,6 +685,12 @@ static const struct mpam_quirk mpam_quirks[] = { .iidr_mask = MPAM_IIDR_MATCH_ONE, .workaround = T241_SCRUB_SHADOW_REGS, }, + { + /* NVIDIA t241 erratum T241-MPAM-4 */ + .iidr = MPAM_IIDR_NVIDIA_T241, + .iidr_mask = MPAM_IIDR_MATCH_ONE, + .workaround = T241_FORCE_MBW_MIN_TO_ONE, + }, { NULL } /* Sentinel */ }; @@ -1460,6 +1466,37 @@ static void mpam_quirk_post_config_change(struct mpam_msc_ris *ris, u16 partid, mpam_apply_t241_erratum(ris, partid); } +static u16 mpam_wa_t241_force_mbw_min_to_one(struct mpam_props *props) +{ + u16 max_hw_value, min_hw_granule, res0_bits; + + res0_bits = 16 - props->bwa_wd; + max_hw_value = ((1 << props->bwa_wd) - 1) << res0_bits; + min_hw_granule = ~max_hw_value; + + return min_hw_granule + 1; +} + +static u16 mpam_wa_t241_calc_min_from_max(struct mpam_props *props, + struct mpam_config *cfg) +{ + u16 val = 0; + u16 max; + u16 delta = ((5 * MPAMCFG_MBW_MAX_MAX) / 100) - 1; + + if (mpam_has_feature(mpam_feat_mbw_max, cfg)) { + max = cfg->mbw_max; + } else { + /* Resetting. Hence, use the ris specific default. */ + max = GENMASK(15, 16 - props->bwa_wd); + } + + if (max > delta) + val = max - delta; + + return val; +} + /* Called via IPI. Call while holding an SRCU reference */ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, struct mpam_config *cfg) @@ -1500,9 +1537,18 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, mpam_write_partsel_reg(msc, MBW_PBM, cfg->mbw_pbm); } - if (mpam_has_feature(mpam_feat_mbw_min, rprops) && - mpam_has_feature(mpam_feat_mbw_min, cfg)) - mpam_write_partsel_reg(msc, MBW_MIN, 0); + if (mpam_has_feature(mpam_feat_mbw_min, rprops)) { + u16 val = 0; + + if (mpam_has_quirk(T241_FORCE_MBW_MIN_TO_ONE, msc)) { + u16 min = mpam_wa_t241_force_mbw_min_to_one(rprops); + + val = mpam_wa_t241_calc_min_from_max(rprops, cfg); + val = max(val, min); + } + + mpam_write_partsel_reg(msc, MBW_MIN, val); + } if (mpam_has_feature(mpam_feat_mbw_max, rprops)) { if (mpam_has_feature(mpam_feat_mbw_max, cfg)) @@ -2286,6 +2332,9 @@ static void mpam_enable_merge_class_features(struct mpam_component *comp) list_for_each_entry(vmsc, &comp->vmsc, comp_list) __class_props_mismatch(class, vmsc); + + if (mpam_has_quirk(T241_FORCE_MBW_MIN_TO_ONE, class)) + mpam_clear_feature(mpam_feat_mbw_min, &class->props); } /* diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 413315ba96320..15088e55b0bdc 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -222,6 +222,7 @@ struct mpam_props { /* Workaround bits for msc->quirks */ enum mpam_device_quirks { T241_SCRUB_SHADOW_REGS, + T241_FORCE_MBW_MIN_TO_ONE, MPAM_QUIRK_LAST }; From 5f1b4f92f84141c87574bd0a771eca877f6bf189 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Fri, 13 Mar 2026 14:46:15 +0000 Subject: [PATCH 314/464] arm_mpam: Add workaround for T241-MPAM-6 BugLink: https://bugs.launchpad.net/bugs/2154527 The registers MSMON_MBWU_L and MSMON_MBWU return the number of requests rather than the number of bytes transferred. Bandwidth resource monitoring is performed at the last level cache, where each request arrive in 64Byte granularity. The current implementation returns the number of transactions received at the last level cache but does not provide the value in bytes. Scaling by 64 gives an accurate byte count to match the MPAM specification for the MSMON_MBWU and MSMON_MBWU_L registers. This patch fixes the issue by reporting the actual number of bytes instead of the number of transactions from __ris_msmon_read(). Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Punit Agrawal Tested-by: Peter Newman Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Gavin Shan Signed-off-by: Shanker Donthineni Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit dc48eb1ff27cc3169c3c5cca5eb20645d04d9e22) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 2 ++ drivers/resctrl/mpam_devices.c | 26 +++++++++++++++++++-- drivers/resctrl/mpam_internal.h | 1 + 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index a4b246655e37e..1aa3326bb3200 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -251,6 +251,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 MPAM | T241-MPAM-4 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | T241 MPAM | T241-MPAM-6 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+ | Freescale/NXP | LS2080A/LS1043A | A-008585 | FSL_ERRATUM_A008585 | +----------------+-----------------+-----------------+-----------------------------+ diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 809b31a8f6ef7..7419e2135ad15 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -691,6 +691,12 @@ static const struct mpam_quirk mpam_quirks[] = { .iidr_mask = MPAM_IIDR_MATCH_ONE, .workaround = T241_FORCE_MBW_MIN_TO_ONE, }, + { + /* NVIDIA t241 erratum T241-MPAM-6 */ + .iidr = MPAM_IIDR_NVIDIA_T241, + .iidr_mask = MPAM_IIDR_MATCH_ONE, + .workaround = T241_MBW_COUNTER_SCALE_64, + }, { NULL } /* Sentinel */ }; @@ -1146,7 +1152,7 @@ static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val, } } -static u64 mpam_msmon_overflow_val(enum mpam_device_features type) +static u64 __mpam_msmon_overflow_val(enum mpam_device_features type) { /* TODO: implement scaling counters */ switch (type) { @@ -1161,6 +1167,18 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type) } } +static u64 mpam_msmon_overflow_val(enum mpam_device_features type, + struct mpam_msc *msc) +{ + u64 overflow_val = __mpam_msmon_overflow_val(type); + + if (mpam_has_quirk(T241_MBW_COUNTER_SCALE_64, msc) && + type != mpam_feat_msmon_mbwu_63counter) + overflow_val *= 64; + + return overflow_val; +} + static void __ris_msmon_read(void *arg) { u64 now; @@ -1247,13 +1265,17 @@ static void __ris_msmon_read(void *arg) now = FIELD_GET(MSMON___VALUE, now); } + if (mpam_has_quirk(T241_MBW_COUNTER_SCALE_64, msc) && + m->type != mpam_feat_msmon_mbwu_63counter) + now *= 64; + if (nrdy) break; mbwu_state = &ris->mbwu_state[ctx->mon]; if (overflow) - mbwu_state->correction += mpam_msmon_overflow_val(m->type); + mbwu_state->correction += mpam_msmon_overflow_val(m->type, msc); /* * Include bandwidth consumed before the last hardware reset and diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 15088e55b0bdc..1af7f640037a4 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -223,6 +223,7 @@ struct mpam_props { enum mpam_device_quirks { T241_SCRUB_SHADOW_REGS, T241_FORCE_MBW_MIN_TO_ONE, + T241_MBW_COUNTER_SCALE_64, MPAM_QUIRK_LAST }; From 1014dd0962f95b656c9933a85c5b7ac92b054334 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 13 Mar 2026 14:46:16 +0000 Subject: [PATCH 315/464] arm_mpam: Quirk CMN-650's CSU NRDY behaviour BugLink: https://bugs.launchpad.net/bugs/2154527 CMN-650 is afflicted with an erratum where the CSU NRDY bit never clears. This tells us the monitor never finishes scanning the cache. The erratum document says to wait the maximum time, then ignore the field. Add a flag to indicate whether this is the final attempt to read the counter, and when this quirk is applied, ignore the NRDY field. This means accesses to this counter will always retry, even if the counter was previously programmed to the same values. The counter value is not expected to be stable, it drifts up and down with each allocation and eviction. The CSU register provides the value for a point in time. Tested-by: Punit Agrawal Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Reviewed-by: Zeng Heng Reviewed-by: Gavin Shan Co-developed-by: Ben Horgan Signed-off-by: Ben Horgan Signed-off-by: James Morse (cherry picked from commit aeb8595a5f8ba4aac8b5c265a8bcc3f18b473cb5) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 3 +++ drivers/resctrl/mpam_devices.c | 12 ++++++++++++ drivers/resctrl/mpam_internal.h | 6 ++++++ 3 files changed, 21 insertions(+) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index 1aa3326bb3200..65ed6ea33751f 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -214,6 +214,9 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | SI L1 | #4311569 | ARM64_ERRATUM_4311569 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | CMN-650 | #3642720 | N/A | ++----------------+-----------------+-----------------+-----------------------------+ ++----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_845719 | +----------------+-----------------+-----------------+-----------------------------+ | Broadcom | Brahma-B53 | N/A | ARM64_ERRATUM_843419 | diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 7419e2135ad15..988fc291241d2 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -697,6 +697,12 @@ static const struct mpam_quirk mpam_quirks[] = { .iidr_mask = MPAM_IIDR_MATCH_ONE, .workaround = T241_MBW_COUNTER_SCALE_64, }, + { + /* ARM CMN-650 CSU erratum 3642720 */ + .iidr = MPAM_IIDR_ARM_CMN_650, + .iidr_mask = MPAM_IIDR_MATCH_ONE, + .workaround = IGNORE_CSU_NRDY, + }, { NULL } /* Sentinel */ }; @@ -1003,6 +1009,7 @@ struct mon_read { enum mpam_device_features type; u64 *val; int err; + bool waited_timeout; }; static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris) @@ -1247,6 +1254,10 @@ static void __ris_msmon_read(void *arg) now = mpam_read_monsel_reg(msc, CSU); nrdy = now & MSMON___NRDY; now = FIELD_GET(MSMON___VALUE, now); + + if (mpam_has_quirk(IGNORE_CSU_NRDY, msc) && m->waited_timeout) + nrdy = false; + break; case mpam_feat_msmon_mbwu_31counter: case mpam_feat_msmon_mbwu_44counter: @@ -1382,6 +1393,7 @@ int mpam_msmon_read(struct mpam_component *comp, struct mon_cfg *ctx, .ctx = ctx, .type = type, .val = val, + .waited_timeout = true, }; *val = 0; diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 1af7f640037a4..04d1a59f02afb 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -224,6 +224,7 @@ enum mpam_device_quirks { T241_SCRUB_SHADOW_REGS, T241_FORCE_MBW_MIN_TO_ONE, T241_MBW_COUNTER_SCALE_64, + IGNORE_CSU_NRDY, MPAM_QUIRK_LAST }; @@ -249,6 +250,11 @@ struct mpam_quirk { FIELD_PREP_CONST(MPAMF_IIDR_REVISION, 0) | \ FIELD_PREP_CONST(MPAMF_IIDR_IMPLEMENTER, 0x36b)) +#define MPAM_IIDR_ARM_CMN_650 (FIELD_PREP_CONST(MPAMF_IIDR_PRODUCTID, 0) | \ + FIELD_PREP_CONST(MPAMF_IIDR_VARIANT, 0) | \ + FIELD_PREP_CONST(MPAMF_IIDR_REVISION, 0) | \ + FIELD_PREP_CONST(MPAMF_IIDR_IMPLEMENTER, 0x43b)) + /* The values for MSMON_CFG_MBWU_FLT.RWBW */ enum mon_filter_options { COUNT_BOTH = 0, From 11485cfea20e6615772e09e4821266e8e801e850 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Fri, 13 Mar 2026 14:46:17 +0000 Subject: [PATCH 316/464] arm64: mpam: Add initial MPAM documentation BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM (Memory Partitioning and Monitoring) is now exposed to user-space via resctrl. Add some documentation so the user knows what features to expect. Reviewed-by: Zeng Heng Reviewed-by: Shaopeng Tan Reviewed-by: Jonathan Cameron Acked-by: Catalin Marinas Signed-off-by: Ben Horgan Reviewed-by: Gavin Shan Tested-by: Gavin Shan Tested-by: Shaopeng Tan Tested-by: Jesse Chick Signed-off-by: James Morse (cherry picked from commit 4ce0a2ccc0358f3f746fa50815a599f861fd5d68) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/arch/arm64/index.rst | 1 + Documentation/arch/arm64/mpam.rst | 72 ++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 Documentation/arch/arm64/mpam.rst diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst index af52edc8c0ac6..98052b4ef4a1e 100644 --- a/Documentation/arch/arm64/index.rst +++ b/Documentation/arch/arm64/index.rst @@ -23,6 +23,7 @@ ARM64 Architecture memory memory-tagging-extension mops + mpam perf pointer-authentication ptdump diff --git a/Documentation/arch/arm64/mpam.rst b/Documentation/arch/arm64/mpam.rst new file mode 100644 index 0000000000000..570f51a8d4ebf --- /dev/null +++ b/Documentation/arch/arm64/mpam.rst @@ -0,0 +1,72 @@ +.. SPDX-License-Identifier: GPL-2.0 + +==== +MPAM +==== + +What is MPAM +============ +MPAM (Memory Partitioning and Monitoring) is a feature in the CPUs and memory +system components such as the caches or memory controllers that allow memory +traffic to be labelled, partitioned and monitored. + +Traffic is labelled by the CPU, based on the control or monitor group the +current task is assigned to using resctrl. Partitioning policy can be set +using the schemata file in resctrl, and monitor values read via resctrl. +See Documentation/filesystems/resctrl.rst for more details. + +This allows tasks that share memory system resources, such as caches, to be +isolated from each other according to the partitioning policy (so called noisy +neighbours). + +Supported Platforms +=================== +Use of this feature requires CPU support, support in the memory system +components, and a description from firmware of where the MPAM device controls +are in the MMIO address space. (e.g. the 'MPAM' ACPI table). + +The MMIO device that provides MPAM controls/monitors for a memory system +component is called a memory system component. (MSC). + +Because the user interface to MPAM is via resctrl, only MPAM features that are +compatible with resctrl can be exposed to user-space. + +MSC are considered as a group based on the topology. MSC that correspond with +the L3 cache are considered together, it is not possible to mix MSC between L2 +and L3 to 'cover' a resctrl schema. + +The supported features are: + +* Cache portion bitmap controls (CPOR) on the L2 or L3 caches. To expose + CPOR at L2 or L3, every CPU must have a corresponding CPU cache at this + level that also supports the feature. Mismatched big/little platforms are + not supported as resctrl's controls would then also depend on task + placement. + +* Memory bandwidth maximum controls (MBW_MAX) on or after the L3 cache. + resctrl uses the L3 cache-id to identify where the memory bandwidth + control is applied. For this reason the platform must have an L3 cache + with cache-id's supplied by firmware. (It doesn't need to support MPAM.) + + To be exported as the 'MB' schema, the topology of the group of MSC chosen + must match the topology of the L3 cache so that the cache-id's can be + repainted. For example: Platforms with Memory bandwidth maximum controls + on CPU-less NUMA nodes cannot expose the 'MB' schema to resctrl as these + nodes do not have a corresponding L3 cache. If the memory bandwidth + control is on the memory rather than the L3 then there must be a single + global L3 as otherwise it is unknown which L3 the traffic came from. There + must be no caches between the L3 and the memory so that the two ends of + the path have equivalent traffic. + + When the MPAM driver finds multiple groups of MSC it can use for the 'MB' + schema, it prefers the group closest to the L3 cache. + +* Cache Storage Usage (CSU) counters can expose the 'llc_occupancy' provided + there is at least one CSU monitor on each MSC that makes up the L3 group. + Exposing CSU counters from other caches or devices is not supported. + +Reporting Bugs +============== +If you are not seeing the counters or controls you expect please share the +debug messages produced when enabling dynamic debug and booting with: +dyndbg="file mpam_resctrl.c +pl" From faa716771643df7142fe578346029b24ff27129f Mon Sep 17 00:00:00 2001 From: Aaron Tomlin Date: Tue, 24 Mar 2026 20:11:59 -0400 Subject: [PATCH 317/464] fs/resctrl: Add "*" shorthand to set io_alloc CBM for all domains BugLink: https://bugs.launchpad.net/bugs/2154527 Configuring the io_alloc_cbm interface requires an explicit domain ID for each cache domain. On systems with high core counts and numerous cache clusters, this requirement becomes cumbersome for automation and management tasks that aim to apply a uniform policy. Introduce a wildcard domain ID selector "*" for the io_alloc_cbm interface. This enables users to set the same Capacity Bitmask (CBM) across all cache domains in a single operation. Signed-off-by: Aaron Tomlin Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Reinette Chatre Reviewed-by: Babu Moger Tested-by: Babu Moger Link: https://patch.msgid.link/20260325001159.447075-3-atomlin@atomlin.com (cherry picked from commit d2bf45d067c728b0fe6e8f99a7386b8291e391e3) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/filesystems/resctrl.rst | 8 ++++++++ fs/resctrl/ctrlmondata.c | 21 +++++++++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst index ba609f8d4de57..b003bed339fdd 100644 --- a/Documentation/filesystems/resctrl.rst +++ b/Documentation/filesystems/resctrl.rst @@ -215,6 +215,14 @@ related to allocation: # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=00ff;1=000f + An ID of "*" configures all domains with the provided CBM. + + Example on a system that does not require a minimum number of consecutive bits in the mask:: + + # echo "*=0" > /sys/fs/resctrl/info/L3/io_alloc_cbm + # cat /sys/fs/resctrl/info/L3/io_alloc_cbm + 0=0;1=0 + When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE resources may reflect the same values. For example, values read from and written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 2ef53161ce119..9a7dfc48cb2e2 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -954,25 +954,34 @@ static int resctrl_io_alloc_parse_line(char *line, struct rdt_resource *r, struct resctrl_schema *s, u32 closid) { enum resctrl_conf_type peer_type; + unsigned long dom_id = ULONG_MAX; struct rdt_parse_data data; struct rdt_ctrl_domain *d; + bool update_all = false; char *dom = NULL, *id; - unsigned long dom_id; next: if (!line || line[0] == '\0') return 0; + if (update_all) { + rdt_last_cmd_puts("Configurations after global '*'\n"); + return -EINVAL; + } + dom = strsep(&line, ";"); id = strsep(&dom, "="); - if (!dom || kstrtoul(id, 10, &dom_id)) { + + if (dom && !strcmp(id, "*")) { + update_all = true; + } else if (!dom || kstrtoul(id, 10, &dom_id)) { rdt_last_cmd_puts("Missing '=' or non-numeric domain\n"); return -EINVAL; } dom = strim(dom); list_for_each_entry(d, &r->ctrl_domains, hdr.list) { - if (d->hdr.id == dom_id) { + if (update_all || d->hdr.id == dom_id) { data.buf = dom; data.mode = RDT_MODE_SHAREABLE; data.closid = closid; @@ -988,10 +997,14 @@ static int resctrl_io_alloc_parse_line(char *line, struct rdt_resource *r, &d->staged_config[s->conf_type], sizeof(d->staged_config[0])); } - goto next; + if (!update_all) + goto next; } } + if (update_all) + goto next; + rdt_last_cmd_printf("Invalid domain %lu\n", dom_id); return -EINVAL; } From 34400e7c58eef1e7a84f31652e7276c8b41ac69d Mon Sep 17 00:00:00 2001 From: Reinette Chatre Date: Tue, 7 Apr 2026 09:01:58 -0700 Subject: [PATCH 318/464] MAINTAINERS: Update resctrl entry BugLink: https://bugs.launchpad.net/bugs/2154527 The x86 maintainers handle the resctrl filesystem and x86 architectural resctrl code. Even so, the x86 maintainers are not part of the resctrl section and not returned when scripts/get_maintainer.pl is run on resctrl filesystem code. With patches flowing via x86 maintainers resctrl should also ensure it follows the tip rules. Add the x86 maintainer alias, x86@kernel.org, to the resctrl section to ensure x86 maintainers are included in associated resctrl submissions. Add a reference to the tip tree handbook to make it clear which rules resctrl follows. Signed-off-by: Reinette Chatre Signed-off-by: Borislav Petkov (AMD) Link: https://patch.msgid.link/4c14dd82e81737c6413e10fe097475b1cc0886fc.1775576382.git.reinette.chatre@intel.com (cherry picked from commit c611752be9d73d12fca9b456a0b8f5c8409a2346) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 85bbf2d242458..b9028c49b421e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -22198,11 +22198,13 @@ F: tools/testing/selftests/net/rds/ RDT - RESOURCE ALLOCATION M: Tony Luck M: Reinette Chatre +M: x86@kernel.org R: Dave Martin R: James Morse R: Babu Moger L: linux-kernel@vger.kernel.org S: Supported +P: Documentation/process/maintainer-tip.rst F: Documentation/filesystems/resctrl.rst F: arch/x86/include/asm/resctrl.h F: arch/x86/kernel/cpu/resctrl/ From e4474235b105466169c8fc5bffd87d3f5221f766 Mon Sep 17 00:00:00 2001 From: Reinette Chatre Date: Tue, 7 Apr 2026 09:01:59 -0700 Subject: [PATCH 319/464] fs/resctrl: Add missing return value descriptions BugLink: https://bugs.launchpad.net/bugs/2154527 Using the stricter "./tools/docs/kernel-doc -Wall -v" to verify proper formatting of documentation comments includes warnings related to return markup on functions that are omitted during the default verification checks. This stricter verification reports a couple of missing return descriptions in resctrl: Warning: .../fs/resctrl/rdtgroup.c:1536 No description found for return value of 'rdtgroup_cbm_to_size' Warning: .../fs/resctrl/rdtgroup.c:3131 No description found for return value of 'mon_get_kn_priv' Warning: .../fs/resctrl/rdtgroup.c:3523 No description found for return value of 'cbm_ensure_valid' Warning: .../fs/resctrl/monitor.c:238 No description found for return value of 'resctrl_find_cleanest_closid' Add the missing return descriptions. Signed-off-by: Reinette Chatre Signed-off-by: Borislav Petkov (AMD) Link: https://patch.msgid.link/1c50b9f7c73251c007133590986f127e1af57780.1775576382.git.reinette.chatre@intel.com (cherry picked from commit 79727019ce3da234d877ec0cb6a3985f001e2b2d) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/monitor.c | 2 ++ fs/resctrl/rdtgroup.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c index 49f3f6b846b27..9fd901c78dc66 100644 --- a/fs/resctrl/monitor.c +++ b/fs/resctrl/monitor.c @@ -234,6 +234,8 @@ static struct rmid_entry *resctrl_find_free_rmid(u32 closid) * * When the CLOSID and RMID are independent numbers, the first free CLOSID will * be returned. + * + * Return: Free CLOSID on success, < 0 on failure. */ int resctrl_find_cleanest_closid(void) { diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 5da305bd36c96..5dfdaa6f9d8ff 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1519,6 +1519,8 @@ static ssize_t rdtgroup_mode_write(struct kernfs_open_file *of, * * @cbm is unsigned long, even if only 32 bits are used to make the * bitmap functions work correctly. + * + * Return: Size (in bytes) of cache portion represented by CBM, 0 on failure. */ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r, struct rdt_ctrl_domain *d, unsigned long cbm) @@ -3102,6 +3104,8 @@ static void rmdir_all_sub(void) * @mevt: The type of event file being created. * @do_sum: Whether SNC summing monitors are being created. Only set * when @rid == RDT_RESOURCE_L3. + * + * Return: Pointer to mon_data private data of the event, NULL on failure. */ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid, struct mon_evt *mevt, @@ -3496,6 +3500,8 @@ static int mkdir_mondata_all(struct kernfs_node *parent_kn, * resource group is initialized. The user can follow this with a * modification to the CBM if the default does not satisfy the * requirements. + * + * Return: A CBM that is valid for resource @r. */ static u32 cbm_ensure_valid(u32 _val, struct rdt_resource *r) { From 4fc9f0bdedd84e332f79e31b2819931f071e92a5 Mon Sep 17 00:00:00 2001 From: Zeng Heng Date: Mon, 13 Apr 2026 17:00:41 +0800 Subject: [PATCH 320/464] arm_mpam: resctrl: Fix MBA CDP alloc_capable handling on unmount BugLink: https://bugs.launchpad.net/bugs/2154527 The code to set MBA's alloc_capable to true appears to be trying to restore alloc_capable on unmount. This can never work because resctrl_arch_set_cdp_enabled() is never invoked with RDT_RESOURCE_MBA as the rid parameter. Consequently, mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled always remains false. The alloc_capable setting in resctrl_arch_set_cdp_enabled() is to re-enable MBA if the caller opts in to separate control values using CDP for this resource. This doesn't happen today. Add a comment to describe this. However a bug remains where MBA allocation is permanently disabled after the mount with CDP option. Remounting without CDP cannot restore the MBA partition capability. Add a check to re-enable MBA when CDP is disabled, which happens on unmount. Fixes: 6789fb99282c ("arm_mpam: resctrl: Add CDP emulation") Signed-off-by: Zeng Heng [ morse: Added comment for existing code, added hunk to fix this bug from Ben H ] Reviewed-by: James Morse Signed-off-by: James Morse (cherry picked from commit f758340da529ccb12531c3f83d5992e912f6c8d5) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index a9938006d0e6e..4205fb2ee312b 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -220,10 +220,18 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) if (cdp_enabled && !mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled) mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = false; + /* + * If resctrl has attempted to enable CDP on MBA, re-enable MBA as two + * configurations will be provided so there is no aliasing problem. + */ if (mpam_resctrl_controls[RDT_RESOURCE_MBA].cdp_enabled && mpam_resctrl_controls[RDT_RESOURCE_MBA].class) mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = true; + /* On unmount when CDP is disabled, re-enable MBA */ + if (!cdp_enabled && mpam_resctrl_controls[RDT_RESOURCE_MBA].class) + mpam_resctrl_controls[RDT_RESOURCE_MBA].resctrl_res.alloc_capable = true; + if (enable) { if (mpam_partid_max < 1) return -EINVAL; From 655eca2be0d8637196062ec5493c1cc1dab9934d Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Tue, 14 Apr 2026 14:27:56 +0100 Subject: [PATCH 321/464] arm_mpam: resctrl: Fix the check for no monitor components found BugLink: https://bugs.launchpad.net/bugs/2154527 Dan Carpenter reports that, in mpam_resctrl_alloc_domain(), any_mon_comp is used in an 'if' condition when it may be uninitialized. Initialize it to NULL so that the check behaves correctly when no monitor components are found. Reported-by: Dan Carpenter Fixes: 264c285999fc ("arm_mpam: resctrl: Add monitor initialisation and domain boilerplate") Signed-off-by: Ben Horgan Reviewed-by: Gavin Shan Signed-off-by: James Morse (cherry picked from commit 67c0a487efa542cca9477ea84915db2e091f98d0) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 4205fb2ee312b..1b0b37da12afc 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1407,7 +1407,7 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) } if (r->mon_capable) { - struct mpam_component *any_mon_comp; + struct mpam_component *any_mon_comp = NULL; struct mpam_resctrl_mon *mon; enum resctrl_event_id eventid; From fcef2266fa571f9d64a57c5b7af49fce6f5dfce7 Mon Sep 17 00:00:00 2001 From: Ben Horgan Date: Tue, 14 Apr 2026 14:27:58 +0100 Subject: [PATCH 322/464] arm_mpam: resctrl: Make resctrl_mon_ctx_waiters static BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl_mon_ctx_waiters is not used outside of this file, so make it static. This fixes the sparse warning: drivers/resctrl/mpam_resctrl.c:25:1: warning: symbol 'resctrl_mon_ctx_waiters' was not declared. Should it be static? Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202603281842.c2K96tJA-lkp@intel.com/ Fixes: 2a3c79c61539 ("arm_mpam: resctrl: Allow resctrl to allocate monitors") Signed-off-by: Ben Horgan Reviewed-by: Gavin Shan Signed-off-by: James Morse (cherry picked from commit 4d5bbbafc170eb21474a37d844211fce6b0f3c51) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 1b0b37da12afc..226ff6f532fab 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -22,7 +22,7 @@ #include "mpam_internal.h" -DECLARE_WAIT_QUEUE_HEAD(resctrl_mon_ctx_waiters); +static DECLARE_WAIT_QUEUE_HEAD(resctrl_mon_ctx_waiters); /* * The classes we've picked to map to resctrl resources, wrapped From 6df337e5a4e0f631685aea22fc131df59ca25c4b Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Thu, 30 Apr 2026 01:19:15 +0000 Subject: [PATCH 323/464] NVIDIA: SAUCE: Update annotations to set CONFIG_RESCTRL_FS BugLink: https://bugs.launchpad.net/bugs/2154527 Eanble resctrl by CONFIG_RESCTRL_FS=y Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- debian.nvidia-bos/config/annotations | 2 ++ 1 file changed, 2 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 562d154cdd155..3787b2b11b2c2 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -282,6 +282,8 @@ CONFIG_VFIO_CONTAINER note<'LP: #2095028'> CONFIG_VFIO_IOMMU_TYPE1 policy<{'amd64': 'm', 'arm64': '-'}> CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> +CONFIG_RESCTRL_FS policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_RESCTRL_FS note<'LP: #2122432'> # ---- Annotations without notes ---- From eaff6c2bde68fbca18ce8ccfadc1a5831228452b Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 7 Sep 2021 17:21:42 +0100 Subject: [PATCH 324/464] NVIDIA: SAUCE: untested: arm_mpam: resctrl: pick classes for use as mbm counters BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl has two types of counters, NUMA-local and global. MPAM has only bandwidth counters, but the position of the MSC may mean it counts NUMA-local, or global traffic. But the topology information is not available. Apply a heuristic: the L2 or L3 supports bandwidth monitors, these are probably NUMA-local. If the memory controller supports bandwidth monitors, they are probably global. This also allows us to assert that we don't have the same class backing two different resctrl events. Because the class or component backing the event may not be 'the L3', it is necessary for mpam_resctrl_get_domain_from_cpu() to search the monitor domains too. This matters the most for 'monitor only' systems, where 'the L3' control domains may be empty, and the ctrl_comp pointer NULL. Signed-off-by: James Morse (cherry picked from commit 40e0b0792745d65ea76f7b28f2642c590fe4dd9a https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - mon_comp[] is defined in upstream. Remove its definition in this patch. - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 57 ++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 226ff6f532fab..a5463c59910b4 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -75,6 +75,14 @@ static DECLARE_WAIT_QUEUE_HEAD(wait_cacheinfo_ready); */ static bool resctrl_enabled; +/* Whether this num_mbw_mon could result in a free_running system */ +static int __mpam_monitors_free_running(u16 num_mbwu_mon) +{ + if (num_mbwu_mon >= resctrl_arch_system_num_rmid_idx()) + return resctrl_arch_system_num_rmid_idx(); + return 0; +} + bool resctrl_arch_alloc_capable(void) { struct mpam_resctrl_res *res; @@ -606,6 +614,24 @@ static bool cache_has_usable_csu(struct mpam_class *class) return true; } +static bool class_has_usable_mbwu(struct mpam_class *class) +{ + struct mpam_props *cprops = &class->props; + + if (!mpam_has_feature(mpam_feat_msmon_mbwu, cprops)) + return false; + + /* + * resctrl expects the bandwidth counters to be free running, + * which means we need as many monitors as resctrl has + * control/monitor groups. + */ + if (!__mpam_monitors_free_running(cprops->num_mbwu_mon)) + return false; + + return true; +} + /* * Calculate the worst-case percentage change from each implemented step * in the control. @@ -949,6 +975,7 @@ static void counter_update_class(enum resctrl_event_id evt_id, static void mpam_resctrl_pick_counters(void) { struct mpam_class *class; + bool has_mbwu; lockdep_assert_cpus_held(); @@ -983,7 +1010,37 @@ static void mpam_resctrl_pick_counters(void) break; } } + + has_mbwu = class_has_usable_mbwu(class); + if (has_mbwu && topology_matches_l3(class)) { + pr_debug("class %u has usable MBWU, and matches L3 topology", class->level); + + /* + * MBWU counters may be 'local' or 'total' depending on + * where they are in the topology. Counters on caches + * are assumed to be local. If it's on the memory + * controller, its assumed to be global. + * TODO: check mbm_local matches NUMA boundaries... + */ + switch (class->type) { + case MPAM_CLASS_CACHE: + counter_update_class(QOS_L3_MBM_LOCAL_EVENT_ID, + class); + break; + case MPAM_CLASS_MEMORY: + counter_update_class(QOS_L3_MBM_TOTAL_EVENT_ID, + class); + break; + default: + break; + } + } } + + /* Allocation of MBWU monitors assumes that the class is unique... */ + if (mpam_resctrl_counters[QOS_L3_MBM_LOCAL_EVENT_ID].class) + WARN_ON_ONCE(mpam_resctrl_counters[QOS_L3_MBM_LOCAL_EVENT_ID].class == + mpam_resctrl_counters[QOS_L3_MBM_TOTAL_EVENT_ID].class); } static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) From 0ffb5dceee04cf048c4ec59151962479091aac7c Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 15 Jul 2025 15:39:36 +0100 Subject: [PATCH 325/464] NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate free running monitors BugLink: https://bugs.launchpad.net/bugs/2154527 When there are enough monitors, the resctrl mbm local and total files can be exposed. These need all the monitors that resctrl may use to be allocated up front. Add helpers to do this. If a different candidate class is discovered, the old array should be free'd and the allocated monitors returned to the driver. Signed-off-by: James Morse (cherry picked from commit 355bc5f578a4f17887f2574191c01fae5202abd7 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_internal.h | 8 ++- drivers/resctrl/mpam_resctrl.c | 92 +++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 6 deletions(-) diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 04d1a59f02afb..816a8245f67b0 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -409,7 +409,13 @@ struct mpam_resctrl_res { struct mpam_resctrl_mon { struct mpam_class *class; - /* per-class data that resctrl needs will live here */ + /* + * Array of allocated MBWU monitors, indexed by (closid, rmid). + * When ABMC is not in use, this array directly maps (closid, rmid) + * to the allocated monitor. Otherwise this array is sparse, and + * un-assigned (closid, rmid) are -1. + */ + int *mbwu_idx_to_mon; }; static inline int mpam_alloc_csu_mon(struct mpam_class *class) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index a5463c59910b4..82801feb5211b 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -626,10 +626,12 @@ static bool class_has_usable_mbwu(struct mpam_class *class) * which means we need as many monitors as resctrl has * control/monitor groups. */ - if (!__mpam_monitors_free_running(cprops->num_mbwu_mon)) - return false; + if (__mpam_monitors_free_running(cprops->num_mbwu_mon)) { + pr_debug("monitors usable in free-running mode\n"); + return true; + } - return true; + return false; } /* @@ -951,10 +953,58 @@ static void mpam_resctrl_pick_mba(void) } } +static void __free_mbwu_mon(struct mpam_class *class, int *array, + u16 num_mbwu_mon) +{ + for (int i = 0; i < num_mbwu_mon; i++) { + if (array[i] < 0) + continue; + + mpam_free_mbwu_mon(class, array[i]); + array[i] = ~0; + } +} + +static int __alloc_mbwu_mon(struct mpam_class *class, int *array, + u16 num_mbwu_mon) +{ + for (int i = 0; i < num_mbwu_mon; i++) { + int mbwu_mon = mpam_alloc_mbwu_mon(class); + + if (mbwu_mon < 0) { + __free_mbwu_mon(class, array, num_mbwu_mon); + return mbwu_mon; + } + array[i] = mbwu_mon; + } + + return 0; +} + +static int *__alloc_mbwu_array(struct mpam_class *class, u16 num_mbwu_mon) +{ + int err; + size_t array_size = num_mbwu_mon * sizeof(int); + int *array __free(kfree) = kmalloc(array_size, GFP_KERNEL); + + if (!array) + return ERR_PTR(-ENOMEM); + + memset(array, -1, array_size); + + err = __alloc_mbwu_mon(class, array, num_mbwu_mon); + if (err) + return ERR_PTR(err); + return_ptr(array); +} + static void counter_update_class(enum resctrl_event_id evt_id, struct mpam_class *class) { - struct mpam_class *existing_class = mpam_resctrl_counters[evt_id].class; + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evt_id]; + struct mpam_class *existing_class = mon->class; + u16 num_mbwu_mon = class->props.num_mbwu_mon; + int *existing_array = mon->mbwu_idx_to_mon; if (existing_class) { if (class->level == 3) { @@ -969,7 +1019,39 @@ static void counter_update_class(enum resctrl_event_id evt_id, } } - mpam_resctrl_counters[evt_id].class = class; + pr_debug("Updating event %u to use class %u\n", evt_id, class->level); + mon->class = class; + + if (evt_id == QOS_L3_OCCUP_EVENT_ID) + return; + + /* Might not need all the monitors */ + num_mbwu_mon = __mpam_monitors_free_running(num_mbwu_mon); + if (!num_mbwu_mon) { + pr_debug("Not pre-allocating free-running counters\n"); + return; + } + + /* + * This is the pre-allocated free-running monitors path. It always + * allocates one monitor per PARTID * PMG. + */ + WARN_ON_ONCE(num_mbwu_mon != resctrl_arch_system_num_rmid_idx()); + + mon->mbwu_idx_to_mon = __alloc_mbwu_array(class, num_mbwu_mon); + if (IS_ERR(mon->mbwu_idx_to_mon)) { + pr_debug("Failed to allocate MBWU array\n"); + mon->class = existing_class; + mon->mbwu_idx_to_mon = existing_array; + return; + } + + if (existing_array) { + pr_debug("Releasing previous class %u's monitors\n", + existing_class->level); + __free_mbwu_mon(existing_class, existing_array, num_mbwu_mon); + kfree(existing_array); + } } static void mpam_resctrl_pick_counters(void) From 2d109004824fc4b397387fce03e25874da45eec3 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 15 Oct 2025 12:33:20 +0100 Subject: [PATCH 326/464] NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate assignable monitors BugLink: https://bugs.launchpad.net/bugs/2154527 When there are not enough monitors, MPAM is able to emulate ABMC by making a smaller number of monitors assignable. These monitors still need to be allocated from the driver, and mapped to whichever control/monitor group resctrl wants to use them with. Add a second array to hold the monitor values indexed by resctrl's cntr_id. When CDP is in use, two monitors are needed so the available number of counters halves. Platforms witih one monitor will have zero monitors when CDP is in use. Signed-off-by: James Morse (forward ported from commit d8a0ad3da1831147810bb58fc2459a6e36e26873 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; - Report returned value from mpam_resctrl_monitor_init_abmc(); ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_internal.h | 7 +++ drivers/resctrl/mpam_resctrl.c | 108 +++++++++++++++++++++++++++++--- 2 files changed, 105 insertions(+), 10 deletions(-) diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 816a8245f67b0..60a8cebb33882 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -416,6 +416,13 @@ struct mpam_resctrl_mon { * un-assigned (closid, rmid) are -1. */ int *mbwu_idx_to_mon; + + /* + * Array of assigned MBWU monitors, indexed by idx argument. + * When ABMC is not in use, this array can be NULL. Otherwise + * it maps idx to the allocated monitor. + */ + int *assigned_counters; }; static inline int mpam_alloc_csu_mon(struct mpam_class *class) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 82801feb5211b..e8eefa69658ad 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -75,6 +75,12 @@ static DECLARE_WAIT_QUEUE_HEAD(wait_cacheinfo_ready); */ static bool resctrl_enabled; +/* + * L3 local/total may come from different classes - what is the number of MBWU + * 'on L3'? + */ +static unsigned int l3_num_allocated_mbwu = ~0; + /* Whether this num_mbw_mon could result in a free_running system */ static int __mpam_monitors_free_running(u16 num_mbwu_mon) { @@ -83,6 +89,15 @@ static int __mpam_monitors_free_running(u16 num_mbwu_mon) return 0; } +/* + * If l3_num_allocated_mbwu is forced below PARTID * PMG, then the counters + * are not free running, and ABMC's user-interface must be used to assign them. + */ +static bool mpam_resctrl_abmc_enabled(void) +{ + return l3_num_allocated_mbwu < resctrl_arch_system_num_rmid_idx(); +} + bool resctrl_arch_alloc_capable(void) { struct mpam_resctrl_res *res; @@ -146,16 +161,6 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d, return -EOPNOTSUPP; } -bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) -{ - return false; -} - -int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) -{ - return -EINVAL; -} - int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable) { return -EOPNOTSUPP; @@ -193,6 +198,21 @@ static void resctrl_reset_task_closids(void) read_unlock(&tasklist_lock); } +static void mpam_resctrl_monitor_sync_abmc_vals(struct rdt_resource *l3) +{ + l3->mon.num_mbm_cntrs = l3_num_allocated_mbwu; + if (cdp_enabled) + l3->mon.num_mbm_cntrs /= 2; + + if (l3->mon.num_mbm_cntrs) { + l3->mon.mbm_cntr_assignable = mpam_resctrl_abmc_enabled(); + l3->mon.mbm_assign_on_mkdir = mpam_resctrl_abmc_enabled(); + } else { + l3->mon.mbm_cntr_assignable = false; + l3->mon.mbm_assign_on_mkdir = false; + } +} + int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) { u32 partid_i = RESCTRL_RESERVED_CLOSID, partid_d = RESCTRL_RESERVED_CLOSID; @@ -252,6 +272,7 @@ int resctrl_arch_set_cdp_enabled(enum resctrl_res_level rid, bool enable) WRITE_ONCE(arm64_mpam_global_default, mpam_get_regval(current)); resctrl_reset_task_closids(); + mpam_resctrl_monitor_sync_abmc_vals(l3); for_each_possible_cpu(cpu) mpam_set_cpu_defaults(cpu, partid_d, partid_i, 0, 0); @@ -631,6 +652,11 @@ static bool class_has_usable_mbwu(struct mpam_class *class) return true; } + if (cprops->num_mbwu_mon) { + pr_debug("monitors usable via ABMC assignment\n"); + return true; + } + return false; } @@ -978,6 +1004,8 @@ static int __alloc_mbwu_mon(struct mpam_class *class, int *array, array[i] = mbwu_mon; } + l3_num_allocated_mbwu = min(l3_num_allocated_mbwu, num_mbwu_mon); + return 0; } @@ -1125,6 +1153,23 @@ static void mpam_resctrl_pick_counters(void) mpam_resctrl_counters[QOS_L3_MBM_TOTAL_EVENT_ID].class); } +bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) +{ + if (r != &mpam_resctrl_controls[RDT_RESOURCE_L3].resctrl_res) + return false; + + return mpam_resctrl_abmc_enabled(); +} + +int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) +{ + lockdep_assert_cpus_held(); + + WARN_ON_ONCE(1); + + return 0; +} + static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) { struct mpam_class *class = res->class; @@ -1202,6 +1247,41 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) return comp->comp_id; } +/* + * This must run after all event counters have been picked so that any free + * running counters have already been allocated. + */ +static int mpam_resctrl_monitor_init_abmc(struct mpam_resctrl_mon *mon) +{ + struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + size_t array_size = resctrl_arch_system_num_rmid_idx() * sizeof(int); + int *rmid_array __free(kfree) = kmalloc(array_size, GFP_KERNEL); + struct rdt_resource *l3 = &res->resctrl_res; + struct mpam_class *class = mon->class; + u16 num_mbwu_mon; + + if (mon->mbwu_idx_to_mon) { + pr_debug("monitors free running\n"); + return 0; + } + + if (!rmid_array) { + pr_debug("Failed to allocate RMID array\n"); + return -ENOMEM; + } + memset(rmid_array, -1, array_size); + + num_mbwu_mon = class->props.num_mbwu_mon; + mon->assigned_counters = __alloc_mbwu_array(mon->class, num_mbwu_mon); + if (IS_ERR(mon->assigned_counters)) + return PTR_ERR(mon->assigned_counters); + mon->mbwu_idx_to_mon = no_free_ptr(rmid_array); + + mpam_resctrl_monitor_sync_abmc_vals(l3); + + return 0; +} + static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, enum resctrl_event_id type) { @@ -1249,6 +1329,14 @@ static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, if (resctrl_enable_mon_event(type, false, 0, NULL)) l3->mon_capable = true; + switch (type) { + case QOS_L3_MBM_LOCAL_EVENT_ID: + case QOS_L3_MBM_TOTAL_EVENT_ID: + return mpam_resctrl_monitor_init_abmc(mon); + default: + return 0; + } + return 0; } From a199cc64eb183d62eed7064987752a8c1a5b165a Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 16 Oct 2025 14:31:11 +0100 Subject: [PATCH 327/464] NVIDIA: VR: SAUCE: arm_mpam: resctrl: Add kunit test for ABMC/CDP interactions BugLink: https://bugs.launchpad.net/bugs/2154527 ABMC exposes a fun corner case where a platform with one monitor can use ABMC for assignable counters - but not when CDP is enabled. Add some tests. Signed-off-by: James Morse (cherry picked from commit a861a0f40d75549387301244a228b519c86c063b https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/test_mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/test_mpam_resctrl.c | 62 +++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/drivers/resctrl/test_mpam_resctrl.c b/drivers/resctrl/test_mpam_resctrl.c index b93d6ad87e43f..4145f057bd31a 100644 --- a/drivers/resctrl/test_mpam_resctrl.c +++ b/drivers/resctrl/test_mpam_resctrl.c @@ -296,6 +296,67 @@ static void test_percent_to_max_rounding(struct kunit *test) KUNIT_EXPECT_LE(test, 4 * num_rounded_up, 3 * total); } +static void test_num_assignable_counters(struct kunit *test) +{ + unsigned int orig_l3_num_allocated_mbwu = l3_num_allocated_mbwu; + u32 orig_mpam_partid_max = mpam_partid_max; + u32 orig_mpam_pmg_max = mpam_pmg_max; + bool orig_cdp_enabled = cdp_enabled; + struct rdt_resource fake_l3; + + /* Force there to be some PARTID/PMG */ + mpam_partid_max = 3; + mpam_pmg_max = 1; + + cdp_enabled = false; + + /* ABMC off, CDP off */ + l3_num_allocated_mbwu = resctrl_arch_system_num_rmid_idx(); + mpam_resctrl_monitor_sync_abmc_vals(&fake_l3); + KUNIT_EXPECT_EQ(test, fake_l3.mon.num_mbm_cntrs, resctrl_arch_system_num_rmid_idx()); + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_cntr_assignable); + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_assign_on_mkdir); + + /* ABMC on, CDP off */ + l3_num_allocated_mbwu = 4; + mpam_resctrl_monitor_sync_abmc_vals(&fake_l3); + KUNIT_EXPECT_EQ(test, fake_l3.mon.num_mbm_cntrs, 4); + KUNIT_EXPECT_TRUE(test, fake_l3.mon.mbm_cntr_assignable); + KUNIT_EXPECT_TRUE(test, fake_l3.mon.mbm_assign_on_mkdir); + + cdp_enabled = true; + + /* ABMC off, CDP on */ + l3_num_allocated_mbwu = resctrl_arch_system_num_rmid_idx(); + mpam_resctrl_monitor_sync_abmc_vals(&fake_l3); + + /* (value not consumed by resctrl) */ + KUNIT_EXPECT_EQ(test, fake_l3.mon.num_mbm_cntrs, resctrl_arch_system_num_rmid_idx() / 2); + + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_cntr_assignable); + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_assign_on_mkdir); + + /* ABMC on, CDP on */ + l3_num_allocated_mbwu = 4; + mpam_resctrl_monitor_sync_abmc_vals(&fake_l3); + KUNIT_EXPECT_EQ(test, fake_l3.mon.num_mbm_cntrs, 2); + KUNIT_EXPECT_TRUE(test, fake_l3.mon.mbm_cntr_assignable); + KUNIT_EXPECT_TRUE(test, fake_l3.mon.mbm_assign_on_mkdir); + + /* ABMC 'on', CDP on - but not enough counters */ + l3_num_allocated_mbwu = 1; + mpam_resctrl_monitor_sync_abmc_vals(&fake_l3); + KUNIT_EXPECT_EQ(test, fake_l3.mon.num_mbm_cntrs, 0); + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_cntr_assignable); + KUNIT_EXPECT_FALSE(test, fake_l3.mon.mbm_assign_on_mkdir); + + /* Restore global variables that were messed with */ + l3_num_allocated_mbwu = orig_l3_num_allocated_mbwu; + mpam_partid_max = orig_mpam_partid_max; + mpam_pmg_max = orig_mpam_pmg_max; + cdp_enabled = orig_cdp_enabled; +} + static struct kunit_case mpam_resctrl_test_cases[] = { KUNIT_CASE(test_get_mba_granularity), KUNIT_CASE_PARAM(test_mbw_max_to_percent, test_percent_value_gen_params), @@ -304,6 +365,7 @@ static struct kunit_case mpam_resctrl_test_cases[] = { KUNIT_CASE(test_percent_to_max_rounding), KUNIT_CASE_PARAM(test_percent_max_roundtrip_stability, test_all_bwa_wd_gen_params), + KUNIT_CASE(test_num_assignable_counters), {} }; From 77c4e9f9d6292960578be6a40c21fee2a2da2400 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 15 Oct 2025 14:33:11 +0100 Subject: [PATCH 328/464] NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_config_cntr() for ABMC use BugLink: https://bugs.launchpad.net/bugs/2154527 ABMC has a helper resctrl_arch_config_cntr() for changing the mapping between 'cntr_id' and a CLOSID/RMID pair. Add the helper. For MPAM this is done by updating the mon->mbwu_idx_to_mon[] array, and as usual CDP means it needs doing in three different ways. Signed-off-by: James Morse (cherry picked from commit ce6ad9dcc0fd43bb2a7558fdae6c11e96cf2f066 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Remove empty definition of resctrl_arch_config_cntr() - Resolve struct rdt_l3_mon_domain parameter in resctrl_arch_config_cntr() ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 43 +++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index e8eefa69658ad..753e11bd9d0b2 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -148,12 +148,6 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d { } -void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, - enum resctrl_event_id evtid, u32 rmid, u32 closid, - u32 cntr_id, bool assign) -{ -} - int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d, u32 unused, u32 rmid, int cntr_id, enum resctrl_event_id eventid, u64 *val) @@ -1153,6 +1147,43 @@ static void mpam_resctrl_pick_counters(void) mpam_resctrl_counters[QOS_L3_MBM_TOTAL_EVENT_ID].class); } +static void __config_cntr(struct mpam_resctrl_mon *mon, u32 cntr_id, + enum resctrl_conf_type cdp_type, u32 closid, u32 rmid, + bool assign) +{ + u32 mbwu_idx, mon_idx = resctrl_get_config_index(cntr_id, cdp_type); + + closid = resctrl_get_config_index(closid, cdp_type); + mbwu_idx = resctrl_arch_rmid_idx_encode(closid, rmid); + WARN_ON_ONCE(mon_idx > l3_num_allocated_mbwu); + + if (assign) + mon->mbwu_idx_to_mon[mbwu_idx] = mon->assigned_counters[mon_idx]; + else + mon->mbwu_idx_to_mon[mbwu_idx] = -1; +} + +void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + enum resctrl_event_id evtid, u32 rmid, u32 closid, + u32 cntr_id, bool assign) +{ + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evtid]; + + if (!mon->mbwu_idx_to_mon || !mon->assigned_counters) { + pr_debug("monitor arrays not allocated\n"); + return; + } + + if (cdp_enabled) { + __config_cntr(mon, cntr_id, CDP_CODE, closid, rmid, assign); + __config_cntr(mon, cntr_id, CDP_DATA, closid, rmid, assign); + } else { + __config_cntr(mon, cntr_id, CDP_NONE, closid, rmid, assign); + } + + resctrl_arch_reset_rmid(r, d, closid, rmid, evtid); +} + bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) { if (r != &mpam_resctrl_controls[RDT_RESOURCE_L3].resctrl_res) From bc141c8193bac3b5a805c5bfe2f14e89fffa88f9 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 25 Jun 2021 16:36:58 +0100 Subject: [PATCH 329/464] NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_rmid_read() and resctrl_arch_reset_rmid() BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl uses resctrl_arch_rmid_read() to read counters. CDP emulation means the counter may need reading in three different ways. The same goes for reset. The helpers behind the resctrl_arch_ functions will be re-used for the ABMC equivalent functions. Add the rounding helper for checking monitor values while we're here. Signed-off-by: James Morse (cherry picked from commit d45ffcb70f8a2c055b1b449b0a0780773cc5ca55 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - resctrl_arch_round_mon_val() has been defined in upstream. No need to re-define it here; - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 182 ++++++++++++++++++++++----------- 1 file changed, 123 insertions(+), 59 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 753e11bd9d0b2..82c0b58bf4f27 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -137,11 +137,6 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domai { } -void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d, - u32 closid, u32 rmid, enum resctrl_event_id eventid) -{ -} - void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, u32 closid, u32 rmid, int cntr_id, enum resctrl_event_id eventid) @@ -464,12 +459,49 @@ void resctrl_arch_mon_ctx_free(struct rdt_resource *r, resctrl_arch_mon_ctx_free_no_wait(evtid, mon_idx); } -static int __read_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, - enum mpam_device_features mon_type, - int mon_idx, - enum resctrl_conf_type cdp_type, u32 closid, u32 rmid, u64 *val) +/* + * The rmid realloc threshold should be for the smallest cache exposed to + * resctrl. + */ +static int update_rmid_limits(struct mpam_class *class) +{ + u32 num_unique_pmg = resctrl_arch_system_num_rmid_idx(); + struct mpam_props *cprops = &class->props; + struct cacheinfo *ci; + + lockdep_assert_cpus_held(); + + if (!mpam_has_feature(mpam_feat_msmon_csu, cprops)) + return 0; + + /* + * Assume cache levels are the same size for all CPUs... + * The check just requires any online CPU and it can't go offline as we + * hold the cpu lock. + */ + ci = get_cpu_cacheinfo_level(raw_smp_processor_id(), class->level); + if (!ci || ci->size == 0) { + pr_debug("Could not read cache size for class %u\n", + class->level); + return -EINVAL; + } + + if (!resctrl_rmid_realloc_limit || + ci->size < resctrl_rmid_realloc_limit) { + resctrl_rmid_realloc_limit = ci->size; + resctrl_rmid_realloc_threshold = ci->size / num_unique_pmg; + } + + return 0; +} + +static int +__read_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, + enum mpam_device_features mon_type, + int mon_idx, + enum resctrl_conf_type cdp_type, u32 closid, u32 rmid, u64 *val) { - struct mon_cfg cfg; + struct mon_cfg cfg = { }; if (!mpam_is_enabled()) return -EINVAL; @@ -477,18 +509,29 @@ static int __read_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_c /* Shift closid to account for CDP */ closid = resctrl_get_config_index(closid, cdp_type); + if (mon_idx == USE_PRE_ALLOCATED) { + int mbwu_idx = resctrl_arch_rmid_idx_encode(closid, rmid); + mon_idx = mon->mbwu_idx_to_mon[mbwu_idx]; + if (mon_idx == -1) { + if (mpam_resctrl_abmc_enabled()) { + /* Report Unassigned */ + return -ENOENT; + } + /* Report Unavailable */ + return -EINVAL; + } + } + + cfg.mon = mon_idx; + cfg.match_pmg = true; + cfg.partid = closid; + cfg.pmg = rmid; + if (irqs_disabled()) { /* Check if we can access this domain without an IPI */ return -EIO; } - cfg = (struct mon_cfg) { - .mon = mon_idx, - .match_pmg = true, - .partid = closid, - .pmg = rmid, - }; - return mpam_msmon_read(mon_comp, &cfg, mon_type, val); } @@ -497,29 +540,27 @@ static int read_mon_cdp_safe(struct mpam_resctrl_mon *mon, struct mpam_component int mon_idx, u32 closid, u32 rmid, u64 *val) { if (cdp_enabled) { - u64 code_val = 0, data_val = 0; + u64 cdp_val = 0; int err; err = __read_mon(mon, mon_comp, mon_type, mon_idx, - CDP_CODE, closid, rmid, &code_val); + CDP_CODE, closid, rmid, &cdp_val); if (err) return err; err = __read_mon(mon, mon_comp, mon_type, mon_idx, - CDP_DATA, closid, rmid, &data_val); - if (err) - return err; - - *val += code_val + data_val; - return 0; + CDP_DATA, closid, rmid, &cdp_val); + if (!err) + *val += cdp_val; + return err; } return __read_mon(mon, mon_comp, mon_type, mon_idx, CDP_NONE, closid, rmid, val); } -/* MBWU when not in ABMC mode (not supported), and CSU counters. */ -int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, +/* MBWU when not in ABMC mode, and CSU counters. */ +int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, u32 closid, u32 rmid, enum resctrl_event_id eventid, void *arch_priv, u64 *val, void *arch_mon_ctx) { @@ -531,58 +572,81 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, resctrl_arch_rmid_read_context_check(); - if (!mpam_is_enabled()) - return -EINVAL; - if (eventid >= QOS_NUM_EVENTS || !mon->class) return -EINVAL; l3_dom = container_of(hdr, struct mpam_resctrl_dom, resctrl_mon_dom.hdr); mon_comp = l3_dom->mon_comp[eventid]; - if (eventid != QOS_L3_OCCUP_EVENT_ID) + switch (eventid) { + case QOS_L3_OCCUP_EVENT_ID: + mon_type = mpam_feat_msmon_csu; + break; + case QOS_L3_MBM_LOCAL_EVENT_ID: + case QOS_L3_MBM_TOTAL_EVENT_ID: + mon_type = mpam_feat_msmon_mbwu; + break; + default: return -EINVAL; - - mon_type = mpam_feat_msmon_csu; + } return read_mon_cdp_safe(mon, mon_comp, mon_type, mon_idx, closid, rmid, val); } -/* - * The rmid realloc threshold should be for the smallest cache exposed to - * resctrl. - */ -static int update_rmid_limits(struct mpam_class *class) +static void __reset_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, + int mon_idx, + enum resctrl_conf_type cdp_type, u32 closid, u32 rmid) { - u32 num_unique_pmg = resctrl_arch_system_num_rmid_idx(); - struct mpam_props *cprops = &class->props; - struct cacheinfo *ci; + struct mon_cfg cfg = { }; - lockdep_assert_cpus_held(); + if (!mpam_is_enabled()) + return; - if (!mpam_has_feature(mpam_feat_msmon_csu, cprops)) - return 0; + /* Shift closid to account for CDP */ + closid = resctrl_get_config_index(closid, cdp_type); - /* - * Assume cache levels are the same size for all CPUs... - * The check just requires any online CPU and it can't go offline as we - * hold the cpu lock. - */ - ci = get_cpu_cacheinfo_level(raw_smp_processor_id(), class->level); - if (!ci || ci->size == 0) { - pr_debug("Could not read cache size for class %u\n", - class->level); - return -EINVAL; + if (mon_idx == USE_PRE_ALLOCATED) { + int mbwu_idx = resctrl_arch_rmid_idx_encode(closid, rmid); + mon_idx = mon->mbwu_idx_to_mon[mbwu_idx]; } - if (!resctrl_rmid_realloc_limit || - ci->size < resctrl_rmid_realloc_limit) { - resctrl_rmid_realloc_limit = ci->size; - resctrl_rmid_realloc_threshold = ci->size / num_unique_pmg; + if (mon_idx == -1) + return; + cfg.mon = mon_idx; + mpam_msmon_reset_mbwu(mon_comp, &cfg); +} + +static void reset_mon_cdp_safe(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, + int mon_idx, u32 closid, u32 rmid) +{ + if (cdp_enabled) { + __reset_mon(mon, mon_comp, mon_idx, CDP_CODE, closid, rmid); + __reset_mon(mon, mon_comp, mon_idx, CDP_DATA, closid, rmid); + } else { + __reset_mon(mon, mon_comp, mon_idx, CDP_NONE, closid, rmid); } +} - return 0; +/* Called via IPI. Call with read_cpus_lock() held. */ +void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 closid, u32 rmid, enum resctrl_event_id eventid) +{ + struct mpam_resctrl_dom *l3_dom; + struct mpam_component *mon_comp; + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[eventid]; + + if (!mpam_is_enabled()) + return; + + /* Only MBWU counters are relevant, and for supported event types. */ + if (eventid == QOS_L3_OCCUP_EVENT_ID || !mon->class) + return; + + l3_dom = container_of(d, struct mpam_resctrl_dom, resctrl_mon_dom); + mon_comp = l3_dom->mon_comp[eventid]; + + reset_mon_cdp_safe(mon, mon_comp, USE_PRE_ALLOCATED, closid, rmid); } static bool cache_has_usable_cpor(struct mpam_class *class) From a639261dd3089a14dcb813f693004ae574ac0736 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 26 Aug 2025 16:05:07 +0100 Subject: [PATCH 330/464] NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_cntr_read() & resctrl_arch_reset_cntr() BugLink: https://bugs.launchpad.net/bugs/2154527 When used in ABMC mode, resctrl uses a different set of helpers to read and reset the counters. Add these. Signed-off-by: James Morse (cherry picked from commit 81af700d29ca8d39ed835ad1cee1ab8095517a9d https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Remove empty definitions of resctrl_arch_cntr_read() and resctrl_arch_reset_cntr() - Resolve struct rdt_l3_mon_domain parameter in resctrl_arch_cntr_read() and resctrl_arch_reset_cntr() ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 56 ++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 82c0b58bf4f27..aa260a1e11869 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -137,19 +137,6 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domai { } -void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, - u32 closid, u32 rmid, int cntr_id, - enum resctrl_event_id eventid) -{ -} - -int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d, - u32 unused, u32 rmid, int cntr_id, - enum resctrl_event_id eventid, u64 *val) -{ - return -EOPNOTSUPP; -} - int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable) { return -EOPNOTSUPP; @@ -594,6 +581,28 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, closid, rmid, val); } +/* MBWU counters when in ABMC mode */ +int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 closid, u32 rmid, int mon_idx, + enum resctrl_event_id eventid, u64 *val) +{ + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[eventid]; + struct mpam_resctrl_dom *l3_dom; + struct mpam_component *mon_comp; + + if (!mpam_is_enabled()) + return -EINVAL; + + if (eventid == QOS_L3_OCCUP_EVENT_ID || !mon->class) + return -EINVAL; + + l3_dom = container_of(d, struct mpam_resctrl_dom, resctrl_mon_dom); + mon_comp = l3_dom->mon_comp[eventid]; + + return read_mon_cdp_safe(mon, mon_comp, mpam_feat_msmon_mbwu, mon_idx, + closid, rmid, val); +} + static void __reset_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, int mon_idx, enum resctrl_conf_type cdp_type, u32 closid, u32 rmid) @@ -649,6 +658,27 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d reset_mon_cdp_safe(mon, mon_comp, USE_PRE_ALLOCATED, closid, rmid); } +/* Reset an assigned counter */ +void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d, + u32 closid, u32 rmid, int cntr_id, + enum resctrl_event_id eventid) +{ + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[eventid]; + struct mpam_resctrl_dom *l3_dom; + struct mpam_component *mon_comp; + + if (!mpam_is_enabled()) + return; + + if (eventid == QOS_L3_OCCUP_EVENT_ID || !mon->class) + return; + + l3_dom = container_of(d, struct mpam_resctrl_dom, resctrl_mon_dom); + mon_comp = l3_dom->mon_comp[eventid]; + + reset_mon_cdp_safe(mon, mon_comp, USE_PRE_ALLOCATED, closid, rmid); +} + static bool cache_has_usable_cpor(struct mpam_class *class) { struct mpam_props *cprops = &class->props; From 6be0053c3bf7a9ce82a3f8a8f52181ad881aa3f5 Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 10 Jun 2024 17:20:48 +0100 Subject: [PATCH 331/464] NVIDIA: VR: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and closid_num_dirty_rmid[] BugLink: https://bugs.launchpad.net/bugs/2154527 On MPAM systems if an error occurs the arhictecture code will call resctrl_exit(). This calls dom_data_exit() which takes the rdrgroup_mutex and kfree()s closid_num_dirty_rmid[]. It is possible that another syscall tries to access that same array in the meantime, but is blocked on the mutex. Once dom_data_exit() completes, that syscall will see a NULL pointer. Pull the IS_ENABLED() Kconfig checks into a helper and additionally check that the array has been allocated. This will cause callers to fallback to the regular CLOSID allocation strategy. Signed-off-by: James Morse (cherry picked from commit b9be9ec43910a549fb4f5eaced3bffcebc6a180e https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/monitor.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c index 9fd901c78dc66..65a761fb4c1ec 100644 --- a/fs/resctrl/monitor.c +++ b/fs/resctrl/monitor.c @@ -113,6 +113,20 @@ static inline struct rmid_entry *__rmid_entry(u32 idx) return entry; } +static bool __has_closid_num_dirty_rmid_array(void) +{ + lockdep_assert_held(&rdtgroup_mutex); + + if (!IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) + return false; + + /* + * Avoid a race with dom_data_exit() freeing the array under + * rdtgroup_mutex. + */ + return closid_num_dirty_rmid; +} + static void limbo_release_entry(struct rmid_entry *entry) { lockdep_assert_held(&rdtgroup_mutex); @@ -120,7 +134,7 @@ static void limbo_release_entry(struct rmid_entry *entry) rmid_limbo_count--; list_add_tail(&entry->list, &rmid_free_lru); - if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) + if (__has_closid_num_dirty_rmid_array()) closid_num_dirty_rmid[entry->closid]--; } @@ -244,7 +258,7 @@ int resctrl_find_cleanest_closid(void) lockdep_assert_held(&rdtgroup_mutex); - if (!IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) + if (!__has_closid_num_dirty_rmid_array()) return -EIO; for (i = 0; i < closids_supported(); i++) { @@ -317,7 +331,7 @@ static void add_rmid_to_limbo(struct rmid_entry *entry) } rmid_limbo_count++; - if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) + if (__has_closid_num_dirty_rmid_array()) closid_num_dirty_rmid[entry->closid]++; } From 4dee6626ad779c75c6597a49a7d898404e2a8cda Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 10 Jun 2024 17:41:58 +0100 Subject: [PATCH 332/464] NVIDIA: VR: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and rmid_ptrs[] BugLink: https://bugs.launchpad.net/bugs/2154527 On MPAM systems if an error occurs the arhictecture code will call resctrl_exit(). This calls dom_data_exit() which takes the rdrgroup_mutex and kfree()s rmid_ptrs[]. It is possible that another syscall tries to access that same array in the meantime, but is blocked on the mutex. Once dom_data_exit() completes, that syscall will see a NULL pointer. Make __rmid_entry() return NULL in this case. Neither __check_limbo() nor free_rmid() return an error, and can silently stop their work if this occurs. dom_data_init() has only just allocated the array and still holds the lock, so __rmid_entry() should never return NULL here. Signed-off-by: James Morse (cherry picked from commit c1ac3a4e7a0d09175fb84eb7be2b7b23e8c09f09 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/monitor.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/monitor.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c index 65a761fb4c1ec..e338b8d484054 100644 --- a/fs/resctrl/monitor.c +++ b/fs/resctrl/monitor.c @@ -98,12 +98,17 @@ unsigned int resctrl_rmid_realloc_limit; * * The domain's rmid_busy_llc and rmid_ptrs[] are sized by index. The arch code * must accept an attempt to read every index. + * + * Returns NULL if the rmid_ptrs[] array is not allocated. */ static inline struct rmid_entry *__rmid_entry(u32 idx) { struct rmid_entry *entry; u32 closid, rmid; + if (!rmid_ptrs) + return NULL; + entry = &rmid_ptrs[idx]; resctrl_arch_rmid_idx_decode(idx, &closid, &rmid); @@ -175,6 +180,8 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free) break; entry = __rmid_entry(idx); + if (!entry) + break; if (resctrl_arch_rmid_read(r, &d->hdr, entry->closid, entry->rmid, QOS_L3_OCCUP_EVENT_ID, arch_priv, &val, arch_mon_ctx)) { @@ -353,6 +360,8 @@ void free_rmid(u32 closid, u32 rmid) return; entry = __rmid_entry(idx); + if (!entry) + return; if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID)) add_rmid_to_limbo(entry); @@ -959,6 +968,7 @@ int setup_rmid_lru_list(void) idx = resctrl_arch_rmid_idx_encode(RESCTRL_RESERVED_CLOSID, RESCTRL_RESERVED_RMID); entry = __rmid_entry(idx); + WARN_ON_ONCE(!entry); list_del(&entry->list); return 0; From c033ebcbffc20ed1e175a73f11cea6b543d4226c Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 22 Dec 2022 17:01:52 +0000 Subject: [PATCH 333/464] NVIDIA: VR: SAUCE: debugfs: Add helpers for creating cpumask entries in debugfs BugLink: https://bugs.launchpad.net/bugs/2154527 debugfs has handy helpers to make a bool, integer or string available through debugfs. Add helpers to do the same for cpumasks. These are read only. CC: Ben Horgan Signed-off-by: James Morse (cherry picked from commit 25c2e6fafcfd8044ea148672d3e6b4b29be0d756 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/debugfs/file.c | 64 +++++++++++++++++++++++++++++++++++++++++ include/linux/debugfs.h | 6 ++++ 2 files changed, 70 insertions(+) diff --git a/fs/debugfs/file.c b/fs/debugfs/file.c index edd6aafbfbaaf..6e8c15016ece2 100644 --- a/fs/debugfs/file.c +++ b/fs/debugfs/file.c @@ -1143,6 +1143,70 @@ void debugfs_create_str(const char *name, umode_t mode, } EXPORT_SYMBOL_GPL(debugfs_create_str); +static ssize_t debugfs_read_file_cpumask(struct file *file, + char __user *user_buf, + size_t count, loff_t *ppos) +{ + struct dentry *dentry = F_DENTRY(file); + struct cpumask *cpumask; + char *kernel_buf; + ssize_t ret; + int len; + + ret = debugfs_file_get(dentry); + if (unlikely(ret)) + return ret; + + /* How long is a piece of string? */ + kernel_buf = kmalloc(PAGE_SIZE, GFP_KERNEL); + if (!kernel_buf) { + debugfs_file_put(dentry); + return -ENOMEM; + } + + cpumask = (struct cpumask *)file->private_data; + len = scnprintf(kernel_buf, PAGE_SIZE, + "%*pb\n", cpumask_pr_args(cpumask)); + debugfs_file_put(dentry); + if (len + 1 >= PAGE_SIZE) { + kfree(kernel_buf); + return -EIO; + } + + ret = simple_read_from_buffer(user_buf, count, ppos, kernel_buf, len); + kfree(kernel_buf); + + return ret; +} + +static const struct file_operations fops_cpumask_ro = { + .read = debugfs_read_file_cpumask, + .open = simple_open, + .llseek = default_llseek, +}; + +/** + * debugfs_create_cpumask - create a read-only debugfs file that is used to read a cpumask + * @name: a pointer to a string containing the name of the file to create. + * @mode: the permission that the file should have + * @parent: a pointer to the parent dentry for this file. This should be a + * directory dentry if set. If this parameter is %NULL, then the + * file will be created in the root of the debugfs filesystem. + * @value: a pointer to the variable that the file should read from. + * + * This function creates a file in debugfs with the given name that + * contains the value of the variable @value. + */ +void debugfs_create_cpumask(const char *name, umode_t mode, + struct dentry *parent, struct cpumask *value) +{ + /* Only read-only is supported */ + WARN_ON_ONCE(mode & S_IWUGO); + + debugfs_create_mode_unsafe(name, mode, parent, value, &fops_cpumask_ro, + &fops_cpumask_ro, &fops_cpumask_ro); +} + static ssize_t read_file_blob(struct file *file, char __user *user_buf, size_t count, loff_t *ppos) { diff --git a/include/linux/debugfs.h b/include/linux/debugfs.h index 4177c47382826..591d4b7267d8b 100644 --- a/include/linux/debugfs.h +++ b/include/linux/debugfs.h @@ -202,6 +202,8 @@ void debugfs_create_bool(const char *name, umode_t mode, struct dentry *parent, bool *value); void debugfs_create_str(const char *name, umode_t mode, struct dentry *parent, char **value); +void debugfs_create_cpumask(const char *name, umode_t mode, + struct dentry *parent, struct cpumask *value); struct dentry *debugfs_create_blob(const char *name, umode_t mode, struct dentry *parent, @@ -407,6 +409,10 @@ static inline void debugfs_create_str(const char *name, umode_t mode, char **value) { } +static inline void debugfs_create_cpumask(const char *name, umode_t mode, + struct dentry *parent, struct cpumask *value) +{ } + static inline struct dentry *debugfs_create_blob(const char *name, umode_t mode, struct dentry *parent, struct debugfs_blob_wrapper *blob) From 56836799ed9af1b43e1211d861d8d380a82d8a44 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 1 Sep 2021 15:13:12 +0100 Subject: [PATCH 334/464] NVIDIA: VR: SAUCE: arm_mpam: Add debugfs entries to show the MSC/RIS the driver discovered BugLink: https://bugs.launchpad.net/bugs/2154527 Not all of MPAM is visible through the resctrl user-space interface. To make it easy to debug why certain devices were not exposed through resctrl, allow the properties of the devices to be read through debugfs. This adds an mpam directory to debugfs, and exposes the devices as well as the hierarchy that was built. Signed-off-by: James Morse (cherry picked from commit e8f0f2147103bec25b367a273abfb7b6805df914 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 136 +++++++++++++++++++++++++++++--- drivers/resctrl/mpam_internal.h | 9 +++ 2 files changed, 136 insertions(+), 9 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 988fc291241d2..8884ff775dd52 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -83,6 +83,8 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable); /* When mpam is disabled, the printed reason to aid debugging */ static char *mpam_disable_reason; +static struct dentry *mpam_debugfs; + /* * Whether resctrl has been setup. Used by cpuhp in preference to * mpam_is_enabled(). The disable call after an error interrupt makes @@ -339,6 +341,8 @@ static void mpam_class_destroy(struct mpam_class *class) { lockdep_assert_held(&mpam_list_lock); + debugfs_remove_recursive(class->debugfs); + class->debugfs = NULL; list_del_rcu(&class->classes_list); add_to_garbage(class); } @@ -391,6 +395,8 @@ static void mpam_component_destroy(struct mpam_component *comp) __destroy_component_cfg(comp); + debugfs_remove_recursive(comp->debugfs); + comp->debugfs = NULL; list_del_rcu(&comp->class_list); add_to_garbage(comp); @@ -441,6 +447,8 @@ static void mpam_vmsc_destroy(struct mpam_vmsc *vmsc) lockdep_assert_held(&mpam_list_lock); + debugfs_remove_recursive(vmsc->debugfs); + vmsc->debugfs = NULL; list_del_rcu(&vmsc->comp_list); add_to_garbage(vmsc); @@ -600,6 +608,8 @@ static void mpam_ris_destroy(struct mpam_msc_ris *ris) cpumask_andnot(&class->affinity, &class->affinity, &ris->affinity); cpumask_andnot(&comp->affinity, &comp->affinity, &ris->affinity); clear_bit(ris->ris_idx, &msc->ris_idxs); + debugfs_remove_recursive(ris->debugfs); + ris->debugfs = NULL; list_del_rcu(&ris->msc_list); list_del_rcu(&ris->vmsc_list); add_to_garbage(ris); @@ -782,32 +792,32 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) /* Cache Capacity Partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) { - u32 ccap_features = mpam_read_partsel_reg(msc, CCAP_IDR); + ris->ccap_idr = mpam_read_partsel_reg(msc, CCAP_IDR); - props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features); + props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ris->ccap_idr); if (props->cmax_wd && - FIELD_GET(MPAMF_CCAP_IDR_HAS_CMAX_SOFTLIM, ccap_features)) + FIELD_GET(MPAMF_CCAP_IDR_HAS_CMAX_SOFTLIM, ris->ccap_idr)) mpam_set_feature(mpam_feat_cmax_softlim, props); if (props->cmax_wd && - !FIELD_GET(MPAMF_CCAP_IDR_NO_CMAX, ccap_features)) + !FIELD_GET(MPAMF_CCAP_IDR_NO_CMAX, ris->ccap_idr)) mpam_set_feature(mpam_feat_cmax_cmax, props); if (props->cmax_wd && - FIELD_GET(MPAMF_CCAP_IDR_HAS_CMIN, ccap_features)) + FIELD_GET(MPAMF_CCAP_IDR_HAS_CMIN, ris->ccap_idr)) mpam_set_feature(mpam_feat_cmax_cmin, props); - props->cassoc_wd = FIELD_GET(MPAMF_CCAP_IDR_CASSOC_WD, ccap_features); + props->cassoc_wd = FIELD_GET(MPAMF_CCAP_IDR_CASSOC_WD, ris->ccap_idr); if (props->cassoc_wd && - FIELD_GET(MPAMF_CCAP_IDR_HAS_CASSOC, ccap_features)) + FIELD_GET(MPAMF_CCAP_IDR_HAS_CASSOC, ris->ccap_idr)) mpam_set_feature(mpam_feat_cmax_cassoc, props); } /* Cache Portion partitioning */ if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) { - u32 cpor_features = mpam_read_partsel_reg(msc, CPOR_IDR); + ris->cpor_idr = mpam_read_partsel_reg(msc, CPOR_IDR); - props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features); + props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, ris->cpor_idr); if (props->cpbm_wd) mpam_set_feature(mpam_feat_cpor_part, props); } @@ -1995,6 +2005,9 @@ static void mpam_msc_destroy(struct mpam_msc *msc) list_del_rcu(&msc->all_msc_list); platform_set_drvdata(pdev, NULL); + debugfs_remove_recursive(msc->debugfs); + msc->debugfs = NULL; + add_to_garbage(msc); } @@ -2013,6 +2026,7 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) { int err; u32 tmp; + char name[20]; struct mpam_msc *msc; struct resource *msc_res; struct device *dev = &pdev->dev; @@ -2080,6 +2094,10 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) list_add_rcu(&msc->all_msc_list, &mpam_all_msc); platform_set_drvdata(pdev, msc); + snprintf(name, sizeof(name), "msc.%u", msc->id); + msc->debugfs = debugfs_create_dir(name, mpam_debugfs); + debugfs_create_x32("max_nrdy_usec", 0400, msc->debugfs, &msc->nrdy_usec); + return msc; } @@ -2688,6 +2706,102 @@ static int mpam_allocate_config(void) return 0; } +static void mpam_debugfs_setup_ris(struct mpam_msc_ris *ris) +{ + char name[40]; + struct dentry *d; + struct mpam_props *rprops = &ris->props; + + snprintf(name, sizeof(name), "ris.%u", ris->ris_idx); + d = debugfs_create_dir(name, ris->vmsc->msc->debugfs); + debugfs_create_x64("mpamf_idr", 0400, d, &ris->idr); + debugfs_create_x32("mpamf_cpor_idr", 0400, d, &ris->cpor_idr); + debugfs_create_x32("mpamf_ccap_idr", 0400, d, &ris->ccap_idr); + debugfs_create_ulong("features", 0400, d, &rprops->features[0]); + debugfs_create_x16("cpbm_wd", 0400, d, &rprops->cpbm_wd); + debugfs_create_x16("mbw_pbm_bits", 0400, d, &rprops->mbw_pbm_bits); + debugfs_create_x16("num_csu_mon", 0400, d, &rprops->num_csu_mon); + debugfs_create_x16("num_mbwu_mon", 0400, d, &rprops->num_mbwu_mon); + debugfs_create_cpumask("affinity", 0400, d, &ris->affinity); + ris->debugfs = d; +} + +static void mpam_debugfs_setup_vmsc(struct mpam_component *comp, + struct mpam_vmsc *vmsc) +{ + u8 ris_idx; + char name[40]; + char path[40]; + struct dentry *d; + struct mpam_msc_ris *ris; + int msc_id = vmsc->msc->id; + + snprintf(name, sizeof(name), "vmsc.%u", msc_id); + d = debugfs_create_dir(name, comp->debugfs); + debugfs_create_ulong("features", 0400, d, &vmsc->props.features[0]); + vmsc->debugfs = d; + + list_for_each_entry_rcu(ris, &vmsc->ris, vmsc_list) { + ris_idx = ris->ris_idx; + + snprintf(name, sizeof(name), "msc.%u_ris.%u", msc_id, + ris_idx); + snprintf(path, sizeof(path), "../../../msc.%u/ris.%u", + msc_id, ris_idx); + debugfs_create_symlink(name, d, path); + } +} + +static void mpam_debugfs_setup_comp(struct mpam_class *class, + struct mpam_component *comp) +{ + char name[40]; + struct dentry *d; + struct mpam_vmsc *vmsc; + + snprintf(name, sizeof(name), "comp.%u", comp->comp_id); + d = debugfs_create_dir(name, class->debugfs); + comp->debugfs = d; + + list_for_each_entry_rcu(vmsc, &comp->vmsc, comp_list) + mpam_debugfs_setup_vmsc(comp, vmsc); +} + +static void mpam_debugfs_setup(void) +{ + char name[40]; + struct dentry *d; + struct mpam_msc *msc; + struct mpam_class *class; + struct mpam_msc_ris *ris; + struct mpam_component *comp; + + lockdep_assert_held(&mpam_list_lock); + + list_for_each_entry(msc, &mpam_all_msc, all_msc_list) { + d = msc->debugfs; + debugfs_create_u32("fw_id", 0400, d, &msc->pdev->id); + debugfs_create_x32("iface", 0400, d, &msc->iface); + debugfs_create_x32("mpamf_iidr", 0400, d, &msc->iidr); + list_for_each_entry(ris, &msc->ris, msc_list) + mpam_debugfs_setup_ris(ris); + } + + list_for_each_entry_rcu(class, &mpam_classes, classes_list) { + snprintf(name, sizeof(name), "class.%u", class->level); + d = debugfs_create_dir(name, mpam_debugfs); + debugfs_create_ulong("features", 0400, d, &class->props.features[0]); + debugfs_create_x32("nrdy_usec", 0400, d, &class->nrdy_usec); + debugfs_create_x16("quirks", 0400, d, &class->quirks); + debugfs_create_x8("level", 0400, d, &class->level); + debugfs_create_cpumask("affinity", 0400, d, &class->affinity); + class->debugfs = d; + + list_for_each_entry_rcu(comp, &class->components, class_list) + mpam_debugfs_setup_comp(class, comp); + } +} + static void mpam_enable_once(void) { int err; @@ -2721,6 +2835,8 @@ static void mpam_enable_once(void) pr_err("Failed to allocate configuration arrays.\n"); break; } + + mpam_debugfs_setup(); } while (0); mutex_unlock(&mpam_list_lock); cpus_read_unlock(); @@ -2942,6 +3058,8 @@ static int __init mpam_msc_driver_init(void) return -EINVAL; } + mpam_debugfs = debugfs_create_dir("mpam", NULL); + return platform_driver_register(&mpam_msc_driver); } diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 60a8cebb33882..317936d47526d 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -130,6 +131,8 @@ struct mpam_msc { void __iomem *mapped_hwpage; size_t mapped_hwpage_sz; + struct dentry *debugfs; + /* Values only used on some platforms for quirks */ u32 t241_id; @@ -308,6 +311,7 @@ struct mpam_class { struct ida ida_csu_mon; struct ida ida_mbwu_mon; + struct dentry *debugfs; struct mpam_garbage garbage; }; @@ -342,6 +346,7 @@ struct mpam_component { /* parent: */ struct mpam_class *class; + struct dentry *debugfs; struct mpam_garbage garbage; }; @@ -360,12 +365,15 @@ struct mpam_vmsc { /* parent: */ struct mpam_component *comp; + struct dentry *debugfs; struct mpam_garbage garbage; }; struct mpam_msc_ris { u8 ris_idx; u64 idr; + u32 cpor_idr; + u32 ccap_idr; struct mpam_props props; bool in_reset_state; @@ -383,6 +391,7 @@ struct mpam_msc_ris { /* msmon mbwu configuration is preserved over reset */ struct msmon_mbwu_state *mbwu_state; + struct dentry *debugfs; struct mpam_garbage garbage; }; From a74a309590ebe99109369d8bd8f6e9241e1bc1db Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 29 Jul 2024 17:05:31 +0100 Subject: [PATCH 335/464] NVIDIA: VR: SAUCE: arm_mpam: Add force-disable debugfs trigger BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM has an error interrupt that can be triggered by an MSC when corrupt or out of range values are seen. The hardware only needs to raise an error interrupt if the error was detected, it is also permissible for the hardware to just use the corrupt or our of range value. All the reasons to raise an error indicate a software bug. When the error interrupt is triggered, the MPAM driver attempts to reset all the CPUs back to PARTID-0 and reset PARTID-0 to be unrestricted. This is done to ensure important tasks aren't accidentally given the performance of unimportant tasks. This teardown path in the driver is hard to trigger. Add a debugfs file to poke this manually. It is expected you have to reboot to make MPAM work again after this. Signed-off-by: James Morse (cherry picked from commit 2c4e1fed02be2c50642680f9d99a1c3424e5b7b6 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 8884ff775dd52..22f0e99efaa4f 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -2802,6 +2802,33 @@ static void mpam_debugfs_setup(void) } } +static int mpam_force_disable_show(struct seq_file *s, void *data) +{ + seq_puts(s, "Write 1 to this file to trigger an MPAM error.\n"); + return 0; +} + +static ssize_t mpam_force_disable_write(struct file *file, + const char __user *userbuf, size_t count, + loff_t *ppos) +{ + u32 user_val; + int err; + + err = kstrtou32_from_user(userbuf, count, 10, &user_val); + if (err) + return err; + + if (user_val == 1) { + mpam_disable_reason = "debugfs trigger"; + mpam_disable(NULL); + } + + return count; +} + +DEFINE_SHOW_STORE_ATTRIBUTE(mpam_force_disable); + static void mpam_enable_once(void) { int err; @@ -2841,6 +2868,9 @@ static void mpam_enable_once(void) mutex_unlock(&mpam_list_lock); cpus_read_unlock(); + debugfs_create_file("force_disable", 0600, mpam_debugfs, NULL, + &mpam_force_disable_fops); + if (!err) { err = mpam_resctrl_setup(); if (err) From 154491fb081507c88e7e003f7162fe995cb720ae Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 18 Jul 2025 12:02:57 +0100 Subject: [PATCH 336/464] NVIDIA: VR: SAUCE: arm_mpam: Expose the number of NRDY retries in debugfs BugLink: https://bugs.launchpad.net/bugs/2154527 It's really popular to tie NRDY high, and then act surprised when the OS never reads the counters, because they aren't ready. The spec obliges hardware to clear this bit automatically before the firmware advertised timeout. To make it easier to find errant hardware, count the number of retries and expose that number in debugfs. Signed-off-by: James Morse (cherry picked from commit 4fa427c7f312e037a8080dffc62663664b976905 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 5 ++++- drivers/resctrl/mpam_internal.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 22f0e99efaa4f..dc4b778255bd4 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -1309,8 +1309,10 @@ static void __ris_msmon_read(void *arg) } mpam_mon_sel_unlock(msc); - if (nrdy) + if (nrdy) { + msc->nrdy_retry_count++; m->err = -EBUSY; + } if (m->err) return; @@ -2783,6 +2785,7 @@ static void mpam_debugfs_setup(void) debugfs_create_u32("fw_id", 0400, d, &msc->pdev->id); debugfs_create_x32("iface", 0400, d, &msc->iface); debugfs_create_x32("mpamf_iidr", 0400, d, &msc->iidr); + debugfs_create_x64("nrdy_retry_count", 0400, d, &msc->nrdy_retry_count); list_for_each_entry(ris, &msc->ris, msc_list) mpam_debugfs_setup_ris(ris); } diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 317936d47526d..caff36dc2c146 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -68,6 +68,7 @@ struct mpam_msc { /* Not modified after mpam_is_enabled() becomes true */ enum mpam_msc_iface iface; u32 nrdy_usec; + u64 nrdy_retry_count; cpumask_t accessibility; bool has_extd_esr; From 3a607bbc0c3562b7a666300208fa242c87d392a3 Mon Sep 17 00:00:00 2001 From: Dave Martin Date: Fri, 15 Aug 2025 15:43:56 +0100 Subject: [PATCH 337/464] NVIDIA: SAUCE: arm_mpam: Add resctrl_arch_round_bw() BugLink: https://bugs.launchpad.net/bugs/2154527 Add the required hook to pre-round a userspace memory bandwidth allocation percentage value to a value acceptable to the driver backend. For MPAM, no rounding is needed because the driver has all the information necessary for rounding the value when resctrl_arch_update_one() is called. So, just "round" the value to itself here. Signed-off-by: Dave Martin Signed-off-by: James Morse (cherry picked from commit 935611d607afe707a00b0311fdbb500b8acdd654 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `include/linux/arm_mpam.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/arm_mpam.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index f92a36187a527..4ccf32fe07fd5 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -5,6 +5,7 @@ #define __LINUX_ARM_MPAM_H #include +#include #include #include @@ -76,6 +77,19 @@ static inline void resctrl_arch_disable_mon(void) { } static inline void resctrl_arch_enable_alloc(void) { } static inline void resctrl_arch_disable_alloc(void) { } +struct resctrl_schema; + +struct rdt_resource; +static inline u32 resctrl_arch_round_bw(u32 val, + const struct rdt_resource *r __always_unused) +{ + /* + * Do nothing: for MPAM, resctrl_arch_update_one() has the necessary + * context to round the incoming value correctly. + */ + return val; +} + static inline unsigned int resctrl_arch_round_mon_val(unsigned int val) { return val; From f02d99c529c6e251803a9ca99d337a4751966e8f Mon Sep 17 00:00:00 2001 From: Dave Martin Date: Fri, 15 Aug 2025 15:43:55 +0100 Subject: [PATCH 338/464] NVIDIA: SAUCE: fs/resctrl,x86/resctrl: Factor mba rounding to be per-arch BugLink: https://bugs.launchpad.net/bugs/2154527 The control value parser for the MB resource currently coerces the memory bandwidth percentage value from userspace to be an exact multiple of the bw_gran parameter. On MPAM systems, this results in somewhat worse-than-worst-case rounding, since bw_gran is in general only an approximation to the actual hardware granularity, and the hardware bandwidth allocation control value is not natively a percentage. Allow the arch to provide its own conversion that is appropriate for the hardware, and move the existing conversion to x86. This will avoid accumulated error from rounding the value twice on MPAM systems. Clarify the documentation, but avoid overly exact promises. Clamping to bw_min and bw_max still feels generic: leave it in the core code, for now. No functional change. Signed-off-by: Dave Martin Signed-off-by: James Morse (cherry picked from commit cabdc680e1dde14521ab2a61ff32b525b3ba334e https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/filesystems/resctrl.rst | 7 +++---- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 6 ++++++ fs/resctrl/ctrlmondata.c | 2 +- include/linux/resctrl.h | 2 ++ 4 files changed, 12 insertions(+), 5 deletions(-) diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst index b003bed339fdd..e9ff59c2e57e8 100644 --- a/Documentation/filesystems/resctrl.rst +++ b/Documentation/filesystems/resctrl.rst @@ -236,12 +236,11 @@ with respect to allocation: user can request. "bandwidth_gran": - The granularity in which the memory bandwidth + The approximate granularity in which the memory bandwidth percentage is allocated. The allocated b/w percentage is rounded off to the next - control step available on the hardware. The - available bandwidth control steps are: - min_bandwidth + N * bandwidth_gran. + control step available on the hardware. The available + steps are at least as small as this value. "delay_linear": Indicates if the delay scale is linear or diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c index b20e705606b8f..d539e56c2b1f0 100644 --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c @@ -16,9 +16,15 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include "internal.h" +u32 resctrl_arch_round_bw(u32 val, const struct rdt_resource *r) +{ + return roundup(val, (unsigned long)r->membw.bw_gran); +} + int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, u32 closid, enum resctrl_conf_type t, u32 cfg_val) { diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 9a7dfc48cb2e2..0c02451c687b2 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -71,7 +71,7 @@ static bool bw_validate(char *buf, u32 *data, struct rdt_resource *r) return false; } - *data = roundup(bw, (unsigned long)r->membw.bw_gran); + *data = resctrl_arch_round_bw(bw, r); return true; } diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 006e57fd7ca58..b236c4e9cb619 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -500,6 +500,8 @@ bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r); */ int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable); +u32 resctrl_arch_round_bw(u32 val, const struct rdt_resource *r); + /* * Update the ctrl_val and apply this config right now. * Must be called on one of the domain's CPUs. From 0d9b271e94669cbe1865d8d081525dd2ed953628 Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 29 Sep 2025 14:29:42 +0100 Subject: [PATCH 339/464] NVIDIA: VR: SAUCE: arm_mpam: Split the locking around the mon_sel registers BugLink: https://bugs.launchpad.net/bugs/2154527 The MSC MON_SEL register needs to be accessed from hardirq for the overflow interrupt, and when taking an IPI to access these registers on platforms where MSC are not accesible from every CPU. This makes an irqsave spinlock the obvious lock to protect these registers. On systems with SCMI mailboxes it must be able to sleep, meaning a mutex must be used. The SCMI platforms can't support an overflow interrupt. Clearly these two can't exist for one MSC at the same time. Split the existing helper into a raw spinlock and a mutex, named inner and outer. The outer lock must be taken in an a pre-emptible context befroe the inner lock can be taken. On systems with SCMI mailboxes where the MON_SEL accesses must sleep - the inner lock will fail tobe taken if the caller is unable to sleep. This will allow callers to fail withuot having to explicitly check the interface type of each MSC. Signed-off-by: James Morse (forward ported from commit 46584f5584d0d2eb939b0ab0e43b93e6a0665096 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; - Resolve minor conflicts in `drivers/resctrl/mpam_internal.h`; - Add outer lock in mpam_cpu_offline(); - Fix outer lock issue in __allocate_component_cfg() by moving __destroy_component_cfg() outside outer lock scope; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 52 +++++++++++++++++-------- drivers/resctrl/mpam_internal.h | 68 ++++++++++++++++++++++++--------- 2 files changed, 86 insertions(+), 34 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index dc4b778255bd4..fe3da1d805f11 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -750,7 +750,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) bool can_set, can_clear; struct mpam_msc *msc = ris->vmsc->msc; - if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) + if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc))) return false; mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) | @@ -774,7 +774,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1); now = _mpam_read_monsel_reg(msc, MSMON_CSU); can_clear = !(now & MSMON___NRDY); - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); return (!can_set || !can_clear); } @@ -895,7 +895,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) mpam_set_feature(mpam_feat_msmon_csu_xcl, props); /* Is NRDY hardware managed? */ + mpam_mon_sel_outer_lock(msc); hw_managed = mpam_ris_hw_probe_csu_nrdy(ris); + mpam_mon_sel_outer_unlock(msc); /* * Accept the missing firmware property if NRDY appears @@ -1210,7 +1212,7 @@ static void __ris_msmon_read(void *arg) struct mpam_msc *msc = m->ris->vmsc->msc; u32 mon_sel, ctl_val, flt_val, cur_ctl, cur_flt; - if (!mpam_mon_sel_lock(msc)) { + if (!mpam_mon_sel_inner_lock(msc)) { m->err = -EIO; return; } @@ -1307,7 +1309,7 @@ static void __ris_msmon_read(void *arg) default: m->err = -EINVAL; } - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); if (nrdy) { msc->nrdy_retry_count++; @@ -1331,6 +1333,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg) struct mpam_msc *msc = vmsc->msc; struct mpam_msc_ris *ris; + mpam_mon_sel_outer_lock(msc); list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list, srcu_read_lock_held(&mpam_srcu)) { arg->ris = ris; @@ -1349,6 +1352,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg) if (err) any_err = err; } + mpam_mon_sel_outer_unlock(msc); } return any_err; @@ -1431,18 +1435,20 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx) continue; msc = vmsc->msc; + mpam_mon_sel_outer_lock(msc); list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list, srcu_read_lock_held(&mpam_srcu)) { if (!mpam_has_feature(mpam_feat_msmon_mbwu, &ris->props)) continue; - if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) + if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc))) continue; ris->mbwu_state[ctx->mon].correction = 0; ris->mbwu_state[ctx->mon].reset_on_next_read = true; - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); } + mpam_mon_sel_outer_unlock(msc); } } @@ -1643,8 +1649,11 @@ static int mpam_restore_mbwu_state(void *_ris) u64 val; struct mon_read mwbu_arg; struct mpam_msc_ris *ris = _ris; + struct mpam_msc *msc = ris->vmsc->msc; struct mpam_class *class = ris->vmsc->comp->class; + mpam_mon_sel_outer_lock(msc); + for (i = 0; i < ris->props.num_mbwu_mon; i++) { if (ris->mbwu_state[i].enabled) { mwbu_arg.ris = ris; @@ -1656,10 +1665,12 @@ static int mpam_restore_mbwu_state(void *_ris) } } + mpam_mon_sel_outer_unlock(msc); + return 0; } -/* Call with MSC cfg_lock held */ +/* Call with MSC lock and outer mon_sel lock held */ static int mpam_save_mbwu_state(void *arg) { int i; @@ -1674,7 +1685,7 @@ static int mpam_save_mbwu_state(void *arg) mbwu_state = &ris->mbwu_state[i]; cfg = &mbwu_state->cfg; - if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc))) + if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc))) return -EIO; mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) | @@ -1699,7 +1710,7 @@ static int mpam_save_mbwu_state(void *arg) cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt); mbwu_state->correction += val; mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl); - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); } return 0; @@ -1894,6 +1905,7 @@ static int mpam_cpu_offline(unsigned int cpu) struct mpam_msc_ris *ris; mutex_lock(&msc->cfg_lock); + mpam_mon_sel_outer_lock(msc); list_for_each_entry_srcu(ris, &msc->ris, msc_list, srcu_read_lock_held(&mpam_srcu)) { mpam_touch_msc(msc, &mpam_reset_ris, ris); @@ -1907,6 +1919,7 @@ static int mpam_cpu_offline(unsigned int cpu) if (mpam_is_enabled()) mpam_touch_msc(msc, &mpam_save_mbwu_state, ris); } + mpam_mon_sel_outer_unlock(msc); mutex_unlock(&msc->cfg_lock); } } @@ -2608,11 +2621,13 @@ static void __destroy_component_cfg(struct mpam_component *comp) list_for_each_entry(vmsc, &comp->vmsc, comp_list) { msc = vmsc->msc; - if (mpam_mon_sel_lock(msc)) { + mpam_mon_sel_outer_lock(msc); + if (mpam_mon_sel_inner_lock(msc)) { list_for_each_entry(ris, &vmsc->ris, vmsc_list) add_to_garbage(ris->mbwu_state); - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); } + mpam_mon_sel_outer_unlock(msc); } } @@ -2659,6 +2674,7 @@ static int __allocate_component_cfg(struct mpam_component *comp) mpam_reset_component_cfg(comp); list_for_each_entry(vmsc, &comp->vmsc, comp_list) { + int err = 0; struct mpam_msc *msc; struct mpam_msc_ris *ris; struct msmon_mbwu_state *mbwu_state; @@ -2667,6 +2683,7 @@ static int __allocate_component_cfg(struct mpam_component *comp) continue; msc = vmsc->msc; + mpam_mon_sel_outer_lock(msc); list_for_each_entry(ris, &vmsc->ris, vmsc_list) { if (!ris->props.num_mbwu_mon) continue; @@ -2674,17 +2691,22 @@ static int __allocate_component_cfg(struct mpam_component *comp) mbwu_state = kzalloc_objs(*ris->mbwu_state, ris->props.num_mbwu_mon); if (!mbwu_state) { - __destroy_component_cfg(comp); - return -ENOMEM; + err = -ENOMEM; + break; } init_garbage(&mbwu_state[0].garbage); - if (mpam_mon_sel_lock(msc)) { + if (mpam_mon_sel_inner_lock(msc)) { ris->mbwu_state = mbwu_state; - mpam_mon_sel_unlock(msc); + mpam_mon_sel_inner_unlock(msc); } } + mpam_mon_sel_outer_unlock(msc); + if (err) { + __destroy_component_cfg(comp); + return err; + } } return 0; diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index caff36dc2c146..5505d26d53daf 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -118,16 +118,20 @@ struct mpam_msc { /* * mon_sel_lock protects access to the MSC hardware registers that are * affected by MPAMCFG_MON_SEL, and the mbwu_state. - * Access to mon_sel is needed from both process and interrupt contexts, - * but is complicated by firmware-backed platforms that can't make any - * access unless they can sleep. - * Always use the mpam_mon_sel_lock() helpers. - * Accesses to mon_sel need to be able to fail if they occur in the wrong - * context. + * Both the 'inner' and 'outer' must be taken. + * For real MMIO MSC, the outer lock is unnecessary - but keeps the + * code common with: + * Firmware backed MSC need to sleep when accessing the MSC, which + * means some code-paths will always fail. For these MSC the outer + * lock is providing the protection, and the inner lock fails to + * be taken if the task is unable to sleep. + * * If needed, take msc->probe_lock first. */ - raw_spinlock_t _mon_sel_lock; - unsigned long _mon_sel_flags; + struct mutex outer_mon_sel_lock; + bool outer_lock_held; + raw_spinlock_t inner_mon_sel_lock; + unsigned long inner_mon_sel_flags; void __iomem *mapped_hwpage; size_t mapped_hwpage_sz; @@ -140,30 +144,56 @@ struct mpam_msc { struct mpam_garbage garbage; }; -/* Returning false here means accesses to mon_sel must fail and report an error. */ -static inline bool __must_check mpam_mon_sel_lock(struct mpam_msc *msc) +static inline bool __must_check mpam_mon_sel_inner_lock(struct mpam_msc *msc) { - /* Locking will require updating to support a firmware backed interface */ - if (WARN_ON_ONCE(msc->iface != MPAM_IFACE_MMIO)) - return false; + /* + * The outer lock may be taken by a CPU that then issues an IPI to run + * a helper that takes the inner lock. lockdep can't help us here. + */ + WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held)); + + if (msc->iface == MPAM_IFACE_MMIO) { + raw_spin_lock_irqsave(&msc->inner_mon_sel_lock, msc->inner_mon_sel_flags); + return true; + } + + /* Accesses must fail if we are not pre-emptible */ + return !!preemptible(); +} - raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags); - return true; +static inline void mpam_mon_sel_inner_unlock(struct mpam_msc *msc) +{ + WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held)); + + if (msc->iface == MPAM_IFACE_MMIO) + raw_spin_unlock_irqrestore(&msc->inner_mon_sel_lock, msc->inner_mon_sel_flags); +} + +static inline void mpam_mon_sel_outer_lock(struct mpam_msc *msc) +{ + mutex_lock(&msc->outer_mon_sel_lock); + msc->outer_lock_held = true; } -static inline void mpam_mon_sel_unlock(struct mpam_msc *msc) +static inline void mpam_mon_sel_outer_unlock(struct mpam_msc *msc) { - raw_spin_unlock_irqrestore(&msc->_mon_sel_lock, msc->_mon_sel_flags); + msc->outer_lock_held = false; + mutex_unlock(&msc->outer_mon_sel_lock); } static inline void mpam_mon_sel_lock_held(struct mpam_msc *msc) { - lockdep_assert_held_once(&msc->_mon_sel_lock); + WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held)); + if (msc->iface == MPAM_IFACE_MMIO) + lockdep_assert_held_once(&msc->inner_mon_sel_lock); + else + lockdep_assert_preemption_enabled(); } static inline void mpam_mon_sel_lock_init(struct mpam_msc *msc) { - raw_spin_lock_init(&msc->_mon_sel_lock); + raw_spin_lock_init(&msc->inner_mon_sel_lock); + mutex_init(&msc->outer_mon_sel_lock); } /* Bits for mpam features bitmaps */ From 7a882f69a85bd7ed265e4d6b4080b58fd677514d Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 29 Oct 2021 16:13:51 +0100 Subject: [PATCH 340/464] NVIDIA: VR: SAUCE: arm_mpam: Allow the maximum partid to be overridden from the command line BugLink: https://bugs.launchpad.net/bugs/2154527 MPAMs bandwidth monitors are only available via resctrl if there are enough monitors for each combination of partid and pmg to have one. As it is unlikely anyone built that many monitors, allow the maximum partid the system will use to be set from the kernel command-line. With this, it should be possible for bandwidth monitors to be enabled by reducing the number of partid in use. Signed-off-by: James Morse (cherry picked from commit f12f00ec8d977b8ea8c78986ef34cd9c898e8b2b https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 38 ++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index fe3da1d805f11..a689a288867f8 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -65,6 +66,8 @@ static DEFINE_MUTEX(mpam_cpuhp_state_lock); u16 mpam_partid_max; u8 mpam_pmg_max; static bool partid_max_init, partid_max_published; +static u16 mpam_cmdline_partid_max; +static bool mpam_cmdline_partid_max_overridden; static DEFINE_SPINLOCK(partid_max_lock); /* @@ -308,6 +311,9 @@ int mpam_register_requestor(u16 partid_max, u8 pmg_max) return -EBUSY; } + if (mpam_cmdline_partid_max_overridden) + mpam_partid_max = min(mpam_cmdline_partid_max, mpam_partid_max); + return 0; } EXPORT_SYMBOL(mpam_register_requestor); @@ -3121,6 +3127,38 @@ static int __init mpam_msc_driver_init(void) /* Must occur after arm64_mpam_register_cpus() from arch_initcall() */ subsys_initcall(mpam_msc_driver_init); +static int mpam_cmdline_partid_max_set(const char *arg, + const struct kernel_param *kp) +{ + int ret; + + spin_lock(&partid_max_lock); + ret = kstrtou16(arg, 10, &mpam_cmdline_partid_max); + if (!ret) + mpam_cmdline_partid_max_overridden = true; + spin_unlock(&partid_max_lock); + + return 0; +} +static int mpam_cmdline_partid_max_get(char *buffer, + const struct kernel_param *kp) +{ + u16 val = 0xffff; + + spin_lock(&partid_max_lock); + if (mpam_cmdline_partid_max_overridden) + val = mpam_cmdline_partid_max; + spin_unlock(&partid_max_lock); + + return sprintf(buffer, "%u\n", val); +} +static const struct kernel_param_ops mpam_cmdline_partid_max_ops = { + .set = mpam_cmdline_partid_max_set, + .get = mpam_cmdline_partid_max_get, +}; +module_param_cb(partid_max, &mpam_cmdline_partid_max_ops, NULL, 0644); +MODULE_PARM_DESC(partid_max, "Override for reducing the number of PARTID."); + #ifdef CONFIG_MPAM_KUNIT_TEST #include "test_mpam_devices.c" #endif From 5ddc3a8946344caac624e699987aba2aae7c1bf6 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 1 Jul 2025 17:03:13 +0100 Subject: [PATCH 341/464] NVIDIA: VR: SAUCE: arm_mpam: Allow MSC to be forced to have an unknown location BugLink: https://bugs.launchpad.net/bugs/2154527 The MPAM driver discovers which MSC control which system resources from firmware tables. The MPAM resctrl picking code then attempts to export platforms that are Xeon shaped via resctrl. Occasionally, the presence of one or more MSC prevents the platform being described as Xeon shaped, and exposed via resctrl. For example with CPU-less NUMA nodes. The additional node doensn't have an L3, so can't have domain-ids exposed for the 'MB' memory bandwidth controls. In this example, some users would prefer to control bandwidth on just the CPU nodes, instead of having nothing at all. Allow users an amount of wiggle room by allowing MSC to be forced to be treated as unknown. This effectively disables parts of the MPAM functionality. Unknown MSC are not disabled, They are still probed and contribute to the system wide properties. Suggested-by: Dave Martin Signed-off-by: James Morse (cherry picked from commit 542e79e9f52b4a9889de0c586a9db2bed5ecfa03 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 64 +++++++++++++++++++++++++++++++++ drivers/resctrl/mpam_internal.h | 2 ++ 2 files changed, 66 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index a689a288867f8..2e9b08eabb9b1 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -27,6 +28,7 @@ #include #include #include +#include #include "mpam_internal.h" @@ -629,6 +631,9 @@ int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, { int err; + if (mpam_force_unknown_msc_test(msc)) + type = MPAM_CLASS_UNKNOWN; + mutex_lock(&mpam_list_lock); err = mpam_ris_create_locked(msc, ris_idx, type, class_id, component_id); @@ -3159,6 +3164,65 @@ static const struct kernel_param_ops mpam_cmdline_partid_max_ops = { module_param_cb(partid_max, &mpam_cmdline_partid_max_ops, NULL, 0644); MODULE_PARM_DESC(partid_max, "Override for reducing the number of PARTID."); +static DEFINE_XARRAY(mpam_force_unkown_msc); + +static void mpam_force_unknown_msc_add(u32 msc_id, gfp_t gfp) +{ + xa_store(&mpam_force_unkown_msc, msc_id, xa_mk_value(msc_id), gfp); +} + +bool mpam_force_unknown_msc_test(struct mpam_msc *msc) +{ + return !!xa_load(&mpam_force_unkown_msc, msc->pdev->id); +} + +static int mpam_force_unknown_msc_set(const char *_str, + const struct kernel_param *kp) +{ + int err; + u32 val; + char *tok, *iter; + char *str __free(kfree) = kstrdup(_str, GFP_KERNEL); + + iter = str; + do { + tok = strsep(&iter, ","); + err = kstrtou32(tok, 10, &val); + if (err) { + pr_err("Failed to parse commandline: %d\n", err); + break; + } + mpam_force_unknown_msc_add(val, GFP_KERNEL); + } while (iter); + + return 0; +} +static int mpam_force_unknown_msc_get(char *buffer, + const struct kernel_param *kp) +{ + unsigned long index, count = 0; + int result = 0; + void *entry; + + xa_for_each(&mpam_force_unkown_msc, index, entry) { + if (count) + result += sprintf(buffer + result, ","); + + result += sprintf(buffer + result, "%lu", index); + count += 1; + } + + result += sprintf(buffer + result, "\n"); + + return result; +} +static const struct kernel_param_ops mpam_force_unknown_msc_ops = { + .set = mpam_force_unknown_msc_set, + .get = mpam_force_unknown_msc_get, +}; +subsys_param_cb(force_unknown_msc, &mpam_force_unknown_msc_ops, NULL, 0644); +MODULE_PARM_DESC(force_unknown_msc, "Disabling a set of probed MSC."); + #ifdef CONFIG_MPAM_KUNIT_TEST #include "test_mpam_devices.c" #endif diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 5505d26d53daf..1ec5e7072c0fd 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -522,6 +522,8 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx); int mpam_get_cpumask_from_cache_id(unsigned long cache_id, u32 cache_level, cpumask_t *affinity); +bool mpam_force_unknown_msc_test(struct mpam_msc *msc); + #ifdef CONFIG_RESCTRL_FS int mpam_resctrl_setup(void); void mpam_resctrl_exit(void); From 1bf2d6c7b57803c20fe3b1f8bd53ebecd245639b Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 15 Sep 2022 18:00:40 +0100 Subject: [PATCH 342/464] NVIDIA: VR: SAUCE: fs/resctrl: Add this_is_not_abi mount option BugLink: https://bugs.launchpad.net/bugs/2154527 Some later things in the MPAM tree enable behaviour that resctrl doesn't have upstream. To make it clear to people using the out-of-tree code that they shouldn't be relying on this in user-space, add a mount option to enable this stuff. Signed-off-by: James Morse (forward ported from commit 8bd00259ac52ebb244ced984c744135e8d7f4b7d https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/rdtgroup.c`; - Call disable_abi_playground() on out path in rdt_get_tree(); ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/internal.h | 3 +++ fs/resctrl/rdtgroup.c | 63 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h index 1a9b29119f88f..740e32a7c78e2 100644 --- a/fs/resctrl/internal.h +++ b/fs/resctrl/internal.h @@ -42,6 +42,7 @@ struct rdt_fs_context { bool enable_cdpl3; bool enable_mba_mbps; bool enable_debug; + bool enable_abi_playground; }; static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) @@ -314,6 +315,8 @@ struct mbm_state { u32 prev_bw; }; +DECLARE_STATIC_KEY_FALSE(resctrl_abi_playground); + extern struct mutex rdtgroup_mutex; static inline const char *rdt_kn_name(const struct kernfs_node *kn) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 5dfdaa6f9d8ff..52dd273046722 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -87,6 +87,9 @@ enum resctrl_event_id mba_mbps_default_event; static bool resctrl_debug; +/* Enable wacky behaviour that is not supported upstream. */ +DEFINE_STATIC_KEY_FALSE(resctrl_abi_playground); + void rdt_last_cmd_clear(void) { lockdep_assert_held(&rdtgroup_mutex); @@ -2782,6 +2785,42 @@ static void schemata_list_destroy(void) } } +static void hack_file_mode(const char *name, u16 mode) +{ + struct rftype *rfts, *rft; + int len; + + mutex_lock(&rdtgroup_mutex); + + rfts = res_common_files; + len = ARRAY_SIZE(res_common_files); + + for (rft = rfts; rft < rfts + len; rft++) { + if (!strcmp(rft->name, name)) + rft->mode = mode; + } + + mutex_unlock(&rdtgroup_mutex); +} + +static void enable_abi_playground(void) +{ + static_key_enable(&resctrl_abi_playground.key); + + /* Make the tasks file read only */ + if (IS_ENABLED(CONFIG_CGROUP_RESCTRL)) + hack_file_mode("tasks", 0444); +} + +static void disable_abi_playground(void) +{ + static_key_disable(&resctrl_abi_playground.key); + + /* Make the tasks file read/write only */ + if (IS_ENABLED(CONFIG_CGROUP_RESCTRL)) + hack_file_mode("tasks", 0644); +} + static int rdt_get_tree(struct fs_context *fc) { struct rdt_fs_context *ctx = rdt_fc2context(fc); @@ -2792,6 +2831,9 @@ static int rdt_get_tree(struct fs_context *fc) DO_ONCE_SLEEPABLE(resctrl_arch_pre_mount); + if (ctx->enable_abi_playground) + enable_abi_playground(); + cpus_read_lock(); mutex_lock(&rdtgroup_mutex); /* @@ -2899,6 +2941,10 @@ static int rdt_get_tree(struct fs_context *fc) rdt_last_cmd_clear(); mutex_unlock(&rdtgroup_mutex); cpus_read_unlock(); + + if (ret && ctx->enable_abi_playground) + disable_abi_playground(); + return ret; } @@ -2907,6 +2953,7 @@ enum rdt_param { Opt_cdpl2, Opt_mba_mbps, Opt_debug, + Opt_not_abi_playground, nr__rdt_params }; @@ -2915,6 +2962,13 @@ static const struct fs_parameter_spec rdt_fs_parameters[] = { fsparam_flag("cdpl2", Opt_cdpl2), fsparam_flag("mba_MBps", Opt_mba_mbps), fsparam_flag("debug", Opt_debug), + + /* + * Some of MPAM's out of tree code exposes things through resctrl + * that need much more discussion before they are considered for + * mainline. Add a mount option that can be used to hide these crimes. + */ + fsparam_flag("this_is_not_abi", Opt_not_abi_playground), {} }; @@ -2945,6 +2999,9 @@ static int rdt_parse_param(struct fs_context *fc, struct fs_parameter *param) case Opt_debug: ctx->enable_debug = true; return 0; + case Opt_not_abi_playground: + ctx->enable_abi_playground = true; + return 0; } return -EINVAL; @@ -3191,6 +3248,9 @@ static void rdt_kill_sb(struct super_block *sb) kernfs_kill_sb(sb); mutex_unlock(&rdtgroup_mutex); cpus_read_unlock(); + + if (static_branch_unlikely(&resctrl_abi_playground)) + disable_abi_playground(); } static struct file_system_type rdt_fs_type = { @@ -4243,6 +4303,9 @@ static int rdtgroup_show_options(struct seq_file *seq, struct kernfs_root *kf) if (resctrl_debug) seq_puts(seq, ",debug"); + if (static_branch_unlikely(&resctrl_abi_playground)) + seq_puts(seq, ",this_is_not_abi"); + return 0; } From 4bfc66310af1d74571c18347f9e0f7f16d5b7eb7 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 17 Sep 2021 13:19:13 +0100 Subject: [PATCH 343/464] NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Register SMMU capabilities with MPAM BugLink: https://bugs.launchpad.net/bugs/2154527 Traffic in the system can be tagged with a PARTID and PMG. Different requestors can support a different number of bits for these fields. Before MPAM can be used, the MPAM driver has to discover the minimum number of bits supported by any requestor, which affects the range of PARTID and PMG that can be used. Detect whether the SMMU supports MPAM, if it does provide the MPAM driver with the maximum PARTID and PMG values. Tested-by: Amit Singh Tomar Signed-off-by: James Morse (cherry picked from commit 254691aaeac0832fd3daa0bab0ec5ba18c93bdc2 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 28 +++++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +++++ 2 files changed, 34 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index f033288d91116..48c63bc24f85e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -4448,6 +4449,29 @@ static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg) hw_features, fw_features); } +static void arm_smmu_mpam_register_smmu(struct arm_smmu_device *smmu) +{ + u16 partid_max; + u8 pmg_max; + u32 reg; + + if (!IS_ENABLED(CONFIG_ARM64_MPAM)) + return; + + if (!(smmu->features & ARM_SMMU_FEAT_MPAM)) + return; + + reg = readl_relaxed(smmu->base + ARM_SMMU_MPAMIDR); + if (!reg) + return; + + partid_max = FIELD_GET(SMMU_MPAMIDR_PARTID_MAX, reg); + pmg_max = FIELD_GET(SMMU_MPAMIDR_PMG_MAX, reg); + + if (mpam_register_requestor(partid_max, pmg_max)) + smmu->features &= ~ARM_SMMU_FEAT_MPAM; +} + static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) { u32 reg; @@ -4595,6 +4619,8 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) smmu->features |= ARM_SMMU_FEAT_RANGE_INV; if (FIELD_GET(IDR3_FWB, reg)) smmu->features |= ARM_SMMU_FEAT_S2FWB; + if (FIELD_GET(IDR3_MPAM, reg)) + smmu->features |= ARM_SMMU_FEAT_MPAM; if (FIELD_GET(IDR3_BBM, reg) == 2) smmu->features |= ARM_SMMU_FEAT_BBML2; @@ -4660,6 +4686,8 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (arm_smmu_sva_supported(smmu)) smmu->features |= ARM_SMMU_FEAT_SVA; + arm_smmu_mpam_register_smmu(smmu); + dev_info(smmu->dev, "oas %lu-bit (features 0x%08x)\n", smmu->oas, smmu->features); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index f966d474b61fd..f98b3a403adc5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -58,6 +58,7 @@ struct arm_vsmmu; #define IDR1_SIDSIZE GENMASK(5, 0) #define ARM_SMMU_IDR3 0xc +#define IDR3_MPAM (1 << 7) #define IDR3_FWB (1 << 8) #define IDR3_RIL (1 << 10) #define IDR3_BBM GENMASK(12, 11) @@ -169,6 +170,10 @@ struct arm_vsmmu; #define ARM_SMMU_PRIQ_IRQ_CFG1 0xd8 #define ARM_SMMU_PRIQ_IRQ_CFG2 0xdc +#define ARM_SMMU_MPAMIDR 0x130 +#define SMMU_MPAMIDR_PARTID_MAX GENMASK(15, 0) +#define SMMU_MPAMIDR_PMG_MAX GENMASK(23, 16) + #define ARM_SMMU_REG_SZ 0xe00 /* Common MSI config fields */ @@ -767,6 +772,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_HD (1 << 22) #define ARM_SMMU_FEAT_S2FWB (1 << 23) #define ARM_SMMU_FEAT_BBML2 (1 << 24) +#define ARM_SMMU_FEAT_MPAM (1 << 25) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) From f1c8da0cfad3f5947a0ae78354aff123393a5eea Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 14 Sep 2021 17:57:42 +0100 Subject: [PATCH 344/464] NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Add mpam helpers to query and set state BugLink: https://bugs.launchpad.net/bugs/2154527 To allow an iommu_group to be moved between resctrl groups as if it were a CPU thread, the mpam driver needs to be able to set the partid and pmg for the iommu_group. Use the properties in the STE, as these only apply to one stream. The MPAM driver also needs to know the maximum partid and pmg values that the SMMU can generate. This allows it to determine the system-wide common supported range of values. Add a helper to return this id register. Tested-by: Amit Singh Tomar Signed-off-by: James Morse (cherry picked from commit d847012696d29c61687420d4a6621f1f9e9bf95d https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/iommu/iommu.c`; - Resolve minor conflicts in `include/linux/iommu.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 92 +++++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 7 ++ drivers/iommu/iommu.c | 6 ++ include/linux/iommu.h | 7 ++ 4 files changed, 112 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 48c63bc24f85e..9f6154bed59c6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3819,6 +3819,96 @@ static int arm_smmu_def_domain_type(struct device *dev) return 0; } +static int arm_smmu_group_set_mpam(struct iommu_group *group, u16 partid, + u8 pmg) +{ + int i; + u32 sid; + unsigned long flags; + struct arm_smmu_ste *step; + struct iommu_domain *domain; + struct arm_smmu_device *smmu; + struct arm_smmu_master *master; + struct arm_smmu_cmdq_batch cmds; + struct arm_smmu_domain *smmu_domain; + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_CFGI_STE, + .cfgi = { + .leaf = true, + }, + }; + struct arm_smmu_master_domain *master_domain; + + domain = iommu_get_domain_for_group(group); + smmu_domain = to_smmu_domain(domain); + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_MPAM)) + return -EIO; + smmu = smmu_domain->smmu; + + arm_smmu_cmdq_batch_init(smmu, &cmds, &cmd); + + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + master = master_domain->master; + + for (i = 0; i < master->num_streams; i++) { + sid = master->streams[i].id; + step = arm_smmu_get_step_for_sid(smmu, sid); + + /* These need locking if the VMSPtr is ever used */ + step->data[4] = FIELD_PREP(STRTAB_STE_4_PARTID, partid); + step->data[5] = FIELD_PREP(STRTAB_STE_5_PMG, pmg); + + cmd.cfgi.sid = sid; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); + } + + master->partid = partid; + master->pmg = pmg; + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + arm_smmu_cmdq_batch_submit(smmu, &cmds); + + return 0; +} + +static int arm_smmu_group_get_mpam(struct iommu_group *group, u16 *partid, + u8 *pmg) +{ + int err = -EINVAL; + unsigned long flags; + struct iommu_domain *domain; + struct arm_smmu_master *master; + struct arm_smmu_domain *smmu_domain; + struct arm_smmu_master_domain *master_domain; + + domain = iommu_get_domain_for_group(group); + smmu_domain = to_smmu_domain(domain); + if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_MPAM)) + return -EIO; + + if (!partid && !pmg) + return 0; + + spin_lock_irqsave(&smmu_domain->devices_lock, flags); + list_for_each_entry(master_domain, &smmu_domain->devices, + devices_elm) { + master = master_domain->master; + if (master) { + if (partid) + *partid = master->partid; + if (pmg) + *pmg = master->pmg; + err = 0; + } + } + spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + return err; +} + static const struct iommu_ops arm_smmu_ops = { .identity_domain = &arm_smmu_identity_domain, .blocked_domain = &arm_smmu_blocked_domain, @@ -3832,6 +3922,8 @@ static const struct iommu_ops arm_smmu_ops = { .device_group = arm_smmu_device_group, .of_xlate = arm_smmu_of_xlate, .get_resv_regions = arm_smmu_get_resv_regions, + .get_group_qos_params = arm_smmu_group_get_mpam, + .set_group_qos_params = arm_smmu_group_set_mpam, .page_response = arm_smmu_page_response, .def_domain_type = arm_smmu_def_domain_type, .get_viommu_size = arm_smmu_get_viommu_size, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index f98b3a403adc5..1d9306232e0fe 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -275,6 +275,7 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) #define STRTAB_STE_1_MEV (1UL << 19) #define STRTAB_STE_1_S2FWB (1UL << 25) #define STRTAB_STE_1_S1STALLD (1UL << 27) +#define STRTAB_STE_1_S1MPAM (1UL << 26) #define STRTAB_STE_1_EATS GENMASK_ULL(29, 28) #define STRTAB_STE_1_EATS_ABT 0UL @@ -305,6 +306,10 @@ static inline u32 arm_smmu_strtab_l2_idx(u32 sid) #define STRTAB_STE_3_S2TTB_MASK GENMASK_ULL(51, 4) +#define STRTAB_STE_4_PARTID GENMASK_ULL(31, 16) + +#define STRTAB_STE_5_PMG GENMASK_ULL(7, 0) + /* These bits can be controlled by userspace for STRTAB_STE_0_CFG_NESTED */ #define STRTAB_STE_0_NESTING_ALLOWED \ cpu_to_le64(STRTAB_STE_0_V | STRTAB_STE_0_CFG | STRTAB_STE_0_S1FMT | \ @@ -857,6 +862,8 @@ struct arm_smmu_master { bool ats_always_on; unsigned int ssid_bits; unsigned int iopf_refcount; + u16 partid; + u8 pmg; }; /* SMMU private data for an IOMMU domain */ diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 93c9081707401..e92af77048673 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2145,6 +2145,12 @@ void iommu_domain_free(struct iommu_domain *domain) } EXPORT_SYMBOL_GPL(iommu_domain_free); +struct iommu_domain *iommu_get_domain_for_group(struct iommu_group *group) +{ + return group->domain; +} +EXPORT_SYMBOL_GPL(iommu_get_domain_for_group); + /* * Put the group's domain back to the appropriate core-owned domain - either the * standard kernel-mode DMA configuration or an all-DMA-blocked domain. diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 563d0f104114b..2f6ea73d2b259 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -704,6 +704,12 @@ struct iommu_ops { struct iommu_domain *parent_domain, const struct iommu_user_data *user_data); + /* Per group IOMMU features */ + int (*get_group_qos_params)(struct iommu_group *group, u16 *partition, + u8 *perf_mon_grp); + int (*set_group_qos_params)(struct iommu_group *group, u16 partition, + u8 perf_mon_grp); + const struct iommu_domain_ops *default_domain_ops; struct module *owner; struct iommu_domain *identity_domain; @@ -911,6 +917,7 @@ extern int iommu_attach_device(struct iommu_domain *domain, extern void iommu_detach_device(struct iommu_domain *domain, struct device *dev); extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); +extern struct iommu_domain *iommu_get_domain_for_group(struct iommu_group *group); struct iommu_domain *iommu_driver_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, From 9f1da562380845c1175ecf8bd69ea998f07906e1 Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 31 Jul 2023 12:07:30 +0100 Subject: [PATCH 345/464] NVIDIA: VR: SAUCE: iommu: Add helpers to get and set the QoS state BugLink: https://bugs.launchpad.net/bugs/2154527 To allow an iommu_group to be moved between resctrl groups as if it were a CPU thread, the mpam driver needs to be able to set the partid and pmg for the iommu_group. Add helpers that call the iommu driver's get/set methods for these parameters. Signed-off-by: James Morse (cherry picked from commit 630242d2001b19a0f214de47640202efc3d09260 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `include/linux/iommu.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/iommu.c | 76 +++++++++++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 15 +++++++++ 2 files changed, 91 insertions(+) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e92af77048673..8b56cfdb1e78c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -4221,3 +4221,79 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr) return ret; } #endif /* CONFIG_IRQ_MSI_IOMMU */ + +/* + * iommu_group_set_qos_params() - Set the QoS parameters for a group + * @group: the iommu group. + * @partition: the partition label all traffic from the group should use. + * @perf_mon_grp: the performance label all traffic from the group should use. + * + * Return: 0 on success, or an error. + */ +int iommu_group_set_qos_params(struct iommu_group *group, + u16 partition, u8 perf_mon_grp) +{ + const struct iommu_ops *ops; + struct group_device *device; + int ret; + + mutex_lock(&group->mutex); + device = list_first_entry_or_null(&group->devices, typeof(*device), + list); + if (!device) { + ret = -ENODEV; + goto out_unlock; + } + + ops = dev_iommu_ops(device->dev); + if (!ops->set_group_qos_params) { + ret = -EOPNOTSUPP; + goto out_unlock; + } + + ret = ops->set_group_qos_params(group, partition, perf_mon_grp); + +out_unlock: + mutex_unlock(&group->mutex); + + return ret; +} +EXPORT_SYMBOL_NS_GPL(iommu_group_set_qos_params, "IOMMUFD_INTERNAL"); + +/* + * iommu_group_get_qos_params() - Get the QoS parameters for a group + * @group: the iommu group. + * @partition: the partition label all traffic from the group uses. + * @perf_mon_grp: the performance label all traffic from the group uses. + * + * Return: 0 on success, or an error. + */ +int iommu_group_get_qos_params(struct iommu_group *group, + u16 *partition, u8 *perf_mon_grp) +{ + const struct iommu_ops *ops; + struct group_device *device; + int ret; + + mutex_lock(&group->mutex); + device = list_first_entry_or_null(&group->devices, typeof(*device), + list); + if (!device) { + ret = -ENODEV; + goto out_unlock; + } + + ops = dev_iommu_ops(device->dev); + if (!ops->get_group_qos_params) { + ret = -EOPNOTSUPP; + goto out_unlock; + } + + ret = ops->get_group_qos_params(group, partition, perf_mon_grp); + +out_unlock: + mutex_unlock(&group->mutex); + + return ret; +} +EXPORT_SYMBOL_NS_GPL(iommu_group_get_qos_params, "IOMMUFD_INTERNAL"); diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 2f6ea73d2b259..ae07ee459ff31 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -1197,6 +1197,10 @@ void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid); ioasid_t iommu_alloc_global_pasid(struct device *dev); void iommu_free_global_pasid(ioasid_t pasid); +int iommu_group_set_qos_params(struct iommu_group *group, + u16 partition, u8 perf_mon_grp); +int iommu_group_get_qos_params(struct iommu_group *group, + u16 *partition, u8 *perf_mon_grp); /* PCI device reset functions */ int pci_dev_reset_iommu_prepare(struct pci_dev *pdev); @@ -1524,6 +1528,17 @@ static inline ioasid_t iommu_alloc_global_pasid(struct device *dev) } static inline void iommu_free_global_pasid(ioasid_t pasid) {} +static inline int iommu_group_set_qos_params(struct iommu_group *group, + u16 partition, u8 perf_mon_grp) +{ + return -ENODEV; +} + +static inline int iommu_group_get_qos_params(struct iommu_group *group, + u16 *partition, u8 *perf_mon_grp) +{ + return -ENODEV; +} static inline int pci_dev_reset_iommu_prepare(struct pci_dev *pdev) { From ffbe57694fc03534ee0f11b1092a783107238d74 Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 16 Sep 2021 16:19:43 +0100 Subject: [PATCH 346/464] NVIDIA: VR: SAUCE: iommu: Add helpers to retrieve iommu_groups by id or kobject BugLink: https://bugs.launchpad.net/bugs/2154527 ARM SMMU with MPAM support are able to mark streams of traffic with the QoS labels MPAM uses. The user-space interface for MPAM is the resctrl filesystem, which allows threads to be moved between groups, its natural to do the same for iommu_groups. The resctrl interface lists threads, so will also need to list iommu_groups, it will be necessary to walk the list of iommu_groups. To ensure this matches what user-space sees via sysfs, it is best to walk the kobjects. When making a change, resctrl will only have the id of a group. To avoid walking the list of kobjects in this case, add iommu_group_get_by_id(). Signed-off-by: James Morse (cherry picked from commit 9b7dcc8fab78bf2545b02c53add7af27c21e5e90 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/iommu.c | 34 ++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 12 ++++++++++++ 2 files changed, 46 insertions(+) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 8b56cfdb1e78c..2f1f795e446ec 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1131,6 +1131,40 @@ struct iommu_group *iommu_group_alloc(void) } EXPORT_SYMBOL_GPL(iommu_group_alloc); +struct iommu_group *iommu_group_get_from_kobj(struct kobject *group_kobj) +{ + struct iommu_group *group; + + if (!iommu_group_kset || !group_kobj) + return NULL; + + group = container_of(group_kobj, struct iommu_group, kobj); + + kobject_get(group->devices_kobj); + kobject_put(&group->kobj); + + return group; +} + +struct iommu_group *iommu_group_get_by_id(int id) +{ + struct kobject *group_kobj; + const char *name; + + if (!iommu_group_kset) + return NULL; + + name = kasprintf(GFP_KERNEL, "%d", id); + if (!name) + return NULL; + + group_kobj = kset_find_obj(iommu_group_kset, name); + kfree(name); + + return iommu_group_get_from_kobj(group_kobj); +} +EXPORT_SYMBOL_GPL(iommu_group_get_by_id); + /** * iommu_group_get_iommudata - retrieve iommu_data registered for a group * @group: the group diff --git a/include/linux/iommu.h b/include/linux/iommu.h index ae07ee459ff31..03a03773b2d72 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -911,6 +911,8 @@ static inline struct iommu_domain *iommu_paging_domain_alloc(struct device *dev) { return iommu_paging_domain_alloc_flags(dev, 0); } +struct iommu_group *iommu_group_get_from_kobj(struct kobject *group_kobj); +extern struct iommu_group *iommu_group_get_by_id(int id); extern void iommu_domain_free(struct iommu_domain *domain); extern int iommu_attach_device(struct iommu_domain *domain, struct device *dev); @@ -1232,6 +1234,16 @@ static inline struct iommu_domain *iommu_paging_domain_alloc(struct device *dev) return ERR_PTR(-ENODEV); } +static inline struct iommu_group *iommu_group_get_from_kobj(struct kobject *group_kobj) +{ + return NULL; +} + +static inline struct iommu_group *iommu_group_get_by_id(int id) +{ + return NULL; +} + static inline void iommu_domain_free(struct iommu_domain *domain) { } From 6b474c18137aa00686596d5f10262fda72a06daf Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 31 Jul 2023 13:10:25 +0100 Subject: [PATCH 347/464] NVIDIA: VR: SAUCE: iommu: Add helper to retrieve iommu kset BugLink: https://bugs.launchpad.net/bugs/2154527 To walk the list of iommu groups visible in sysfs, resctrl needs access to iommu_group_kset. Expose it. Signed-off-by: James Morse (cherry picked from commit 99cc3d17db3cbf3038957b25429a98e2a6dd5a58 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/iommu.c | 5 +++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 11 insertions(+) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 2f1f795e446ec..817d78c8568cb 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1165,6 +1165,11 @@ struct iommu_group *iommu_group_get_by_id(int id) } EXPORT_SYMBOL_GPL(iommu_group_get_by_id); +struct kset *iommu_get_group_kset(void) +{ + return kset_get(iommu_group_kset); +} + /** * iommu_group_get_iommudata - retrieve iommu_data registered for a group * @group: the group diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 03a03773b2d72..a9f2a03439495 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -971,6 +971,7 @@ extern struct iommu_group *iommu_group_ref_get(struct iommu_group *group); extern void iommu_group_put(struct iommu_group *group); extern int iommu_group_id(struct iommu_group *group); +struct kset *iommu_get_group_kset(void); extern struct iommu_domain *iommu_group_default_domain(struct iommu_group *); int iommu_set_pgtable_quirks(struct iommu_domain *domain, @@ -1402,6 +1403,11 @@ static inline int iommu_group_id(struct iommu_group *group) return -ENODEV; } +static inline struct kset *iommu_get_group_kset(void) +{ + return NULL; +} + static inline int iommu_set_pgtable_quirks(struct iommu_domain *domain, unsigned long quirks) { From f386e1cd92c5d95a4e0beccbb67eac724db2809f Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 16 Sep 2021 16:45:41 +0100 Subject: [PATCH 348/464] NVIDIA: VR: SAUCE: kobject: Add kset_get_next_obj() to allow a kset to be walked BugLink: https://bugs.launchpad.net/bugs/2154527 To expose iommu_groups via the resctrl filesystem, the resctrl driver needs to be able to walk the list of iommu_groups. These are exposed via sysfs as a kset. Add kset_get_next_obj() to allow resctrl to walk the kobjects in the kset. Signed-off-by: James Morse (cherry picked from commit 10d03a8e2abf6eb69227b8674463d8a70ceb9c94 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/kobject.h | 2 ++ lib/kobject.c | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/linux/kobject.h b/include/linux/kobject.h index bcb5d4e320015..89d35cbc36a54 100644 --- a/include/linux/kobject.h +++ b/include/linux/kobject.h @@ -200,6 +200,8 @@ static inline const struct kobj_type *get_ktype(const struct kobject *kobj) struct kobject *kset_find_obj(struct kset *, const char *); +struct kobject *kset_get_next_obj(struct kset *kset, struct kobject *prev); + /* The global /sys/kernel/ kobject for people to chain off of */ extern struct kobject *kernel_kobj; /* The global /sys/kernel/mm/ kobject for people to chain off of */ diff --git a/lib/kobject.c b/lib/kobject.c index 9c9ff0f5175fb..518d95cce9755 100644 --- a/lib/kobject.c +++ b/lib/kobject.c @@ -920,6 +920,27 @@ struct kobject *kset_find_obj(struct kset *kset, const char *name) } EXPORT_SYMBOL_GPL(kset_find_obj); +struct kobject *kset_get_next_obj(struct kset *kset, struct kobject *prev) +{ + struct kobject *k; + + spin_lock(&kset->list_lock); + + if (!prev) + k = list_first_entry_or_null(&kset->list, typeof(*k), entry); + else + k = list_next_entry(prev, entry); + + if (list_entry_is_head(k, &kset->list, entry)) + k = NULL; + + kobject_get(k); + spin_unlock(&kset->list_lock); + kobject_put(prev); + + return k; +} + static void kset_release(struct kobject *kobj) { struct kset *kset = container_of(kobj, struct kset, kobj); From e0cc0337c078fe0230d826129c87ae2cdf801723 Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 31 Jul 2023 09:51:04 +0100 Subject: [PATCH 349/464] NVIDIA: VR: SAUCE: arm_mpam: resctrl: Add iommu helpers to get/set the partid and pmg BugLink: https://bugs.launchpad.net/bugs/2154527 SMMU that support MPAM can be configured to use a particular partid and pmg for a stream. The assignment of an iommu_group and its corresponding streams should be done via resctrl. Add helpers similar to setting a closid/rmid on a task. We need the same shifting if the CPUs are using CDP. The SMMU only takes one partid, conceptually its always making data accesses. Signed-off-by: James Morse (cherry picked from commit af9d3e292738a626d784e463509344b3dde55880 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/Kconfig | 1 + drivers/resctrl/mpam_resctrl.c | 53 ++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/drivers/resctrl/Kconfig b/drivers/resctrl/Kconfig index 672abea3b03cc..30f455dba5aa0 100644 --- a/drivers/resctrl/Kconfig +++ b/drivers/resctrl/Kconfig @@ -29,3 +29,4 @@ config ARM64_MPAM_RESCTRL_FS default y if ARM64_MPAM_DRIVER && RESCTRL_FS select RESCTRL_RMID_DEPENDS_ON_CLOSID select RESCTRL_ASSIGN_FIXED + select RESCTRL_IOMMU if ARM_SMMU_V3 diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index aa260a1e11869..4e6fb59fe397e 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -367,6 +368,58 @@ bool resctrl_arch_match_rmid(struct task_struct *tsk, u32 closid, u32 rmid) return (tsk_closid == closid) && (tsk_rmid == rmid); } +int resctrl_arch_set_iommu_closid_rmid(struct iommu_group *group, u32 closid, + u32 rmid) +{ + u16 partid; + + if (!IS_ENABLED(CONFIG_RESCTRL_IOMMU)) + return 0; + + if (cdp_enabled) + partid = closid << 1; + else + partid = closid; + + return iommu_group_set_qos_params(group, partid, rmid); +} + +bool resctrl_arch_match_iommu_closid(struct iommu_group *group, u32 closid) +{ + u16 partid; + int err = iommu_group_get_qos_params(group, &partid, NULL); + + if (!IS_ENABLED(CONFIG_RESCTRL_IOMMU)) + return false; + + if (err) + return false; + + if (cdp_enabled) + partid >>= 1; + + return (partid == closid); +} + +bool resctrl_arch_match_iommu_closid_rmid(struct iommu_group *group, + u32 closid, u32 rmid) +{ + u8 pmg; + u16 partid; + int err = iommu_group_get_qos_params(group, &partid, &pmg); + + if (!IS_ENABLED(CONFIG_RESCTRL_IOMMU)) + return false; + + if (err) + return false; + + if (cdp_enabled) + partid >>= 1; + + return (partid == closid) && (rmid == pmg); +} + struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l) { if (l >= RDT_NUM_RESOURCES) From 69037f2738916e016976f64d995ffc9d33039834 Mon Sep 17 00:00:00 2001 From: James Morse Date: Thu, 16 Sep 2021 17:11:58 +0100 Subject: [PATCH 350/464] NVIDIA: VR: SAUCE: fs/resctrl: Add support for assigning iommu_groups to resctrl groups BugLink: https://bugs.launchpad.net/bugs/2154527 Arm's MPAM has support for assigning devices behind an IOMMU to a control or monitor group. This can be used for device-passthrough for a VM, or user-space drivers using VFIO to ensure the device is either in the same control group as the CPU threads. Alternatively, the iommu_group may be assigned to a different control group with preferential schema values. Extend the resctrl tasks file to include iommu_groups. These appear as 'iommu_group:0', where 0 is the group number that can be found from /sys/kernel/iommu_groups/. iommu_groups can be moved between resctrl groups by writing this string in the same way as tasks are moved. No state is preserved by resctrl, an iommu_group that disappears will no longer be listed as being part of a resctrl group. A new iommu_group will appear in the default group. Add helpers to list and move iommu_groups. Architecture specific helpers are used to apply the closid/rmid to the iommu_group due to the way MPAM emulates CDP. Tested-by: Amit Singh Tomar Signed-off-by: James Morse (forward ported from commit 8a09f730ab48859282f518615f993df7f9ccba2a https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/acpi/arm64/mpam.c`; - Resolve minor conflicts in `fs/resctrl/rdtgroup.c`; - Fix guid_equal(&spec_uuid, &int_tbl_uuid) checking issue; - FIx iommu_group_put() issue in show_rdt_iommu(); ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/acpi/arm64/mpam.c | 91 +++++++++++++++++++++++++++++++-- fs/resctrl/Kconfig | 6 +++ fs/resctrl/rdtgroup.c | 104 +++++++++++++++++++++++++++++++++++++- include/linux/resctrl.h | 28 ++++++++++ 4 files changed, 225 insertions(+), 4 deletions(-) diff --git a/drivers/acpi/arm64/mpam.c b/drivers/acpi/arm64/mpam.c index 84963a20c3e78..0bd9ff1fd9781 100644 --- a/drivers/acpi/arm64/mpam.c +++ b/drivers/acpi/arm64/mpam.c @@ -95,17 +95,51 @@ static void acpi_mpam_parse_irqs(struct platform_device *pdev, res[(*res_idx)++] = DEFINE_RES_IRQ_NAMED(irq, "error"); } -static int acpi_mpam_parse_resource(struct mpam_msc *msc, +#define UUID_MPAM_INTERCONNECT_TABLE "fe2bd645-033b-49e6-9479-2e0b8b21d1cd" + +struct acpi_mpam_interconnect_descriptor_table { + u8 type_uuid[16]; + u32 num_descriptors; +}; + +struct acpi_mpam_interconnect_descriptor { + u32 source_id; + u32 destination_id; + u8 link_type; + u8 reserved[3]; +}; + +static int acpi_mpam_parse_resource(struct acpi_mpam_msc_node *tbl_msc, + struct mpam_msc *msc, struct acpi_mpam_resource_node *res) { + struct acpi_mpam_interconnect_descriptor_table *tbl_int_tbl; + struct acpi_mpam_interconnect_descriptor *tbl_int; + guid_t int_tbl_uuid, spec_uuid; int level, nid; u32 cache_id; + off_t offset; + /* + * Class IDs are somewhat arbitrary, but need to be co-ordinated. + * 0-N are caches, + * 64, 65: Interconnect, but ideally these would appear between the + * classes the controls are adjacent to. + * 128: SMMU, + * 192-192+level: Memory Side Caches, nothing checks that N is a + * small number. + * 255: Memory Controllers + * + * ACPI devices would need a class id allocated based on the _HID. + * + * Classes that the mpam driver can't currently plumb into resctrl + * are registered as UNKNOWN. + */ switch (res->locator_type) { case ACPI_MPAM_LOCATION_TYPE_PROCESSOR_CACHE: cache_id = res->locator.cache_locator.cache_reference; level = find_acpi_cache_level_from_id(cache_id); - if (level <= 0) { + if (level <= 0 || level >= 64) { pr_err_once("Bad level (%d) for cache with id %u\n", level, cache_id); return -EINVAL; } @@ -120,6 +154,57 @@ static int acpi_mpam_parse_resource(struct mpam_msc *msc, } return mpam_ris_create(msc, res->ris_index, MPAM_CLASS_MEMORY, MPAM_CLASS_ID_DEFAULT, nid); + case ACPI_MPAM_LOCATION_TYPE_SMMU: + return mpam_ris_create(msc, res->ris_index, MPAM_CLASS_UNKNOWN, + 128, res->locator.smmu_locator.smmu_interface); + case ACPI_MPAM_LOCATION_TYPE_MEMORY_CACHE: + cache_id = res->locator.mem_cache_locator.reference; + level = res->locator.mem_cache_locator.level; + if (192 + level >= 255) { + pr_err_once("Bad level (%u) for memory side cache with reference %u\n", + level, cache_id); + return -EINVAL; + } + + return mpam_ris_create(msc, res->ris_index, MPAM_CLASS_CACHE, + 192 + level, cache_id); + + case ACPI_MPAM_LOCATION_TYPE_INTERCONNECT: + /* Find the descriptor table, and check it lands in the parent msc */ + offset = res->locator.interconnect_ifc_locator.inter_connect_desc_tbl_off; + if (offset >= tbl_msc->length) { + pr_err_once("Bad offset (%lu) for interconnect descriptor on msc %u\n", + offset, tbl_msc->identifier); + return -EINVAL; + } + tbl_int_tbl = ACPI_ADD_PTR(struct acpi_mpam_interconnect_descriptor_table, + tbl_msc, offset); + guid_parse(UUID_MPAM_INTERCONNECT_TABLE, &spec_uuid); + import_guid(&int_tbl_uuid, tbl_int_tbl->type_uuid); + if (!guid_equal(&spec_uuid, &int_tbl_uuid)) { + pr_err_once("Bad UUID for interconnect descriptor on msc %u\n", + tbl_msc->identifier); + return -EINVAL; + } + + offset += sizeof(*tbl_int_tbl); + offset += tbl_int_tbl->num_descriptors * sizeof(*tbl_int); + if (offset >= tbl_msc->length) { + pr_err_once("Bad num_descriptors (%u) for interconnect descriptor on msc %u\n", + tbl_int_tbl->num_descriptors, tbl_msc->identifier); + return -EINVAL; + } + + tbl_int = ACPI_ADD_PTR(struct acpi_mpam_interconnect_descriptor, + tbl_int_tbl, sizeof(*tbl_int_tbl)); + cache_id = tbl_int->source_id; + + /* Unknown link type? */ + if (tbl_int->link_type != 0 && tbl_int->link_type == 1) + return 0; + + return mpam_ris_create(msc, res->ris_index, MPAM_CLASS_UNKNOWN, + 64 + tbl_int->link_type, cache_id); default: /* These get discovered later and are treated as unknown */ return 0; @@ -150,7 +235,7 @@ int acpi_mpam_parse_resources(struct mpam_msc *msc, return -EINVAL; } - err = acpi_mpam_parse_resource(msc, resource); + err = acpi_mpam_parse_resource(tbl_msc, msc, resource); if (err) return err; diff --git a/fs/resctrl/Kconfig b/fs/resctrl/Kconfig index 21671301bd8a4..145d837c190a3 100644 --- a/fs/resctrl/Kconfig +++ b/fs/resctrl/Kconfig @@ -37,3 +37,9 @@ config RESCTRL_RMID_DEPENDS_ON_CLOSID Enabled by the architecture when the RMID values depend on the CLOSID. This causes the CLOSID allocator to search for CLOSID with clean RMID. + +config RESCTRL_IOMMU + bool + help + Enabled by the architecture when some IOMMU are able to be configured + with CLOSID/RMID. diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 52dd273046722..086bba229400a 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -769,10 +770,65 @@ static int rdtgroup_move_task(pid_t pid, struct rdtgroup *rdtgrp, return ret; } +static int rdtgroup_move_iommu(int iommu_group_id, struct rdtgroup *rdtgrp, + struct kernfs_open_file *of) +{ + const struct cred *cred = current_cred(); + struct iommu_group *iommu_group; + int err; + + if (!uid_eq(cred->euid, GLOBAL_ROOT_UID)) { + rdt_last_cmd_printf("No permission to move iommu_group %d\n", + iommu_group_id); + return -EPERM; + } + + iommu_group = iommu_group_get_by_id(iommu_group_id); + if (!iommu_group) { + rdt_last_cmd_printf("No matching iommu_group %d\n", + iommu_group_id); + return -ESRCH; + } + + if (rdtgrp->type == RDTMON_GROUP && + !resctrl_arch_match_iommu_closid(iommu_group, + rdtgrp->mon.parent->closid)) { + rdt_last_cmd_puts("Can't move iommu_group to different control group\n"); + err = -EINVAL; + } else { + err = resctrl_arch_set_iommu_closid_rmid(iommu_group, + rdtgrp->closid, + rdtgrp->mon.rmid); + } + + iommu_group_put(iommu_group); + + return err; +} + +static bool string_is_iommu_group(char *buf, int *val) +{ + if (!IS_ENABLED(CONFIG_RESCTRL_IOMMU) || + !static_branch_unlikely(&resctrl_abi_playground)) + return false; + + if (strlen(buf) <= strlen("iommu_group:")) + return false; + + if (strncmp(buf, "iommu_group:", strlen("iommu_group:"))) + return false; + + buf += strlen("iommu_group:"); + + return !kstrtoint(buf, 0, val); +} + static ssize_t rdtgroup_tasks_write(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { struct rdtgroup *rdtgrp; + int iommu_group_id; + bool is_iommu; char *pid_str; int ret = 0; pid_t pid; @@ -794,7 +850,13 @@ static ssize_t rdtgroup_tasks_write(struct kernfs_open_file *of, while (buf && buf[0] != '\0' && buf[0] != '\n') { pid_str = strim(strsep(&buf, ",")); - if (kstrtoint(pid_str, 0, &pid)) { + is_iommu = string_is_iommu_group(pid_str, &iommu_group_id); + if (is_iommu) { + ret = rdtgroup_move_iommu(iommu_group_id, rdtgrp, of); + if (ret) + break; + continue; + } else if (kstrtoint(pid_str, 0, &pid)) { rdt_last_cmd_printf("Task list parsing error pid %s\n", pid_str); ret = -EINVAL; break; @@ -819,6 +881,44 @@ static ssize_t rdtgroup_tasks_write(struct kernfs_open_file *of, return ret ?: nbytes; } +static bool iommu_matches_rdtgroup(struct iommu_group *group, struct rdtgroup *r) +{ + if (r->type == RDTCTRL_GROUP) + return resctrl_arch_match_iommu_closid(group, r->closid); + + return resctrl_arch_match_iommu_closid_rmid(group, r->closid, + r->mon.rmid); +} + +static void show_rdt_iommu(struct rdtgroup *r, struct seq_file *s) +{ + struct kset *iommu_groups; + struct iommu_group *group; + struct kobject *group_kobj = NULL; + + if (!IS_ENABLED(CONFIG_RESCTRL_IOMMU) || + !static_branch_unlikely(&resctrl_abi_playground)) + return; + + iommu_groups = iommu_get_group_kset(); + + while ((group_kobj = kset_get_next_obj(iommu_groups, group_kobj))) { + /* iommu_group_get_from_kobj() wants to drop a reference */ + kobject_get(group_kobj); + + group = iommu_group_get_from_kobj(group_kobj); + if (!group) + continue; + + if (iommu_matches_rdtgroup(group, r)) + seq_printf(s, "iommu_group:%s\n", group_kobj->name); + + iommu_group_put(group); + } + + kset_put(iommu_groups); +} + static void show_rdt_tasks(struct rdtgroup *r, struct seq_file *s) { struct task_struct *p, *t; @@ -833,6 +933,8 @@ static void show_rdt_tasks(struct rdtgroup *r, struct seq_file *s) } } rcu_read_unlock(); + + show_rdt_iommu(r, s); } static int rdtgroup_tasks_show(struct kernfs_open_file *of, diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index b236c4e9cb619..3ada7ed9f8bec 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -3,6 +3,7 @@ #define _RESCTRL_H #include +#include #include #include #include @@ -707,6 +708,7 @@ extern unsigned int resctrl_rmid_realloc_limit; int resctrl_init(void); void resctrl_exit(void); + #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK u64 resctrl_arch_get_prefetch_disable_bits(void); int resctrl_arch_pseudo_lock_fn(void *_plr); @@ -720,4 +722,30 @@ static inline int resctrl_arch_measure_cycles_lat_fn(void *_plr) { return 0; } static inline int resctrl_arch_measure_l2_residency(void *_plr) { return 0; } static inline int resctrl_arch_measure_l3_residency(void *_plr) { return 0; } #endif /* CONFIG_RESCTRL_FS_PSEUDO_LOCK */ + +/* When supported, the architecture must implement these */ +#ifdef CONFIG_RESCTRL_IOMMU +int resctrl_arch_set_iommu_closid_rmid(struct iommu_group *group, u32 closid, + u32 rmid); +bool resctrl_arch_match_iommu_closid(struct iommu_group *group, u32 closid); +bool resctrl_arch_match_iommu_closid_rmid(struct iommu_group *group, u32 closid, + u32 rmid); +#else +static inline int resctrl_arch_set_iommu_closid_rmid(struct iommu_group *group, + u32 closid, u32 rmid) +{ + return -EOPNOTSUPP; +} +static inline bool resctrl_arch_match_iommu_closid(struct iommu_group *group, + u32 closid) +{ + return false; +} +static inline bool +resctrl_arch_match_iommu_closid_rmid(struct iommu_group *group, + u32 closid, u32 rmid) +{ + return false; +} +#endif /* CONFIG_RESCTRL_IOMMU */ #endif /* _RESCTRL_H */ From 23d6b264abd5157534c94e8020f0a66f59342ff0 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 15 Mar 2024 16:46:12 +0000 Subject: [PATCH 351/464] NVIDIA: SAUCE: x86/resctrl: Add stub to allow other architecture to disable monitor overflow BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl has an overflow handler that runs on each domain every second to ensure that any overflow of the hardware counter is accounted for. MPAM can have counters as large as 63 bits, in which case there is no need to check for overflow. To allow other architectures to disable this, add a helper that reports whether counters can overflow. Signed-off-by: James Morse (cherry picked from commit 6a4360b3e0339ffc510b68d7a7d22941030f0604 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/x86/include/asm/resctrl.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/include/asm/resctrl.h b/arch/x86/include/asm/resctrl.h index 575f8408a9e7c..40a74a0617345 100644 --- a/arch/x86/include/asm/resctrl.h +++ b/arch/x86/include/asm/resctrl.h @@ -191,6 +191,11 @@ static inline void resctrl_arch_mon_ctx_free(struct rdt_resource *r, enum resctrl_event_id evtid, void *ctx) { } +static inline bool resctrl_arch_mon_can_overflow(void) +{ + return true; +} + void resctrl_cpu_detect(struct cpuinfo_x86 *c); #else From 280d1935568525dfb937d9677ee9423daebee762 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 15 Mar 2024 17:32:53 +0000 Subject: [PATCH 352/464] NVIDIA: SAUCE: arm_mpam: resctrl: Determine if any exposed counter can overflow BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl has an overflow handler that runs on each domain every second to ensure that any overflow of the hardware counter is accounted for. MPAM can have counters as large as 63 bits, in which case there is no need to check for overflow. To allow the overflow handler to be disabled, determine if an overflow can happen. If a class is not implemented, or has the 63bit counter, it can't overflow. Signed-off-by: James Morse (cherry picked from commit 0f6aefdf5164dd6be3bd8c6cd82b6257fadbeab2 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 21 +++++++++++++++++++++ include/linux/arm_mpam.h | 1 + 2 files changed, 22 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 4e6fb59fe397e..ec602f6fd6e18 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -535,6 +535,27 @@ static int update_rmid_limits(struct mpam_class *class) return 0; } +static bool __resctrl_arch_mon_can_overflow(enum resctrl_event_id eventid) +{ + struct mpam_props *cprops; + struct mpam_class *class = mpam_resctrl_counters[eventid].class; + + if (!class) + return false; + + /* No need to worry about a 63 bit counter overflowing */ + cprops = &class->props; + return !mpam_has_feature(mpam_feat_msmon_mbwu_63counter, cprops); +} + +bool resctrl_arch_mon_can_overflow(void) +{ + if (__resctrl_arch_mon_can_overflow(QOS_L3_MBM_LOCAL_EVENT_ID)) + return true; + + return __resctrl_arch_mon_can_overflow(QOS_L3_MBM_TOTAL_EVENT_ID); +} + static int __read_mon(struct mpam_resctrl_mon *mon, struct mpam_component *mon_comp, enum mpam_device_features mon_type, diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 4ccf32fe07fd5..b066d57e1a085 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -53,6 +53,7 @@ static inline int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, bool resctrl_arch_alloc_capable(void); bool resctrl_arch_mon_capable(void); +bool resctrl_arch_mon_can_overflow(void); void resctrl_arch_set_cpu_default_closid(int cpu, u32 closid); void resctrl_arch_set_closid_rmid(struct task_struct *tsk, u32 closid, u32 rmid); From 4fb6200d20a9e8231afc7946cdd34a4da26a228f Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 15 Mar 2024 17:36:02 +0000 Subject: [PATCH 353/464] NVIDIA: SAUCE: fs/restrl: Allow the overflow handler to be disabled BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl has an overflow handler that runs on each domain every second to ensure that any overflow of the hardware counter is accounted for. MPAM can have counters as large as 63 bits, in which case there is no need to check for overflow. Call the new arch helpers to determine this. Signed-off-by: James Morse (cherry picked from commit 72e375a4611a0eb5355e5a171a67a419ffd53522 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/monitor.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c index e338b8d484054..94da0360952d5 100644 --- a/fs/resctrl/monitor.c +++ b/fs/resctrl/monitor.c @@ -918,8 +918,10 @@ void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long del /* * When a domain comes online there is no guarantee the filesystem is * mounted. If not, there is no need to catch counter overflow. + * Some architecture may have ~64bit counters, and can ignore overflow. */ - if (!resctrl_mounted || !resctrl_arch_mon_capable()) + if (!resctrl_mounted || !resctrl_arch_mon_capable() || + !resctrl_arch_mon_can_overflow()) return; cpu = cpumask_any_housekeeping(&dom->hdr.cpu_mask, exclude_cpu); dom->mbm_work_cpu = cpu; From 4c116e3b2cd8af23272a4dc9ad98b9699e486b2c Mon Sep 17 00:00:00 2001 From: Rex Nie Date: Mon, 11 Mar 2024 16:18:39 +0800 Subject: [PATCH 354/464] NVIDIA: VR: SAUCE: fs/resctrl: Uniform data type of component_id/domid/id/cache_id BugLink: https://bugs.launchpad.net/bugs/2154527 This patch uniform data type of component_id/domid/id/cache_id to u32 to avoid type confusion. According to ACPI for mpam, cache id is used as locator for cache MSC. Reference to RD_PPTT_CACHE_ID definition from edk2-platforms, u32 is enough for cache_id. ( \ (((PackageId) & 0xF) << 20) | (((ClusterId) & 0xFF) << 12) | \ (((CoreId) & 0xFF) << 4) | ((CacheType) & 0xF) \ ) refs: 1. ACPI for mpam: https://developer.arm.com/documentation/den0065/latest/ 2. RD_PPTT_CACHE_ID from edk2-platforms: https://github.com/tianocore/edk2-platforms/blob/master/Platform/ARM/SgiPkg/Include/SgiAcpiHeader.h#L202 Signed-off-by: Rex Nie Signed-off-by: James Morse (cherry picked from commit 6941241fa2fd78befa42cd442507157701c98878 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `include/linux/arm_mpam.h`; - Resolve minor conflicts in `include/linux/resctrl.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 8 ++++---- include/linux/arm_mpam.h | 4 ++-- include/linux/resctrl.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 2e9b08eabb9b1..d82e9616e7013 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -371,7 +371,7 @@ mpam_class_find(u8 level_idx, enum mpam_class_types type) } static struct mpam_component * -mpam_component_alloc(struct mpam_class *class, int id) +mpam_component_alloc(struct mpam_class *class, u32 id) { struct mpam_component *comp; @@ -413,7 +413,7 @@ static void mpam_component_destroy(struct mpam_component *comp) } static struct mpam_component * -mpam_component_find(struct mpam_class *class, int id) +mpam_component_find(struct mpam_class *class, u32 id) { struct mpam_component *comp; @@ -539,7 +539,7 @@ static int mpam_ris_get_affinity(struct mpam_msc *msc, cpumask_t *affinity, static int mpam_ris_create_locked(struct mpam_msc *msc, u8 ris_idx, enum mpam_class_types type, u8 class_id, - int component_id) + u32 component_id) { int err; struct mpam_vmsc *vmsc; @@ -627,7 +627,7 @@ static void mpam_ris_destroy(struct mpam_msc_ris *ris) } int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, - enum mpam_class_types type, u8 class_id, int component_id) + enum mpam_class_types type, u8 class_id, u32 component_id) { int err; diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index b066d57e1a085..0891f85383674 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -41,11 +41,11 @@ static inline int acpi_mpam_count_msc(void) { return -EINVAL; } #ifdef CONFIG_ARM64_MPAM_DRIVER int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, - enum mpam_class_types type, u8 class_id, int component_id); + enum mpam_class_types type, u8 class_id, u32 component_id); #else static inline int mpam_ris_create(struct mpam_msc *msc, u8 ris_idx, enum mpam_class_types type, u8 class_id, - int component_id) + u32 component_id) { return -EINVAL; } diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 3ada7ed9f8bec..c4448cec07e41 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -138,7 +138,7 @@ enum resctrl_domain_type { */ struct rdt_domain_hdr { struct list_head list; - int id; + u32 id; enum resctrl_domain_type type; enum resctrl_res_level rid; struct cpumask cpu_mask; From 5c72c13ccca3919293d846dc6087e7105e1c7ca5 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 27 Aug 2024 15:24:08 +0100 Subject: [PATCH 355/464] NVIDIA: SAUCE: arm_mpam: Allow cmax/cmin to be configured BugLink: https://bugs.launchpad.net/bugs/2154527 mpam_reprogram_ris_partid() always resets the CMAX/CMIN controls to their 'unrestricted' value. This prevents the controls from being configured. Add fields in struct mpam_config, and program these values when they are set in the features bitmask. Signed-off-by: James Morse (cherry picked from commit e701b2860ae2c02dc9c2015846d61838904a5b0b https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; - Resolve minor conflicts in `drivers/resctrl/mpam_internal.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 23 +++++++++++++++++++---- drivers/resctrl/mpam_internal.h | 4 ++++ 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index d82e9616e7013..0539102b68b62 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -1623,11 +1623,25 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, if (mpam_has_feature(mpam_feat_mbw_prop, rprops)) mpam_write_partsel_reg(msc, MBW_PROP, 0); - if (mpam_has_feature(mpam_feat_cmax_cmax, rprops)) - mpam_write_partsel_reg(msc, CMAX, cmax); + if (mpam_has_feature(mpam_feat_cmax_cmax, rprops)) { + if (mpam_has_feature(mpam_feat_cmax_cmax, cfg)) { + u32 cmax_val = cfg->cmax; - if (mpam_has_feature(mpam_feat_cmax_cmin, rprops)) - mpam_write_partsel_reg(msc, CMIN, 0); + if (cfg->cmax_softlim) + cmax_val |= MPAMCFG_CMAX_SOFTLIM; + mpam_write_partsel_reg(msc, CMAX, cmax_val); + } else { + mpam_write_partsel_reg(msc, CMAX, cmax); + } + } + + if (mpam_has_feature(mpam_feat_cmax_cmin, rprops)) { + if (mpam_has_feature(mpam_feat_cmax_cmin, cfg)) { + mpam_write_partsel_reg(msc, CMIN, cfg->cmin); + } else { + mpam_write_partsel_reg(msc, CMIN, 0); + } + } if (mpam_has_feature(mpam_feat_cmax_cassoc, rprops)) mpam_write_partsel_reg(msc, CASSOC, MPAMCFG_CASSOC_CASSOC); @@ -3068,6 +3082,7 @@ static bool mpam_update_config(struct mpam_config *cfg, bool has_changes = false; maybe_update_config(cfg, mpam_feat_cpor_part, newcfg, cpbm, has_changes); + maybe_update_config(cfg, mpam_feat_cmax_cmax, newcfg, cmax, has_changes); maybe_update_config(cfg, mpam_feat_mbw_part, newcfg, mbw_pbm, has_changes); maybe_update_config(cfg, mpam_feat_mbw_max, newcfg, mbw_max, has_changes); diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index 1ec5e7072c0fd..a08901adc4e09 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -353,6 +353,10 @@ struct mpam_config { u32 cpbm; u32 mbw_pbm; u16 mbw_max; + u16 cmax; + u16 cmin; + + bool cmax_softlim; struct mpam_garbage garbage; }; From 9dab96d0410fe0ad2a4e43335f34690a40322b6b Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 19 Nov 2024 11:37:26 +0000 Subject: [PATCH 356/464] NVIDIA: SAUCE: arm_mpam: Rename mbw conversion to 'fract16' for code re-use BugLink: https://bugs.launchpad.net/bugs/2154527 Functions like mbw_max_to_percent() convert a value into MPAMs 16 bit fixed point fraction format. These are not only used for memory bandwidth, but cache capcity controls too. Rename these functions to convert to/from a 'fract16', and add helpers for the specific mbw_max/cmax controls. Signed-off-by: James Morse (cherry picked from commit 738f1605fb5c796713a429214270a18ec9c5d6c3 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; - Resolve minor conflicts in `drivers/resctrl/test_mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 24 +++++++++++++++++------- drivers/resctrl/test_mpam_resctrl.c | 4 ++-- 2 files changed, 19 insertions(+), 9 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index ec602f6fd6e18..42df149769663 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -854,14 +854,14 @@ static u32 get_mba_granularity(struct mpam_props *cprops) * * Find the nearest percentage value to the upper bound of the selected band: */ -static u32 mbw_max_to_percent(u16 mbw_max, struct mpam_props *cprops) +static u32 fract16_to_percent(u16 fract, u8 wd) { - u32 val = mbw_max; + u32 val = fract; - val >>= 16 - cprops->bwa_wd; + val >>= 16 - wd; val += 1; val *= MAX_MBA_BW; - val = DIV_ROUND_CLOSEST(val, 1 << cprops->bwa_wd); + val = DIV_ROUND_CLOSEST(val, 1 << wd); return val; } @@ -876,18 +876,28 @@ static u32 mbw_max_to_percent(u16 mbw_max, struct mpam_props *cprops) * percentages) and over-commit (where the total of the converted * allocations is greater than expected). */ -static u16 percent_to_mbw_max(u8 pc, struct mpam_props *cprops) +static u16 percent_to_fract16(u8 pc, u8 wd) { u32 val = pc; - val <<= cprops->bwa_wd; + val <<= wd; val = DIV_ROUND_CLOSEST(val, MAX_MBA_BW); val = max(val, 1) - 1; - val <<= 16 - cprops->bwa_wd; + val <<= 16 - wd; return val; } +static u32 mbw_max_to_percent(u16 mbw_max, struct mpam_props *cprops) +{ + return fract16_to_percent(mbw_max, cprops->bwa_wd); +} + +static u16 percent_to_mbw_max(u8 pc, struct mpam_props *cprops) +{ + return percent_to_fract16(pc, cprops->bwa_wd); +} + static u32 get_mba_min(struct mpam_props *cprops) { if (!mba_class_use_mbw_max(cprops)) { diff --git a/drivers/resctrl/test_mpam_resctrl.c b/drivers/resctrl/test_mpam_resctrl.c index 4145f057bd31a..2dd28336b3d1d 100644 --- a/drivers/resctrl/test_mpam_resctrl.c +++ b/drivers/resctrl/test_mpam_resctrl.c @@ -133,7 +133,7 @@ static void test_get_mba_granularity(struct kunit *test) KUNIT_EXPECT_EQ(test, ret, 1); /* DIV_ROUND_UP(100, 1 << 16)% = 1% */ } -static void test_mbw_max_to_percent(struct kunit *test) +static void test_fract16_to_percent(struct kunit *test) { const struct percent_value_case *param = test->param_value; struct percent_value_test_info res; @@ -359,7 +359,7 @@ static void test_num_assignable_counters(struct kunit *test) static struct kunit_case mpam_resctrl_test_cases[] = { KUNIT_CASE(test_get_mba_granularity), - KUNIT_CASE_PARAM(test_mbw_max_to_percent, test_percent_value_gen_params), + KUNIT_CASE_PARAM(test_fract16_to_percent, test_percent_value_gen_params), KUNIT_CASE_PARAM(test_percent_to_mbw_max, test_percent_value_gen_params), KUNIT_CASE_PARAM(test_mbw_max_to_percent_limits, test_all_bwa_wd_gen_params), KUNIT_CASE(test_percent_to_max_rounding), From 2c7dadf85283eb9e207972f1f9a610d3c73282d6 Mon Sep 17 00:00:00 2001 From: James Morse Date: Mon, 18 Nov 2024 18:45:50 +0000 Subject: [PATCH 357/464] NVIDIA: SAUCE: fs/resctrl: Group all the MBA specific properties in a separate struct BugLink: https://bugs.launchpad.net/bugs/2154527 struct resctrl_membw combines parameters that are related to the control value, and parameters that are specific to the MBA resource. To allow the control value parsing and management code to be re-used for other resources, it needs to be separated from the MBA resource. Add struct resctrl_mba that holds all the parameters that are specific to the MBA resource. Signed-off-by: James Morse (cherry picked from commit c1133462aa498d8b75e73b094eb91512d982e067 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/x86/kernel/cpu/resctrl/core.c | 18 +++++++++--------- drivers/resctrl/mpam_resctrl.c | 4 ++-- fs/resctrl/ctrlmondata.c | 3 ++- fs/resctrl/rdtgroup.c | 18 +++++++++--------- include/linux/resctrl.h | 26 +++++++++++++++++--------- 5 files changed, 39 insertions(+), 30 deletions(-) diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c index 7667cf7c4e945..ba3316a41141b 100644 --- a/arch/x86/kernel/cpu/resctrl/core.c +++ b/arch/x86/kernel/cpu/resctrl/core.c @@ -212,21 +212,21 @@ static __init bool __get_mem_config_intel(struct rdt_resource *r) hw_res->num_closid = edx.split.cos_max + 1; max_delay = eax.split.max_delay + 1; r->membw.max_bw = MAX_MBA_BW; - r->membw.arch_needs_linear = true; + r->mba.arch_needs_linear = true; if (ecx & MBA_IS_LINEAR) { - r->membw.delay_linear = true; + r->mba.delay_linear = true; r->membw.min_bw = MAX_MBA_BW - max_delay; r->membw.bw_gran = MAX_MBA_BW - max_delay; } else { if (!rdt_get_mb_table(r)) return false; - r->membw.arch_needs_linear = false; + r->mba.arch_needs_linear = false; } if (boot_cpu_has(X86_FEATURE_PER_THREAD_MBA)) - r->membw.throttle_mode = THREAD_THROTTLE_PER_THREAD; + r->mba.throttle_mode = THREAD_THROTTLE_PER_THREAD; else - r->membw.throttle_mode = THREAD_THROTTLE_MAX; + r->mba.throttle_mode = THREAD_THROTTLE_MAX; r->alloc_capable = true; @@ -249,14 +249,14 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) r->membw.max_bw = 1 << eax; /* AMD does not use delay */ - r->membw.delay_linear = false; - r->membw.arch_needs_linear = false; + r->mba.delay_linear = false; + r->mba.arch_needs_linear = false; /* * AMD does not use memory delay throttle model to control * the allocation like Intel does. */ - r->membw.throttle_mode = THREAD_THROTTLE_UNDEFINED; + r->mba.throttle_mode = THREAD_THROTTLE_UNDEFINED; r->membw.min_bw = 0; r->membw.bw_gran = 1; @@ -325,7 +325,7 @@ static void mba_wrmsr_amd(struct msr_param *m) */ static u32 delay_bw_map(unsigned long bw, struct rdt_resource *r) { - if (r->membw.delay_linear) + if (r->mba.delay_linear) return MAX_MBA_BW - bw; pr_warn_once("Non Linear delay-bw map not supported but queried\n"); diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 42df149769663..25acac6f4d456 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1418,8 +1418,8 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) r->schema_fmt = RESCTRL_SCHEMA_RANGE; r->ctrl_scope = RESCTRL_L3_CACHE; - r->membw.delay_linear = true; - r->membw.throttle_mode = THREAD_THROTTLE_UNDEFINED; + r->mba.delay_linear = true; + r->mba.throttle_mode = THREAD_THROTTLE_UNDEFINED; r->membw.min_bw = get_mba_min(cprops); r->membw.max_bw = MAX_MBA_BW; r->membw.bw_gran = get_mba_granularity(cprops); diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 0c02451c687b2..1eac8f7dc07ac 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -48,7 +48,8 @@ static bool bw_validate(char *buf, u32 *data, struct rdt_resource *r) /* * Only linear delay values is supported for current Intel SKUs. */ - if (!r->membw.delay_linear && r->membw.arch_needs_linear) { + if (r->rid == RDT_RESOURCE_MBA && + !r->mba.delay_linear && r->mba.arch_needs_linear) { rdt_last_cmd_puts("No support for non-linear MB domains\n"); return false; } diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 086bba229400a..590a76d0cf990 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1302,7 +1302,7 @@ static int rdt_delay_linear_show(struct kernfs_open_file *of, struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); struct rdt_resource *r = s->res; - seq_printf(seq, "%u\n", r->membw.delay_linear); + seq_printf(seq, "%u\n", r->mba.delay_linear); return 0; } @@ -1320,7 +1320,7 @@ static int rdt_thread_throttle_mode_show(struct kernfs_open_file *of, struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); struct rdt_resource *r = s->res; - switch (r->membw.throttle_mode) { + switch (r->mba.throttle_mode) { case THREAD_THROTTLE_PER_THREAD: seq_puts(seq, "per-thread\n"); return 0; @@ -1657,7 +1657,7 @@ bool is_mba_sc(struct rdt_resource *r) if (r->rid != RDT_RESOURCE_MBA) return false; - return r->membw.mba_sc; + return r->mba.mba_sc; } /* @@ -2279,13 +2279,13 @@ static void thread_throttle_mode_init(void) r_mba = resctrl_arch_get_resource(RDT_RESOURCE_MBA); if (r_mba->alloc_capable && - r_mba->membw.throttle_mode != THREAD_THROTTLE_UNDEFINED) - throttle_mode = r_mba->membw.throttle_mode; + r_mba->mba.throttle_mode != THREAD_THROTTLE_UNDEFINED) + throttle_mode = r_mba->mba.throttle_mode; r_smba = resctrl_arch_get_resource(RDT_RESOURCE_SMBA); if (r_smba->alloc_capable && - r_smba->membw.throttle_mode != THREAD_THROTTLE_UNDEFINED) - throttle_mode = r_smba->membw.throttle_mode; + r_smba->mba.throttle_mode != THREAD_THROTTLE_UNDEFINED) + throttle_mode = r_smba->mba.throttle_mode; if (throttle_mode == THREAD_THROTTLE_UNDEFINED) return; @@ -2587,7 +2587,7 @@ mongroup_create_dir(struct kernfs_node *parent_kn, struct rdtgroup *prgrp, static inline bool is_mba_linear(void) { - return resctrl_arch_get_resource(RDT_RESOURCE_MBA)->membw.delay_linear; + return resctrl_arch_get_resource(RDT_RESOURCE_MBA)->mba.delay_linear; } static int mba_sc_domain_allocate(struct rdt_resource *r, struct rdt_ctrl_domain *d) @@ -2645,7 +2645,7 @@ static int set_mba_sc(bool mba_sc) if (!supports_mba_mbps() || mba_sc == is_mba_sc(r)) return -EINVAL; - r->membw.mba_sc = mba_sc; + r->mba.mba_sc = mba_sc; rdtgroup_default.mba_mbps_event = mba_mbps_default_event; diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index c4448cec07e41..f0a33e4762201 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -248,22 +248,28 @@ enum membw_throttle_mode { * @min_bw: Minimum memory bandwidth percentage user can request * @max_bw: Maximum memory bandwidth value, used as the reset value * @bw_gran: Granularity at which the memory bandwidth is allocated - * @delay_linear: True if memory B/W delay is in linear scale - * @arch_needs_linear: True if we can't configure non-linear resources - * @throttle_mode: Bandwidth throttling mode when threads request - * different memory bandwidths - * @mba_sc: True if MBA software controller(mba_sc) is enabled - * @mb_map: Mapping of memory B/W percentage to memory B/W delay */ struct resctrl_membw { u32 min_bw; u32 max_bw; u32 bw_gran; - u32 delay_linear; - bool arch_needs_linear; - enum membw_throttle_mode throttle_mode; +}; + +/** + * struct resctrl_mba - Resource properties that are specific to the MBA resource + * @mba_sc: True if MBA software controller(mba_sc) is enabled + * @mb_map: Mapping of memory B/W percentage to memory B/W delay + * @delay_linear: True if control is in linear scale + * @arch_needs_linear: True if we can't configure non-linear resources + * @throttle_mode: Mode when threads request different control values + */ +struct resctrl_mba { bool mba_sc; u32 *mb_map; + bool delay_linear; + bool arch_needs_linear; + enum membw_throttle_mode throttle_mode; + }; struct resctrl_schema; @@ -315,6 +321,7 @@ struct resctrl_mon { * @mon: Monitoring related data. * @ctrl_domains: RCU list of all control domains for this resource * @mon_domains: RCU list of all monitor domains for this resource + * @mba: Properties of the MBA resource * @name: Name to use in "schemata" file. * @schema_fmt: Which format string and parser is used for this schema. * @cdp_capable: Is the CDP feature available on this resource @@ -328,6 +335,7 @@ struct rdt_resource { struct resctrl_cache cache; struct resctrl_membw membw; struct resctrl_mon mon; + struct resctrl_mba mba; struct list_head ctrl_domains; struct list_head mon_domains; char *name; From 07cc8ef2540697093c599445fba9167b60470be9 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 10 Sep 2024 11:33:53 +0100 Subject: [PATCH 358/464] NVIDIA: SAUCE: fs/resctrl: Abstract duplicate domain test to a helper BugLink: https://bugs.launchpad.net/bugs/2154527 parse_cbm() and parse_bw() both test the staged config for an existing entry. These would indicate user-space has provided a schema with a duplicate domain entry. e.g: | L3:0=ffff;1=f00f;0=f00f If new parsers are added this duplicate domain test has to be duplicated. Move it to the caller. Signed-off-by: James Morse (cherry picked from commit 827c80b5ec1b14a0f3d77e12ad13a8fbbf499ccd https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/ctrlmondata.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/ctrlmondata.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 1eac8f7dc07ac..48ebc0f5bafbb 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -84,12 +84,6 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s, u32 closid = data->closid; u32 bw_val; - cfg = &d->staged_config[s->conf_type]; - if (cfg->have_new_ctrl) { - rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id); - return -EINVAL; - } - if (!bw_validate(data->buf, &bw_val, r)) return -EINVAL; @@ -98,6 +92,7 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s, return 0; } + cfg = &d->staged_config[s->conf_type]; cfg->new_ctrl = bw_val; cfg->have_new_ctrl = true; @@ -165,12 +160,6 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, u32 closid = data->closid; u32 cbm_val; - cfg = &d->staged_config[s->conf_type]; - if (cfg->have_new_ctrl) { - rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id); - return -EINVAL; - } - /* * Cannot set up more than one pseudo-locked region in a cache * hierarchy. @@ -207,6 +196,7 @@ static int parse_cbm(struct rdt_parse_data *data, struct resctrl_schema *s, } } + cfg = &d->staged_config[s->conf_type]; cfg->new_ctrl = cbm_val; cfg->have_new_ctrl = true; @@ -264,13 +254,18 @@ static int parse_line(char *line, struct resctrl_schema *s, dom = strim(dom); list_for_each_entry(d, &r->ctrl_domains, hdr.list) { if (d->hdr.id == dom_id) { + cfg = &d->staged_config[t]; + if (cfg->have_new_ctrl) { + rdt_last_cmd_printf("Duplicate domain %d\n", d->hdr.id); + return -EINVAL; + } + data.buf = dom; data.closid = rdtgrp->closid; data.mode = rdtgrp->mode; if (parse_ctrlval(&data, s, d)) return -EINVAL; if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP) { - cfg = &d->staged_config[t]; /* * In pseudo-locking setup mode and just * parsed a valid CBM that should be From 27e5d0f8a50d8e2b03f05cbb42a75ef77edb2fd6 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 19 Nov 2024 15:02:03 +0000 Subject: [PATCH 359/464] NVIDIA: SAUCE: fs/resctrl: Move MBA supported check to parse_line() instead of parse_bw() BugLink: https://bugs.launchpad.net/bugs/2154527 MBA is only supported on platforms where the delay inserted by the control is linear. Resctrl checks the two properties provided by the arch code match each time it parses part of a new control value. This doesn't need to be done so frequently, and obscures changes to parse_bw() to abstract it for use with other control types. Move this check to the parse_line() caller so it only happens once. Signed-off-by: James Morse (cherry picked from commit 85be43b4b1214a6f88d5643a8973ec6808cec56c https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/ctrlmondata.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 48ebc0f5bafbb..ec9ea0f607191 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -45,15 +45,6 @@ static bool bw_validate(char *buf, u32 *data, struct rdt_resource *r) int ret; u32 bw; - /* - * Only linear delay values is supported for current Intel SKUs. - */ - if (r->rid == RDT_RESOURCE_MBA && - !r->mba.delay_linear && r->mba.arch_needs_linear) { - rdt_last_cmd_puts("No support for non-linear MB domains\n"); - return false; - } - ret = kstrtou32(buf, 10, &bw); if (ret) { rdt_last_cmd_printf("Invalid MB value %s\n", buf); @@ -242,6 +233,15 @@ static int parse_line(char *line, struct resctrl_schema *s, return -EINVAL; } + /* + * Only linear delay values is supported for current Intel SKUs. + */ + if (r->rid == RDT_RESOURCE_MBA && + !r->mba.delay_linear && r->mba.arch_needs_linear) { + rdt_last_cmd_puts("No support for non-linear MB domains\n"); + return -EINVAL; + } + next: if (!line || line[0] == '\0') return 0; From 4bbdfc3d059f7eeaeb02294c367c5472491ca520 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 19 Nov 2024 15:55:45 +0000 Subject: [PATCH 360/464] NVIDIA: SAUCE: fs/resctrl: Rename resctrl_get_default_ctrl() to include resource BugLink: https://bugs.launchpad.net/bugs/2154527 resctrl_get_default_ctrl() is called by both the architecture code and filesystem code to return the default value for a control. This depends on the schema format. parse_bw() doesn't bother checking the bounds it is given if the resource is in use by mba_sc. This is because the values parsed from user-space are not the same as those the control should take. To make this disparity easier to work with, a second different copy of the schema format is needed, which would need a version of resctrl_get_default_ctrl(). This would let the resctrl change the schema format presented to user-space, provided it converts it to match what the architecture code expects. Rename resctrl_get_default_ctrl() to make it clear it returns the resource default. Signed-off-by: James Morse (cherry picked from commit a4ba73c6546aaf2eb6805ad910b27c55663843e0 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/x86/kernel/cpu/resctrl/core.c | 2 +- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 2 +- drivers/resctrl/mpam_resctrl.c | 10 +++++----- fs/resctrl/rdtgroup.c | 4 ++-- include/linux/resctrl.h | 13 ++++++++----- 5 files changed, 17 insertions(+), 14 deletions(-) diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c index ba3316a41141b..244f0d2a93e7a 100644 --- a/arch/x86/kernel/cpu/resctrl/core.c +++ b/arch/x86/kernel/cpu/resctrl/core.c @@ -378,7 +378,7 @@ static void setup_default_ctrlval(struct rdt_resource *r, u32 *dc) * For Memory Allocation: Set b/w requested to 100% */ for (i = 0; i < hw_res->num_closid; i++, dc++) - *dc = resctrl_get_default_ctrl(r); + *dc = resctrl_get_resource_default_ctrl(r); } static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom) diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c index 8850264684405..8a017f1111028 100644 --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c @@ -253,7 +253,7 @@ void resctrl_arch_reset_all_ctrls(struct rdt_resource *r) hw_dom = resctrl_to_arch_ctrl_dom(d); for (i = 0; i < hw_res->num_closid; i++) - hw_dom->ctrl_val[i] = resctrl_get_default_ctrl(r); + hw_dom->ctrl_val[i] = resctrl_get_resource_default_ctrl(r); msr_param.dom = d; smp_call_function_any(&d->hdr.cpu_mask, rdt_ctrl_update, &msr_param, 1); } diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 25acac6f4d456..93dfbd6a61dfa 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1411,7 +1411,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) * we have configured the SMMU and GIC not to do this 'all the * bits' is the correct answer here. */ - r->cache.shareable_bits = resctrl_get_default_ctrl(r); + r->cache.shareable_bits = resctrl_get_resource_default_ctrl(r); r->alloc_capable = true; break; case RDT_RESOURCE_MBA: @@ -1562,7 +1562,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, lockdep_assert_cpus_held(); if (!mpam_is_enabled()) - return resctrl_get_default_ctrl(r); + return resctrl_get_resource_default_ctrl(r); res = container_of(r, struct mpam_resctrl_res, resctrl_res); dom = container_of(d, struct mpam_resctrl_dom, resctrl_ctrl_dom); @@ -1591,12 +1591,12 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, } fallthrough; default: - return resctrl_get_default_ctrl(r); + return resctrl_get_resource_default_ctrl(r); } if (!r->alloc_capable || partid >= resctrl_arch_get_num_closid(r) || !mpam_has_feature(configured_by, cfg)) - return resctrl_get_default_ctrl(r); + return resctrl_get_resource_default_ctrl(r); switch (configured_by) { case mpam_feat_cpor_part: @@ -1604,7 +1604,7 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, case mpam_feat_mbw_max: return mbw_max_to_percent(cfg->mbw_max, cprops); default: - return resctrl_get_default_ctrl(r); + return resctrl_get_resource_default_ctrl(r); } } diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 590a76d0cf990..48a56092d909b 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1111,7 +1111,7 @@ static int rdt_default_ctrl_show(struct kernfs_open_file *of, struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); struct rdt_resource *r = s->res; - seq_printf(seq, "%x\n", resctrl_get_default_ctrl(r)); + seq_printf(seq, "%x\n", resctrl_get_resource_default_ctrl(r)); return 0; } @@ -3794,7 +3794,7 @@ static void rdtgroup_init_mba(struct rdt_resource *r, u32 closid) } cfg = &d->staged_config[CDP_NONE]; - cfg->new_ctrl = resctrl_get_default_ctrl(r); + cfg->new_ctrl = resctrl_get_resource_default_ctrl(r); cfg->have_new_ctrl = true; } } diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index f0a33e4762201..d6f68f7e65964 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -323,7 +323,10 @@ struct resctrl_mon { * @mon_domains: RCU list of all monitor domains for this resource * @mba: Properties of the MBA resource * @name: Name to use in "schemata" file. - * @schema_fmt: Which format string and parser is used for this schema. + * @schema_fmt: Which format control parameters should be in for this resource. + * @evt_list: List of monitoring events + * @mbm_cfg_mask: Bandwidth sources that can be tracked when bandwidth + * monitoring events can be configured. * @cdp_capable: Is the CDP feature available on this resource */ struct rdt_resource { @@ -402,11 +405,11 @@ struct resctrl_mon_config_info { void resctrl_arch_sync_cpu_closid_rmid(void *info); /** - * resctrl_get_default_ctrl() - Return the default control value for this - * resource. - * @r: The resource whose default control type is queried. + * resctrl_get_resource_default_ctrl() - Return the default control value for + * this resource. + * @r: The resource whose default control value is queried. */ -static inline u32 resctrl_get_default_ctrl(struct rdt_resource *r) +static inline u32 resctrl_get_resource_default_ctrl(struct rdt_resource *r) { switch (r->schema_fmt) { case RESCTRL_SCHEMA_BITMAP: From 486ccff29755bf78ad80b4733497d6cea8c15693 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 12:21:25 +0000 Subject: [PATCH 361/464] NVIDIA: SAUCE: fs/resctrl: Add a schema format to the schema, allowing it to be different BugLink: https://bugs.launchpad.net/bugs/2154527 parse_bw() doesn't bother checking the bounds it is given if the resource is in use by mba_sc. This is because the values parsed from user-space are not the same as those the control should take. To make this disparity easier to work with, a second different copy of the schema format is needed, which would need a version of resctrl_get_default_ctrl(). This would let the resctrl change the schema format presented to user-space, provided it converts it to match what the architecture code expects. Add a second schema format for use with mba_sc. The membw properties are copied and the schema version is used. When mba_sc is enabled the schema copy of these properties is modified. Signed-off-by: James Morse (cherry picked from commit 225d28eb849877c6b97dcdc466d8e1aa67978272 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/ctrlmondata.c`; - Resolve minor conflicts in `include/linux/arm_mpam.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 ++-- fs/resctrl/ctrlmondata.c | 14 ++++++------ fs/resctrl/rdtgroup.c | 26 +++++++++++++++++------ include/linux/arm_mpam.h | 4 +--- include/linux/resctrl.h | 24 ++++++++++++++++++++- 5 files changed, 52 insertions(+), 20 deletions(-) diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c index d539e56c2b1f0..91ce05256a004 100644 --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c @@ -20,9 +20,9 @@ #include "internal.h" -u32 resctrl_arch_round_bw(u32 val, const struct rdt_resource *r) +u32 resctrl_arch_round_bw(u32 val, const struct resctrl_schema *s) { - return roundup(val, (unsigned long)r->membw.bw_gran); + return roundup(val, (unsigned long)s->membw.bw_gran); } int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index ec9ea0f607191..1e51c4a01e785 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -40,7 +40,7 @@ typedef int (ctrlval_parser_t)(struct rdt_parse_data *data, * hardware. The allocated bandwidth percentage is rounded to the next * control step available on the hardware. */ -static bool bw_validate(char *buf, u32 *data, struct rdt_resource *r) +static bool bw_validate(char *buf, u32 *data, struct resctrl_schema *s) { int ret; u32 bw; @@ -52,18 +52,18 @@ static bool bw_validate(char *buf, u32 *data, struct rdt_resource *r) } /* Nothing else to do if software controller is enabled. */ - if (is_mba_sc(r)) { + if (is_mba_sc(s->res)) { *data = bw; return true; } - if (bw < r->membw.min_bw || bw > r->membw.max_bw) { + if (bw < s->membw.min_bw || bw > s->membw.max_bw) { rdt_last_cmd_printf("MB value %u out of range [%d,%d]\n", - bw, r->membw.min_bw, r->membw.max_bw); + bw, s->membw.min_bw, s->membw.max_bw); return false; } - *data = resctrl_arch_round_bw(bw, r); + *data = resctrl_arch_round_bw(bw, s); return true; } @@ -75,7 +75,7 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s, u32 closid = data->closid; u32 bw_val; - if (!bw_validate(data->buf, &bw_val, r)) + if (!bw_validate(data->buf, &bw_val, s)) return -EINVAL; if (is_mba_sc(r)) { @@ -215,7 +215,7 @@ static int parse_line(char *line, struct resctrl_schema *s, /* Walking r->domains, ensure it can't race with cpuhp */ lockdep_assert_cpus_held(); - switch (r->schema_fmt) { + switch (s->schema_fmt) { case RESCTRL_SCHEMA_BITMAP: parse_ctrlval = &parse_cbm; break; diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 48a56092d909b..633a646337910 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1109,9 +1109,8 @@ static int rdt_default_ctrl_show(struct kernfs_open_file *of, struct seq_file *seq, void *v) { struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); - struct rdt_resource *r = s->res; - seq_printf(seq, "%x\n", resctrl_get_resource_default_ctrl(r)); + seq_printf(seq, "%x\n", resctrl_get_schema_default_ctrl(s)); return 0; } @@ -1252,9 +1251,8 @@ static int rdt_min_bw_show(struct kernfs_open_file *of, struct seq_file *seq, void *v) { struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); - struct rdt_resource *r = s->res; - seq_printf(seq, "%u\n", r->membw.min_bw); + seq_printf(seq, "%u\n", s->membw.min_bw); return 0; } @@ -1290,9 +1288,8 @@ static int rdt_bw_gran_show(struct kernfs_open_file *of, struct seq_file *seq, void *v) { struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); - struct rdt_resource *r = s->res; - seq_printf(seq, "%u\n", r->membw.bw_gran); + seq_printf(seq, "%u\n", s->membw.bw_gran); return 0; } @@ -2834,7 +2831,22 @@ static int schemata_list_add(struct rdt_resource *r, enum resctrl_conf_type type if (cl > max_name_width) max_name_width = cl; - switch (r->schema_fmt) { + s->schema_fmt = r->schema_fmt; + s->membw = r->membw; + + /* + * When mba_sc() is enabled the format used by user space is different + * to that expected by hardware. The conversion is done by + * update_mba_bw(). + */ + if (is_mba_sc(r)) { + s->schema_fmt = RESCTRL_SCHEMA_RANGE; + s->membw.min_bw = 0; + s->membw.max_bw = MBA_MAX_MBPS; + s->membw.bw_gran = 1; + } + + switch (s->schema_fmt) { case RESCTRL_SCHEMA_BITMAP: s->fmt_str = "%d=%x"; break; diff --git a/include/linux/arm_mpam.h b/include/linux/arm_mpam.h index 0891f85383674..d3983af5c6794 100644 --- a/include/linux/arm_mpam.h +++ b/include/linux/arm_mpam.h @@ -79,10 +79,8 @@ static inline void resctrl_arch_enable_alloc(void) { } static inline void resctrl_arch_disable_alloc(void) { } struct resctrl_schema; - -struct rdt_resource; static inline u32 resctrl_arch_round_bw(u32 val, - const struct rdt_resource *r __always_unused) + const struct resctrl_schema *s __always_unused) { /* * Do nothing: for MPAM, resctrl_arch_update_one() has the necessary diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index d6f68f7e65964..348da7c59425d 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -359,9 +359,12 @@ struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l); * @list: Member of resctrl_schema_all. * @name: The name to use in the "schemata" file. * @fmt_str: Format string to show domain value. + * @schema_fmt: Which format string and parser is used for this schema. * @conf_type: Whether this schema is specific to code/data. * @res: The resource structure exported by the architecture to describe * the hardware that is configured by this schema. + * @membw The properties of the schema which may be different to the format + * that was specified by the resource, * @num_closid: The number of closid that can be used with this schema. When * features like CDP are enabled, this will be lower than the * hardware supports for the resource. @@ -370,8 +373,10 @@ struct resctrl_schema { struct list_head list; char name[8]; const char *fmt_str; + enum resctrl_schema_fmt schema_fmt; enum resctrl_conf_type conf_type; struct rdt_resource *res; + struct resctrl_membw membw; u32 num_closid; }; @@ -421,6 +426,23 @@ static inline u32 resctrl_get_resource_default_ctrl(struct rdt_resource *r) return WARN_ON_ONCE(1); } +/** + * resctrl_get_schema_default_ctrl() - Return the default control value for + * this schema. + * @s: The schema whose default control value is queried. + */ +static inline u32 resctrl_get_schema_default_ctrl(struct resctrl_schema *s) +{ + switch (s->schema_fmt) { + case RESCTRL_SCHEMA_BITMAP: + return resctrl_get_resource_default_ctrl(s->res); + case RESCTRL_SCHEMA_RANGE: + return s->membw.max_bw; + } + + return WARN_ON_ONCE(1); +} + /* The number of closid supported by this resource regardless of CDP */ u32 resctrl_arch_get_num_closid(struct rdt_resource *r); u32 resctrl_arch_system_num_rmid_idx(void); @@ -512,7 +534,7 @@ bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r); */ int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable); -u32 resctrl_arch_round_bw(u32 val, const struct rdt_resource *r); +u32 resctrl_arch_round_bw(u32 val, const struct resctrl_schema *s); /* * Update the ctrl_val and apply this config right now. From c32bc5cf2c5a7c76862e4d4c8da0f320f62c8679 Mon Sep 17 00:00:00 2001 From: James Morse Date: Fri, 27 Sep 2024 17:59:15 +0100 Subject: [PATCH 362/464] NVIDIA: VR: SAUCE: fs/resctrl: Use schema format to check the resource is a bitmap BugLink: https://bugs.launchpad.net/bugs/2154527 rdtgroup_cbm_to_size() uses a WARN_ON_ONCE() to assert that the resource it has been passed is one of the L2 or L3 cache. This is to avoid using uninitialised bitmap properties. Updating this list for every resource that is configured by a bitmap doesn't scale. Instead change the WARN_ON_ONCE() to use the schema format the arch code requested for the resource. Signed-off-by: James Morse (cherry picked from commit 04f3b4e4e1fcd4fc02d59a4c7a27619f8abf4902 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/rdtgroup.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 633a646337910..29d7fa97d64d4 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1631,7 +1631,7 @@ unsigned int rdtgroup_cbm_to_size(struct rdt_resource *r, struct cacheinfo *ci; int num_b; - if (WARN_ON_ONCE(r->ctrl_scope != RESCTRL_L2_CACHE && r->ctrl_scope != RESCTRL_L3_CACHE)) + if (WARN_ON_ONCE(r->schema_fmt != RESCTRL_SCHEMA_BITMAP)) return size; num_b = bitmap_weight(&cbm, r->cache.cbm_len); @@ -1718,11 +1718,11 @@ static int rdtgroup_size_show(struct kernfs_open_file *of, ctrl = resctrl_arch_get_config(r, d, closid, type); - if (r->rid == RDT_RESOURCE_MBA || - r->rid == RDT_RESOURCE_SMBA) - size = ctrl; - else + + if (schema->schema_fmt == RESCTRL_SCHEMA_BITMAP) size = rdtgroup_cbm_to_size(r, d, ctrl); + else + size = ctrl; } seq_printf(s, "%d=%u", d->hdr.id, size); sep = true; From 05d07d6f6244cf41b310b7992a86d5d5820e6f1d Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 15:15:54 +0000 Subject: [PATCH 363/464] NVIDIA: SAUCE: fs/resctrl: Add specific schema types for 'range' BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl allows the architecture code to specify the schema format for a control. Controls can either take a bitmap, or some kind of number. If user-space doesn't know what a control is by its name, it could be told the schema format. 'Some kind of number' isn't useful as the difference between a percentage and a value in MB/s affects how these would be programmed, even if resctrl's parsing code doesn't need to care. Add the types resctrl already has in addition to 'range'. This allows architectures to move over before 'range' is removed. These new schema formats are parsed the same, but will additionally affect which files are visible. Schema formats with a double underscore should not be considered portable between architectures, and are likely to be described to user-space as 'platform defined'. AMDs MBA resource is configured with an absolute bandwidth measured in multiples of one eighth of a GB per second. resctrl needs to be aware of this platform defined format to ensure the existing 'MB' files continue to be shown. Signed-off-by: James Morse (cherry picked from commit bb81e4805d5120058ec44f793780bdf1e775cd5a https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/ctrlmondata.c | 3 +++ fs/resctrl/rdtgroup.c | 3 +++ include/linux/resctrl.h | 12 ++++++++++++ 3 files changed, 18 insertions(+) diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 1e51c4a01e785..ec925ce6c8773 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -220,6 +220,9 @@ static int parse_line(char *line, struct resctrl_schema *s, parse_ctrlval = &parse_cbm; break; case RESCTRL_SCHEMA_RANGE: + case RESCTRL_SCHEMA_PERCENT: + case RESCTRL_SCHEMA_MBPS: + case RESCTRL_SCHEMA__AMD_MBA: parse_ctrlval = &parse_bw; break; } diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 29d7fa97d64d4..cda9e35f48698 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2851,6 +2851,9 @@ static int schemata_list_add(struct rdt_resource *r, enum resctrl_conf_type type s->fmt_str = "%d=%x"; break; case RESCTRL_SCHEMA_RANGE: + case RESCTRL_SCHEMA_PERCENT: + case RESCTRL_SCHEMA_MBPS: + case RESCTRL_SCHEMA__AMD_MBA: s->fmt_str = "%d=%u"; break; } diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 348da7c59425d..ca20e7ed32de2 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -285,10 +285,16 @@ enum resctrl_scope { * enum resctrl_schema_fmt - The format user-space provides for a schema. * @RESCTRL_SCHEMA_BITMAP: The schema is a bitmap in hex. * @RESCTRL_SCHEMA_RANGE: The schema is a decimal number. + * @RESCTRL_SCHEMA_PERCENT: The schema is a percentage. + * @RESCTRL_SCHEMA_MBPS: The schema ia a MBps value. + * @RESCTRL_SCHEMA__AMD_MBA: The schema value is MBA for AMD platforms. */ enum resctrl_schema_fmt { RESCTRL_SCHEMA_BITMAP, RESCTRL_SCHEMA_RANGE, + RESCTRL_SCHEMA_PERCENT, + RESCTRL_SCHEMA_MBPS, + RESCTRL_SCHEMA__AMD_MBA, }; /** @@ -420,6 +426,9 @@ static inline u32 resctrl_get_resource_default_ctrl(struct rdt_resource *r) case RESCTRL_SCHEMA_BITMAP: return BIT_MASK(r->cache.cbm_len) - 1; case RESCTRL_SCHEMA_RANGE: + case RESCTRL_SCHEMA_PERCENT: + case RESCTRL_SCHEMA_MBPS: + case RESCTRL_SCHEMA__AMD_MBA: return r->membw.max_bw; } @@ -437,6 +446,9 @@ static inline u32 resctrl_get_schema_default_ctrl(struct resctrl_schema *s) case RESCTRL_SCHEMA_BITMAP: return resctrl_get_resource_default_ctrl(s->res); case RESCTRL_SCHEMA_RANGE: + case RESCTRL_SCHEMA_PERCENT: + case RESCTRL_SCHEMA_MBPS: + case RESCTRL_SCHEMA__AMD_MBA: return s->membw.max_bw; } From 0224d5a9dbca24cdef27efe04f62f818de8937e8 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 15:19:37 +0000 Subject: [PATCH 364/464] NVIDIA: SAUCE: x86/resctrl: Move over to specifying MBA control formats BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl specifies the schema format for MB and SMBA in rdt_resources_all[]. Intel platforms take a percentage for MB, AMD platforms take an absolute value which isn't MB/s. Currently these are both treated as a 'range'. Adding support for additional types of control shows that user-space needs to be told what the control formats are. Today users of resctrl must already know if their platform is Intel or AMD to know how the MB resource will behave. The MPAM support exposes new control types that take a 'percentage'. The Intel MB resource is also configured by a percentage, so should be able to expose this to user-space. Remove the static configuration for schema_fmt in rdt_resources_all[] and specify it with the other control properties in __get_mem_config_intel() or __get_mem_config_amd(). Signed-off-by: James Morse (cherry picked from commit 3323499e5df777ad2eb10be5c7dc29ae5358c93d https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `arch/x86/kernel/cpu/resctrl/core.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- arch/x86/kernel/cpu/resctrl/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c index 244f0d2a93e7a..00bc2e3ed0ff1 100644 --- a/arch/x86/kernel/cpu/resctrl/core.c +++ b/arch/x86/kernel/cpu/resctrl/core.c @@ -88,7 +88,6 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { .name = "MB", .ctrl_scope = RESCTRL_L3_CACHE, .ctrl_domains = ctrl_domain_init(RDT_RESOURCE_MBA), - .schema_fmt = RESCTRL_SCHEMA_RANGE, }, }, [RDT_RESOURCE_SMBA] = @@ -97,7 +96,6 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = { .name = "SMBA", .ctrl_scope = RESCTRL_L3_CACHE, .ctrl_domains = ctrl_domain_init(RDT_RESOURCE_SMBA), - .schema_fmt = RESCTRL_SCHEMA_RANGE, }, }, [RDT_RESOURCE_PERF_PKG] = @@ -211,6 +209,7 @@ static __init bool __get_mem_config_intel(struct rdt_resource *r) cpuid_count(0x00000010, 3, &eax.full, &ebx, &ecx, &edx.full); hw_res->num_closid = edx.split.cos_max + 1; max_delay = eax.split.max_delay + 1; + r->schema_fmt = RESCTRL_SCHEMA_PERCENT; r->membw.max_bw = MAX_MBA_BW; r->mba.arch_needs_linear = true; if (ecx & MBA_IS_LINEAR) { @@ -246,6 +245,7 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r) cpuid_count(0x80000020, subleaf, &eax, &ebx, &ecx, &edx); hw_res->num_closid = edx + 1; + r->schema_fmt = RESCTRL_SCHEMA__AMD_MBA; r->membw.max_bw = 1 << eax; /* AMD does not use delay */ From ffef91cd1ecfe4b56586688423d1c0d58cdcc3c0 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 15:30:08 +0000 Subject: [PATCH 365/464] NVIDIA: VR: SAUCE: arm_mpam: resctrl: Convert MB resource to use percentage BugLink: https://bugs.launchpad.net/bugs/2154527 MPAMs bandwidth controls are both exposed to resctrl as if they take a percentage. Update the schema format so that user-space can be told this is a perentage, and files that describe this control format are exposed. (e.g. min_percent) Existing variation in this area is covered by requiring user-space to know if it is running on an Intel or AMD platform. Exposing the schema format directly will avoid modifying user-space to know it is running on an MPAM or RISCV platform. MPAM can also expose bitmap controls for memory bandwidth, which may become important for use-cases in the future. These are currently converted to a percentage to fit the existing definition of the MB resource. Signed-off-by: James Morse (cherry picked from commit 2baa164d3c899703f228ad0d2e9ad7d4856203e8 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 93dfbd6a61dfa..d498f7aa8ce7e 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1415,7 +1415,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) r->alloc_capable = true; break; case RDT_RESOURCE_MBA: - r->schema_fmt = RESCTRL_SCHEMA_RANGE; + r->schema_fmt = RESCTRL_SCHEMA_PERCENT; r->ctrl_scope = RESCTRL_L3_CACHE; r->mba.delay_linear = true; From 65f91a539900debe91897728bbc2f0e501342257 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 15:32:43 +0000 Subject: [PATCH 366/464] NVIDIA: VR: SAUCE: fs/resctrl: Remove 'range' schema format BugLink: https://bugs.launchpad.net/bugs/2154527 Resctrl previously had a 'range' schema format that took some kind of number. This has since been split into percentage, MB/s and an AMD platform specific scheme. As range is no longer used, remove it. The last user is mba_sc which should be described as taking MB/s. Signed-off-by: James Morse (cherry picked from commit 6c8f021bc7f7070728763a9a5ddaee7d5f291099 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/ctrlmondata.c | 1 - fs/resctrl/rdtgroup.c | 3 +-- include/linux/resctrl.h | 4 ---- 3 files changed, 1 insertion(+), 7 deletions(-) diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index ec925ce6c8773..8d00aeacc337c 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -219,7 +219,6 @@ static int parse_line(char *line, struct resctrl_schema *s, case RESCTRL_SCHEMA_BITMAP: parse_ctrlval = &parse_cbm; break; - case RESCTRL_SCHEMA_RANGE: case RESCTRL_SCHEMA_PERCENT: case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index cda9e35f48698..edf2ed96138b0 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2840,7 +2840,7 @@ static int schemata_list_add(struct rdt_resource *r, enum resctrl_conf_type type * update_mba_bw(). */ if (is_mba_sc(r)) { - s->schema_fmt = RESCTRL_SCHEMA_RANGE; + s->schema_fmt = RESCTRL_SCHEMA_MBPS; s->membw.min_bw = 0; s->membw.max_bw = MBA_MAX_MBPS; s->membw.bw_gran = 1; @@ -2850,7 +2850,6 @@ static int schemata_list_add(struct rdt_resource *r, enum resctrl_conf_type type case RESCTRL_SCHEMA_BITMAP: s->fmt_str = "%d=%x"; break; - case RESCTRL_SCHEMA_RANGE: case RESCTRL_SCHEMA_PERCENT: case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index ca20e7ed32de2..ed6d1b5b39619 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -284,14 +284,12 @@ enum resctrl_scope { /** * enum resctrl_schema_fmt - The format user-space provides for a schema. * @RESCTRL_SCHEMA_BITMAP: The schema is a bitmap in hex. - * @RESCTRL_SCHEMA_RANGE: The schema is a decimal number. * @RESCTRL_SCHEMA_PERCENT: The schema is a percentage. * @RESCTRL_SCHEMA_MBPS: The schema ia a MBps value. * @RESCTRL_SCHEMA__AMD_MBA: The schema value is MBA for AMD platforms. */ enum resctrl_schema_fmt { RESCTRL_SCHEMA_BITMAP, - RESCTRL_SCHEMA_RANGE, RESCTRL_SCHEMA_PERCENT, RESCTRL_SCHEMA_MBPS, RESCTRL_SCHEMA__AMD_MBA, @@ -425,7 +423,6 @@ static inline u32 resctrl_get_resource_default_ctrl(struct rdt_resource *r) switch (r->schema_fmt) { case RESCTRL_SCHEMA_BITMAP: return BIT_MASK(r->cache.cbm_len) - 1; - case RESCTRL_SCHEMA_RANGE: case RESCTRL_SCHEMA_PERCENT: case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: @@ -445,7 +442,6 @@ static inline u32 resctrl_get_schema_default_ctrl(struct resctrl_schema *s) switch (s->schema_fmt) { case RESCTRL_SCHEMA_BITMAP: return resctrl_get_resource_default_ctrl(s->res); - case RESCTRL_SCHEMA_RANGE: case RESCTRL_SCHEMA_PERCENT: case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: From a4a9ec7a0a2e5136cc9a36d0de576b37b435d613 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 15:49:06 +0000 Subject: [PATCH 367/464] NVIDIA: SAUCE: fs/resctrl: Add additional files for percentage and bitmap controls BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM has cache capacity controls that effectively take a percentage. Resctrl supports percentages, but the collection of files that are exposed to describe this control belong to the MB resource. To find the minimum granularity of the percentage cache capacity controls, user-space is expected to rad the banwdidth_gran file, and know this has nothing to do with bandwidth. The only problem here is the name of the file. Add duplicates of these properties with percentage and bitmap in the name. These will be exposed based on the schema format. The existing files must remain tied to the specific resources so that they remain visible to user-space. Using the same helpers ensures the values will always be the same regardless of the file used. These files are not exposed until the new RFTYPE schema flags are set on a resource 'fflags'. Signed-off-by: James Morse (cherry picked from commit a38c11612e84a927e5b6e2dccf765291a4d498fd https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/internal.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/internal.h | 6 ++++++ fs/resctrl/rdtgroup.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h index 740e32a7c78e2..3c02cab6c94e8 100644 --- a/fs/resctrl/internal.h +++ b/fs/resctrl/internal.h @@ -252,6 +252,7 @@ struct rdtgroup { #define RFTYPE_TOP BIT(6) +/* files that are specific to a type of resource, e.g. throttle_mode */ #define RFTYPE_RES_CACHE BIT(8) #define RFTYPE_RES_MB BIT(9) @@ -262,6 +263,11 @@ struct rdtgroup { #define RFTYPE_RES_PERF_PKG BIT(12) +/* files that are specific to a type of control, e.g. percent_min */ +#define RFTYPE_SCHEMA_BITMAP BIT(13) +#define RFTYPE_SCHEMA_PERCENT BIT(14) +#define RFTYPE_SCHEMA_MBPS BIT(15) + #define RFTYPE_CTRL_INFO (RFTYPE_INFO | RFTYPE_CTRL) #define RFTYPE_MON_INFO (RFTYPE_INFO | RFTYPE_MON) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index edf2ed96138b0..4498301805d05 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2033,6 +2033,13 @@ static struct rftype res_common_files[] = { .kf_ops = &rdtgroup_kf_single_ops, .seq_show = resctrl_num_mbm_cntrs_show, }, + { + .name = "bitmap_mask", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdt_default_ctrl_show, + .fflags = RFTYPE_CTRL_INFO | RFTYPE_SCHEMA_BITMAP, + }, { .name = "min_cbm_bits", .mode = 0444, @@ -2040,6 +2047,13 @@ static struct rftype res_common_files[] = { .seq_show = rdt_min_cbm_bits_show, .fflags = RFTYPE_CTRL_INFO | RFTYPE_RES_CACHE, }, + { + .name = "bitmaps_min_bits", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdt_min_cbm_bits_show, + .fflags = RFTYPE_CTRL_INFO | RFTYPE_SCHEMA_BITMAP, + }, { .name = "shareable_bits", .mode = 0444, @@ -2061,6 +2075,13 @@ static struct rftype res_common_files[] = { .seq_show = rdt_min_bw_show, .fflags = RFTYPE_CTRL_INFO | RFTYPE_RES_MB, }, + { + .name = "percent_min", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdt_min_bw_show, + .fflags = RFTYPE_CTRL_INFO | RFTYPE_SCHEMA_PERCENT, + }, { .name = "bandwidth_gran", .mode = 0444, @@ -2068,6 +2089,13 @@ static struct rftype res_common_files[] = { .seq_show = rdt_bw_gran_show, .fflags = RFTYPE_CTRL_INFO | RFTYPE_RES_MB, }, + { + .name = "percent_gran", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdt_bw_gran_show, + .fflags = RFTYPE_CTRL_INFO | RFTYPE_SCHEMA_PERCENT, + }, { .name = "delay_linear", .mode = 0444, From be538d28fd45a7f78f5692f0074f87a61a478c28 Mon Sep 17 00:00:00 2001 From: James Morse Date: Wed, 20 Nov 2024 16:55:39 +0000 Subject: [PATCH 368/464] NVIDIA: SAUCE: fs/resctrl: Add fflags_from_schema() for files based on schema format BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM has cache capacity controls that effectively take a percentage. Resctrl supports percentages, but the collection of files that are exposed to describe this control belong to the MB resource. New files have been added that are selected based on the schema format. Apply the flags to enable these files based on the schema format. Add a new fflags_from_schema() that is used for controls. Signed-off-by: James Morse (cherry picked from commit db005687c69b453ea63389314ba791dc9df18e1a https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/rdtgroup.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/rdtgroup.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 4498301805d05..3cf81ce9ea369 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2532,7 +2532,35 @@ static unsigned long fflags_from_resource(struct rdt_resource *r) return RFTYPE_RES_PERF_PKG; } - return WARN_ON_ONCE(1); + return 0; +} + +static u32 fflags_from_schema(struct resctrl_schema *s) +{ + struct rdt_resource *r = s->res; + u32 fflags = 0; + + /* Some resources are configured purely from their rid */ + fflags |= fflags_from_resource(r); + if (fflags) + return fflags; + + switch (s->schema_fmt) { + case RESCTRL_SCHEMA_BITMAP: + fflags |= RFTYPE_SCHEMA_BITMAP; + break; + case RESCTRL_SCHEMA_PERCENT: + fflags |= RFTYPE_SCHEMA_PERCENT; + break; + case RESCTRL_SCHEMA_MBPS: + fflags |= RFTYPE_SCHEMA_MBPS; + break; + case RESCTRL_SCHEMA__AMD_MBA: + /* No standard files are exposed */ + break; + } + + return fflags; } static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) @@ -2555,7 +2583,7 @@ static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn) /* loop over enabled controls, these are all alloc_capable */ list_for_each_entry(s, &resctrl_schema_all, list) { r = s->res; - fflags = fflags_from_resource(r) | RFTYPE_CTRL_INFO; + fflags = fflags_from_schema(s) | RFTYPE_CTRL_INFO; ret = rdtgroup_mkdir_info_resdir(s, s->name, fflags); if (ret) goto out_destroy; From 326a139bf78318ce038449e1d0bc2f75f49ba31a Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 10 Sep 2024 18:13:37 +0100 Subject: [PATCH 369/464] NVIDIA: SAUCE: fs/resctrl: Expose the schema format to user-space BugLink: https://bugs.launchpad.net/bugs/2154527 If more schemas are added to resctrl, user-space needs to know how to configure them. To allow user-space to configure schema it doesn't know about, it would be helpful to tell user-space the format, e.g. percentage. Add a file under info that describes the schema format. Percentages and 'mbps' are implicitly decimal, bitmaps are expected to be in hex. Signed-off-by: James Morse (cherry picked from commit f0ae6915fc22fa0a7affd46f61e0fe4a7673df06 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/rdtgroup.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/rdtgroup.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 3cf81ce9ea369..a749e36ca0971 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1792,6 +1792,30 @@ static int mbm_local_bytes_config_show(struct kernfs_open_file *of, return 0; } +static int resctrl_schema_format_show(struct kernfs_open_file *of, + struct seq_file *seq, void *v) +{ + struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); + + switch (s->schema_fmt) { + case RESCTRL_SCHEMA_BITMAP: + seq_puts(seq, "bitmap\n"); + break; + case RESCTRL_SCHEMA_PERCENT: + seq_puts(seq, "percentage\n"); + break; + case RESCTRL_SCHEMA_MBPS: + seq_puts(seq, "mbps\n"); + break; + /* The way these schema behave isn't discoverable from resctrl */ + case RESCTRL_SCHEMA__AMD_MBA: + seq_puts(seq, "platform\n"); + break; + } + + return 0; +} + static void mbm_config_write_domain(struct rdt_resource *r, struct rdt_l3_mon_domain *d, u32 evtid, u32 val) { @@ -2248,6 +2272,14 @@ static struct rftype res_common_files[] = { .seq_show = rdtgroup_closid_show, .fflags = RFTYPE_CTRL_BASE | RFTYPE_DEBUG, }, + { + .name = "schema_format", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = resctrl_schema_format_show, + .fflags = RFTYPE_CTRL_INFO, + }, + }; static int rdtgroup_add_files(struct kernfs_node *kn, unsigned long fflags) From db4aba0fa56841f2a311bec0442713a6cbd023fe Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 19 Nov 2024 12:35:13 +0000 Subject: [PATCH 370/464] NVIDIA: SAUCE: fs/resctrl: Add L2 and L3 'MAX' resource schema BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM can have both cache portion and cache capacity controls on any cache that supports MPAM. Cache portion bitmaps can be exposed via resctrl if they are implemented on L2 or L3. The cache capacity controls can not be used to isolate portions, which is in implicit in the L2 or L3 bitmap provided by user-space. These controls need to be configured with something more like a percentage. Add the resource enum entries for these two resources. No additional resctrl code is needed because the architecture code will specify this resource takes a 'percentage', re-using the support previously used only for the MB resource. Signed-off-by: James Morse (cherry picked from commit 2e9f961c2cad4bdcc49f1a598ee131725129337f https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `include/linux/resctrl.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/resctrl.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index ed6d1b5b39619..4bec4543ef6a5 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -55,6 +55,8 @@ enum resctrl_res_level { RDT_RESOURCE_MBA, RDT_RESOURCE_SMBA, RDT_RESOURCE_PERF_PKG, + RDT_RESOURCE_L3_MAX, + RDT_RESOURCE_L2_MAX, /* Must be the last */ RDT_NUM_RESOURCES, From 58e5e2fa08c8f496a57e90b45d06114edd90a8f4 Mon Sep 17 00:00:00 2001 From: James Morse Date: Tue, 19 Nov 2024 11:51:03 +0000 Subject: [PATCH 371/464] NVIDIA: SAUCE: arm_mpam: resctrl: Add the glue code to convert to/from cmax BugLink: https://bugs.launchpad.net/bugs/2154527 MPAM's maximum cache-capacity controls take a fixed point fraction format. Instead of dumping this on user-space, convert it to a percentage. User-space using resctrl already knows how to handle percentages. Signed-off-by: James Morse (cherry picked from commit 10caa1269560b1006811725d9564f0e859a53e2e https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 67 ++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index d498f7aa8ce7e..afc3a688c53d1 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -764,6 +764,13 @@ static bool cache_has_usable_cpor(struct mpam_class *class) return class->props.cpbm_wd <= 32; } +static bool cache_has_usable_cmax(struct mpam_class *class) +{ + struct mpam_props *cprops = &class->props; + + return mpam_has_feature(mpam_feat_cmax_cmax, cprops); +} + static bool mba_class_use_mbw_max(struct mpam_props *cprops) { return (mpam_has_feature(mpam_feat_mbw_max, cprops) && @@ -898,6 +905,11 @@ static u16 percent_to_mbw_max(u8 pc, struct mpam_props *cprops) return percent_to_fract16(pc, cprops->bwa_wd); } +static u16 percent_to_cmax(u8 pc, struct mpam_props *cprops) +{ + return percent_to_fract16(pc, cprops->cmax_wd); +} + static u32 get_mba_min(struct mpam_props *cprops) { if (!mba_class_use_mbw_max(cprops)) { @@ -1055,6 +1067,7 @@ static bool traffic_matches_l3(struct mpam_class *class) /* Test whether we can export MPAM_CLASS_CACHE:{2,3}? */ static void mpam_resctrl_pick_caches(void) { + bool has_cpor, has_cmax; struct mpam_class *class; struct mpam_resctrl_res *res; @@ -1073,7 +1086,9 @@ static void mpam_resctrl_pick_caches(void) continue; } - if (!cache_has_usable_cpor(class)) { + has_cpor = cache_has_usable_cpor(class); + has_cmax = cache_has_usable_cmax(class); + if (!has_cpor && !has_cmax) { pr_debug("class %u cache misses CPOR\n", class->level); continue; } @@ -1084,12 +1099,22 @@ static void mpam_resctrl_pick_caches(void) cpumask_pr_args(cpu_possible_mask)); continue; } - - if (class->level == 2) - res = &mpam_resctrl_controls[RDT_RESOURCE_L2]; - else - res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; - res->class = class; + if (has_cpor) { + pr_debug("pick_caches: Class has CPOR\n"); + if (class->level == 2) + res = &mpam_resctrl_controls[RDT_RESOURCE_L2]; + else + res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + res->class = class; + } + if (has_cmax) { + pr_debug("pick_caches: Class has CMAX\n"); + if (class->level == 2) + res = &mpam_resctrl_controls[RDT_RESOURCE_L2_MAX]; + else + res = &mpam_resctrl_controls[RDT_RESOURCE_L3_MAX]; + res->class = class; + } } } @@ -1413,6 +1438,23 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) */ r->cache.shareable_bits = resctrl_get_resource_default_ctrl(r); r->alloc_capable = true; + break; + case RDT_RESOURCE_L2_MAX: + case RDT_RESOURCE_L3_MAX: + r->alloc_capable = true; + r->schema_fmt = RESCTRL_SCHEMA_PERCENT; + r->membw.min_bw = max(100 / (1 << cprops->cmax_wd), 1); + r->membw.bw_gran = max(100 / (1 << cprops->cmax_wd), 1); + r->membw.max_bw = 100; + + if (r->rid == RDT_RESOURCE_L2_MAX) { + r->name = "L2_MAX"; + r->ctrl_scope = RESCTRL_L2_CACHE; + } else { + r->name = "L3_MAX"; + r->ctrl_scope = RESCTRL_L3_CACHE; + } + break; case RDT_RESOURCE_MBA: r->schema_fmt = RESCTRL_SCHEMA_PERCENT; @@ -1584,6 +1626,10 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, case RDT_RESOURCE_L3: configured_by = mpam_feat_cpor_part; break; + case RDT_RESOURCE_L2_MAX: + case RDT_RESOURCE_L3_MAX: + configured_by = mpam_feat_cmax_cmax; + break; case RDT_RESOURCE_MBA: if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { configured_by = mpam_feat_mbw_max; @@ -1601,6 +1647,8 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, switch (configured_by) { case mpam_feat_cpor_part: return cfg->cpbm; + case mpam_feat_cmax_cmax: + return fract16_to_percent(cfg->cmax, cprops->cmax_wd); case mpam_feat_mbw_max: return mbw_max_to_percent(cfg->mbw_max, cprops); default: @@ -1653,6 +1701,11 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, cfg.cpbm = cfg_val; mpam_set_feature(mpam_feat_cpor_part, &cfg); break; + case RDT_RESOURCE_L2_MAX: + case RDT_RESOURCE_L3_MAX: + cfg.cmax = percent_to_cmax(cfg_val, cprops); + mpam_set_feature(mpam_feat_cmax_cmax, &cfg); + break; case RDT_RESOURCE_MBA: if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { cfg.mbw_max = percent_to_mbw_max(cfg_val, cprops); From 7abd6df46f5e6091bed127f9eac5005a16bdcfa0 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Mon, 24 Nov 2025 17:14:47 -0600 Subject: [PATCH 372/464] NVIDIA: VR: SAUCE: arm_mpam: Avoid MSC teardown for the SW programming errors BugLink: https://bugs.launchpad.net/bugs/2154527 No need to destory MSC instance for the user/admin programming errors sicne it's not causing any functional issues. Signed-off-by: Shanker Donthineni (cherry picked from commit 7d348a2cf872998f094587434d1e6e61f2017445 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 0539102b68b62..666b980d2e7ca 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -2523,6 +2523,12 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) msc->id, mpam_errcode_names[errcode], partid, pmg, ris); + /* No action is required for the MPAM programming errors */ + if ((errcode != MPAM_ERRCODE_REQ_PARTID_RANGE) && + (errcode != MPAM_ERRCODE_REQ_PMG_RANGE)) { + return IRQ_HANDLED; + } + /* Disable this interrupt. */ mpam_disable_msc_ecr(msc); From f893cab3fee6f41a13df3250debd4090eda91964 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Tue, 13 May 2025 11:44:23 -0500 Subject: [PATCH 373/464] NVIDIA: VR: SAUCE: arm_mpam: Handle CPU-less numa nodes BugLink: https://bugs.launchpad.net/bugs/2154527 In a NUMA system, each node may include CPUs, memory, MPAM MSC instances, or any combination thereof. Some high-end servers may have NUMA nodes that include MPAM MSC but no CPUs. In such cases, associate all possible CPUs for those MSCs. Signed-off-by: Shanker Donthineni (cherry picked from commit 95f0fd86a3d4ff75ecda369136e905e329547dc1 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 666b980d2e7ca..849abc89311ac 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -527,6 +527,10 @@ static int mpam_ris_get_affinity(struct mpam_msc *msc, cpumask_t *affinity, case MPAM_CLASS_MEMORY: get_cpumask_from_node_id(comp->comp_id, affinity); /* affinity may be empty for CPU-less memory nodes */ + if (cpumask_empty(affinity)) { + dev_warn_once(&msc->pdev->dev, "CPU-less numa node"); + cpumask_copy(affinity, cpu_possible_mask); + } break; case MPAM_CLASS_UNKNOWN: return 0; From 3ce46c56e7901b27ec6df9fae16b739f1a067718 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Mon, 24 Nov 2025 15:04:47 -0600 Subject: [PATCH 374/464] NVIDIA: SAUCE: arm_mpam: Include all associated MSC components during domain setup BugLink: https://bugs.launchpad.net/bugs/2154527 The current MPAM driver only considers the first component associated with an online/offline CPU during domain creation and teardown. This is insufficient, as CPU-initiated traffic may traverse multiple MSCs before reaching the target, and each MSC must be programmed consistently for proper resource partitioning. Update the MPAM driver to include all components associated with a given CPU during domain setup/teardown to expose expected schemata to userspace for effective resource control. Signed-off-by: Shanker Donthineni (forward ported from commit ac1e5be5e8fddc807e9c5bbc10da3797a601bc95 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Leaves drivers/resctrl/mpam_internal.h untouched; mpam_resctrl_offline_cpu() is already void in the baseline used here. - Tightens callers (mpam_resctrl_pick_mba, mpam_resctrl_pick_counters) around traffic_matches_l3() together with topology_matches_l3() and cpumask_equal(&class->affinity, cpu_possible_mask) and does not add a traffic_matches_l3() function body here, which is already defined in upstream. - Omits any edit to exposed_alloc_capable or exposed_mon_capable; those symbols are already absent from the baseline in favor of resctrl_arch_alloc_capable() / resctrl_arch_mon_capable(). - Does not add MPAM_MAX_EVENT or a new for_each_mpam_resctrl_mon() / mpam_resctrl_counters[] sizing hunk because that monitor macro and array shape are already in the baseline. - Omits INIT_LIST_HEAD_RCU() on res->resctrl_res.ctrl_domains and mon_domains, omits moving mpam_resctrl_domain_insert() after resctrl_online_*(), and omits adding static void mpam_resctrl_online_domain_hdr(); that list setup and insert ordering are already in the baseline. - Does not replay a void->int conversion for mpam_resctrl_monitor_init() or a mpam_pmg_max + 1 num_rmid path; the baseline already has int-returning mpam_resctrl_monitor_init() and resctrl_arch_system_num_rmid_idx() for num_rmid, so only surrounding line context shifts in this file. - Adds mpam_resctrl_mon_from_res() / mpam_resctrl_res_from_mon(), mpam_resctrl_monitor_sync_abmc_vals(struct rdt_resource *r), extends mpam_resctrl_alloc_domain() / mpam_resctrl_get_domain_from_cpu() / mpam_resctrl_get_mon_domain_from_cpu() with struct mpam_component *comp, hardens topology_matches_l3() with matched_once, switches resctrl_arch_mbm_cntr_assign_enabled() to use mon->assigned_counters, and extends mpam_resctrl_pick_domain_id() so memory level > 3 uses component IDs like cache-backed classes] - Modify matched_once and cpu checking in topology_matches_l3() to have a better error handling; Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 3 +- drivers/resctrl/mpam_resctrl.c | 236 +++++++++++++++++++++------------ fs/resctrl/internal.h | 9 +- fs/resctrl/monitor.c | 95 +++++++++---- fs/resctrl/rdtgroup.c | 26 ++-- 5 files changed, 240 insertions(+), 129 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 849abc89311ac..91d27a7dc2f86 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -530,7 +530,8 @@ static int mpam_ris_get_affinity(struct mpam_msc *msc, cpumask_t *affinity, if (cpumask_empty(affinity)) { dev_warn_once(&msc->pdev->dev, "CPU-less numa node"); cpumask_copy(affinity, cpu_possible_mask); - } + } else if (class->level > 3) + cpumask_copy(affinity, cpu_possible_mask); break; case MPAM_CLASS_UNKNOWN: return 0; diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index afc3a688c53d1..df53406cbe3c5 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -175,18 +175,48 @@ static void resctrl_reset_task_closids(void) read_unlock(&tasklist_lock); } -static void mpam_resctrl_monitor_sync_abmc_vals(struct rdt_resource *l3) +static struct mpam_resctrl_mon *mpam_resctrl_mon_from_res(struct mpam_resctrl_res *res) { - l3->mon.num_mbm_cntrs = l3_num_allocated_mbwu; + struct mpam_resctrl_mon *mon; + enum resctrl_event_id eventid; + + if (!res->class) + return NULL; + + for_each_mpam_resctrl_mon(mon, eventid) { + if (mon->class == res->class) + return mon; + } + return NULL; +} + +static struct mpam_resctrl_res *mpam_resctrl_res_from_mon(struct mpam_resctrl_mon *mon) +{ + struct mpam_resctrl_res *res; + enum resctrl_res_level rid; + + if (!mon->class) + return NULL; + + for_each_mpam_resctrl_control(res, rid) { + if (res->class == mon->class) + return res; + } + return NULL; +} + +static void mpam_resctrl_monitor_sync_abmc_vals(struct rdt_resource *r) +{ + r->mon.num_mbm_cntrs = l3_num_allocated_mbwu; if (cdp_enabled) - l3->mon.num_mbm_cntrs /= 2; + r->mon.num_mbm_cntrs /= 2; - if (l3->mon.num_mbm_cntrs) { - l3->mon.mbm_cntr_assignable = mpam_resctrl_abmc_enabled(); - l3->mon.mbm_assign_on_mkdir = mpam_resctrl_abmc_enabled(); + if (r->mon.num_mbm_cntrs) { + r->mon.mbm_cntr_assignable = mpam_resctrl_abmc_enabled(); + r->mon.mbm_assign_on_mkdir = mpam_resctrl_abmc_enabled(); } else { - l3->mon.mbm_cntr_assignable = false; - l3->mon.mbm_assign_on_mkdir = false; + r->mon.mbm_cntr_assignable = false; + r->mon.mbm_assign_on_mkdir = false; } } @@ -957,10 +987,11 @@ static bool topology_matches_l3(struct mpam_class *victim) { int cpu, err; struct mpam_component *victim_iter; + bool matched_once = false; + cpumask_var_t __free(free_cpumask_var) tmp_cpumask = CPUMASK_VAR_NULL; lockdep_assert_cpus_held(); - cpumask_var_t __free(free_cpumask_var) tmp_cpumask = CPUMASK_VAR_NULL; if (!alloc_cpumask_var(&tmp_cpumask, GFP_KERNEL)) return false; @@ -974,8 +1005,11 @@ static bool topology_matches_l3(struct mpam_class *victim) } cpu = cpumask_any_and(&victim_iter->affinity, cpu_online_mask); - if (WARN_ON_ONCE(cpu >= nr_cpu_ids)) + if (cpu >= nr_cpu_ids) { + if (matched_once) + continue; return false; + } cpumask_clear(tmp_cpumask); err = find_l3_equivalent_bitmask(cpu, tmp_cpumask); @@ -995,6 +1029,7 @@ static bool topology_matches_l3(struct mpam_class *victim) return false; } + matched_once = true; } return true; @@ -1146,13 +1181,15 @@ static void mpam_resctrl_pick_mba(void) continue; } - if (!topology_matches_l3(class)) { + if ((class->level == 3) && !topology_matches_l3(class)) { pr_debug("class %u topology doesn't match L3\n", class->level); continue; } - if (!traffic_matches_l3(class)) { + /* Check memory at egress from L3 for MSC with L3 */ + if (!cpumask_equal(&class->affinity, cpu_possible_mask) && + !traffic_matches_l3(class)) { pr_debug("class %u traffic doesn't match L3 egress\n", class->level); continue; @@ -1319,7 +1356,10 @@ static void mpam_resctrl_pick_counters(void) } has_mbwu = class_has_usable_mbwu(class); - if (has_mbwu && topology_matches_l3(class)) { + if (has_mbwu && + ((class->type == MPAM_CLASS_MEMORY) || + (topology_matches_l3(class) && + traffic_matches_l3(class)))) { pr_debug("class %u has usable MBWU, and matches L3 topology", class->level); /* @@ -1389,10 +1429,16 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain * bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) { - if (r != &mpam_resctrl_controls[RDT_RESOURCE_L3].resctrl_res) + struct mpam_resctrl_res *res; + struct mpam_resctrl_mon *mon; + + res = container_of(r, struct mpam_resctrl_res, resctrl_res); + + mon = mpam_resctrl_mon_from_res(res); + if (!mon) return false; - return mpam_resctrl_abmc_enabled(); + return mon->assigned_counters ? true : false; } int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) @@ -1459,7 +1505,6 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) case RDT_RESOURCE_MBA: r->schema_fmt = RESCTRL_SCHEMA_PERCENT; r->ctrl_scope = RESCTRL_L3_CACHE; - r->mba.delay_linear = true; r->mba.throttle_mode = THREAD_THROTTLE_UNDEFINED; r->membw.min_bw = get_mba_min(cprops); @@ -1483,6 +1528,9 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) if (class->type == MPAM_CLASS_CACHE) return comp->comp_id; + if ((class->type == MPAM_CLASS_MEMORY) && (class->level > 3)) + return comp->comp_id; + if (topology_matches_l3(class)) { /* Use the corresponding L3 component ID as the domain ID */ int id = get_cpu_cacheinfo_id(cpu, 3); @@ -1504,10 +1552,10 @@ static int mpam_resctrl_pick_domain_id(int cpu, struct mpam_component *comp) */ static int mpam_resctrl_monitor_init_abmc(struct mpam_resctrl_mon *mon) { - struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + struct mpam_resctrl_res *res = mpam_resctrl_res_from_mon(mon); size_t array_size = resctrl_arch_system_num_rmid_idx() * sizeof(int); int *rmid_array __free(kfree) = kmalloc(array_size, GFP_KERNEL); - struct rdt_resource *l3 = &res->resctrl_res; + struct rdt_resource *r = &res->resctrl_res; struct mpam_class *class = mon->class; u16 num_mbwu_mon; @@ -1528,7 +1576,7 @@ static int mpam_resctrl_monitor_init_abmc(struct mpam_resctrl_mon *mon) return PTR_ERR(mon->assigned_counters); mon->mbwu_idx_to_mon = no_free_ptr(rmid_array); - mpam_resctrl_monitor_sync_abmc_vals(l3); + mpam_resctrl_monitor_sync_abmc_vals(r); return 0; } @@ -1536,8 +1584,15 @@ static int mpam_resctrl_monitor_init_abmc(struct mpam_resctrl_mon *mon) static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, enum resctrl_event_id type) { - struct mpam_resctrl_res *res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; - struct rdt_resource *l3 = &res->resctrl_res; + struct mpam_resctrl_res *res; + struct rdt_resource *r; + + if ((mon->class->type == MPAM_CLASS_MEMORY) && (mon->class->level > 3)) + res = &mpam_resctrl_controls[RDT_RESOURCE_MBA]; + else + res = &mpam_resctrl_controls[RDT_RESOURCE_L3]; + + r = &res->resctrl_res; lockdep_assert_cpus_held(); @@ -1564,8 +1619,12 @@ static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, * monitoring class. * Setting name is necessary on monitor only platforms. */ - l3->name = "L3"; - l3->mon_scope = RESCTRL_L3_CACHE; + if ((mon->class->type == MPAM_CLASS_MEMORY) && (mon->class->level > 3)) { + r->name = "MB"; + } else { + r->name = "L3"; + } + r->mon_scope = RESCTRL_L3_CACHE; /* * num-rmid is the upper bound for the number of monitoring groups that @@ -1575,10 +1634,10 @@ static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, * this does mean userspace needs to know the architecture to correctly * interpret this value. */ - l3->mon.num_rmid = resctrl_arch_system_num_rmid_idx(); + r->mon.num_rmid = resctrl_arch_system_num_rmid_idx(); if (resctrl_enable_mon_event(type, false, 0, NULL)) - l3->mon_capable = true; + r->mon_capable = true; switch (type) { case QOS_L3_MBM_LOCAL_EVENT_ID: @@ -1850,41 +1909,26 @@ static struct mpam_component *find_component(struct mpam_class *class, int cpu) } static struct mpam_resctrl_dom * -mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) +mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res, + struct mpam_component *comp) { int err; struct mpam_resctrl_dom *dom; struct rdt_l3_mon_domain *mon_d; struct rdt_ctrl_domain *ctrl_d; - struct mpam_class *class = res->class; - struct mpam_component *comp_iter, *ctrl_comp; struct rdt_resource *r = &res->resctrl_res; lockdep_assert_held(&domain_list_lock); - ctrl_comp = NULL; - guard(srcu)(&mpam_srcu); - list_for_each_entry_srcu(comp_iter, &class->components, class_list, - srcu_read_lock_held(&mpam_srcu)) { - if (cpumask_test_cpu(cpu, &comp_iter->affinity)) { - ctrl_comp = comp_iter; - break; - } - } - - /* class has no component for this CPU */ - if (WARN_ON_ONCE(!ctrl_comp)) - return ERR_PTR(-EINVAL); - dom = kzalloc_node(sizeof(*dom), GFP_KERNEL, cpu_to_node(cpu)); if (!dom) return ERR_PTR(-ENOMEM); - if (r->alloc_capable) { - dom->ctrl_comp = ctrl_comp; + if (resctrl_arch_alloc_capable()) { + dom->ctrl_comp = comp; ctrl_d = &dom->resctrl_ctrl_dom; - mpam_resctrl_domain_hdr_init(cpu, ctrl_comp, r->rid, &ctrl_d->hdr); + mpam_resctrl_domain_hdr_init(cpu, comp, r->rid, &ctrl_d->hdr); ctrl_d->hdr.type = RESCTRL_CTRL_DOMAIN; err = resctrl_online_ctrl_domain(r, ctrl_d); if (err) @@ -1895,7 +1939,7 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) pr_debug("Skipped control domain online - no controls\n"); } - if (r->mon_capable) { + if (resctrl_arch_mon_capable()) { struct mpam_component *any_mon_comp = NULL; struct mpam_resctrl_mon *mon; enum resctrl_event_id eventid; @@ -1914,7 +1958,7 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) if (!mon->class) continue; // dummy resource - mon_comp = find_component(mon->class, cpu); + mon_comp = comp ? comp: find_component(mon->class, cpu); dom->mon_comp[eventid] = mon_comp; if (mon_comp) any_mon_comp = mon_comp; @@ -1940,7 +1984,7 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) return dom; offline_ctrl_domain: - if (r->alloc_capable) { + if (resctrl_arch_alloc_capable()) { mpam_resctrl_offline_domain_hdr(cpu, &ctrl_d->hdr); resctrl_offline_ctrl_domain(r, ctrl_d); } @@ -1958,7 +2002,8 @@ mpam_resctrl_alloc_domain(unsigned int cpu, struct mpam_resctrl_res *res) * This relies on mpam_resctrl_pick_domain_id() using the L3 cache-id * for anything that is not a cache. */ -static struct mpam_resctrl_dom *mpam_resctrl_get_mon_domain_from_cpu(int cpu) +static struct mpam_resctrl_dom * +mpam_resctrl_get_mon_domain_from_cpu(int cpu, struct mpam_component *comp) { int cache_id; struct mpam_resctrl_dom *dom; @@ -1972,7 +2017,9 @@ static struct mpam_resctrl_dom *mpam_resctrl_get_mon_domain_from_cpu(int cpu) if (cache_id < 0) return NULL; - list_for_each_entry_rcu(dom, &l3->resctrl_res.mon_domains, resctrl_mon_dom.hdr.list) { + list_for_each_entry(dom, &l3->resctrl_res.mon_domains, resctrl_mon_dom.hdr.list) { + if (comp && (dom->ctrl_comp != comp)) + continue; if (dom->resctrl_mon_dom.hdr.id == cache_id) return dom; } @@ -1981,7 +2028,8 @@ static struct mpam_resctrl_dom *mpam_resctrl_get_mon_domain_from_cpu(int cpu) } static struct mpam_resctrl_dom * -mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) +mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res, + struct mpam_component *comp) { struct mpam_resctrl_dom *dom; struct rdt_resource *r = &res->resctrl_res; @@ -1989,6 +2037,8 @@ mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) lockdep_assert_cpus_held(); list_for_each_entry_rcu(dom, &r->ctrl_domains, resctrl_ctrl_dom.hdr.list) { + if (comp && (dom->ctrl_comp != comp)) + continue; if (cpumask_test_cpu(cpu, &dom->ctrl_comp->affinity)) return dom; } @@ -1997,38 +2047,44 @@ mpam_resctrl_get_domain_from_cpu(int cpu, struct mpam_resctrl_res *res) return NULL; /* Search the mon domain list too - needed on monitor only platforms. */ - return mpam_resctrl_get_mon_domain_from_cpu(cpu); + return mpam_resctrl_get_mon_domain_from_cpu(cpu, comp); } int mpam_resctrl_online_cpu(unsigned int cpu) { + struct rdt_l3_mon_domain *mon_d; + struct rdt_ctrl_domain *ctrl_d; struct mpam_resctrl_res *res; enum resctrl_res_level rid; + struct mpam_component *comp; guard(mutex)(&domain_list_lock); for_each_mpam_resctrl_control(res, rid) { struct mpam_resctrl_dom *dom; - struct rdt_resource *r = &res->resctrl_res; if (!res->class) continue; // dummy_resource; + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(comp, &res->class->components, class_list, + srcu_read_lock_held(&mpam_srcu)) { + if (!cpumask_test_cpu(cpu, &comp->affinity)) + continue; - dom = mpam_resctrl_get_domain_from_cpu(cpu, res); - if (!dom) { - dom = mpam_resctrl_alloc_domain(cpu, res); - if (IS_ERR(dom)) - return PTR_ERR(dom); - } else { - if (r->alloc_capable) { - struct rdt_ctrl_domain *ctrl_d = &dom->resctrl_ctrl_dom; - - mpam_resctrl_online_domain_hdr(cpu, &ctrl_d->hdr); - } - if (r->mon_capable) { - struct rdt_l3_mon_domain *mon_d = &dom->resctrl_mon_dom; - - mpam_resctrl_online_domain_hdr(cpu, &mon_d->hdr); + dom = mpam_resctrl_get_domain_from_cpu(cpu, res, comp); + if (!dom) { + dom = mpam_resctrl_alloc_domain(cpu, res, comp); + } else { + if (resctrl_arch_alloc_capable()) { + ctrl_d = &dom->resctrl_ctrl_dom; + mpam_resctrl_online_domain_hdr(cpu, &ctrl_d->hdr); + } + if (resctrl_arch_mon_capable()) { + mon_d = &dom->resctrl_mon_dom; + mpam_resctrl_online_domain_hdr(cpu, &mon_d->hdr); + } } + if (IS_ERR(dom)) + return PTR_ERR(dom); } } @@ -2039,6 +2095,7 @@ int mpam_resctrl_online_cpu(unsigned int cpu) void mpam_resctrl_offline_cpu(unsigned int cpu) { + struct mpam_component *comp; struct mpam_resctrl_res *res; enum resctrl_res_level rid; @@ -2050,35 +2107,38 @@ void mpam_resctrl_offline_cpu(unsigned int cpu) struct rdt_l3_mon_domain *mon_d; struct rdt_ctrl_domain *ctrl_d; bool ctrl_dom_empty, mon_dom_empty; - struct rdt_resource *r = &res->resctrl_res; if (!res->class) continue; // dummy resource - dom = mpam_resctrl_get_domain_from_cpu(cpu, res); - if (WARN_ON_ONCE(!dom)) - continue; + guard(srcu)(&mpam_srcu); + list_for_each_entry_srcu(comp, &res->class->components, class_list, + srcu_read_lock_held(&mpam_srcu)) { + if (!cpumask_test_cpu(cpu, &comp->affinity)) + continue; + dom = mpam_resctrl_get_domain_from_cpu(cpu, res, comp); + if (WARN_ON_ONCE(!dom)) + continue; - if (r->alloc_capable) { - ctrl_d = &dom->resctrl_ctrl_dom; - ctrl_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &ctrl_d->hdr); - if (ctrl_dom_empty) - resctrl_offline_ctrl_domain(&res->resctrl_res, ctrl_d); - } else { ctrl_dom_empty = true; - } + if (resctrl_arch_alloc_capable()) { + ctrl_d = &dom->resctrl_ctrl_dom; + ctrl_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &ctrl_d->hdr); + if (ctrl_dom_empty) + resctrl_offline_ctrl_domain(&res->resctrl_res, ctrl_d); + } - if (r->mon_capable) { - mon_d = &dom->resctrl_mon_dom; - mon_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &mon_d->hdr); - if (mon_dom_empty) - resctrl_offline_mon_domain(&res->resctrl_res, &mon_d->hdr); - } else { mon_dom_empty = true; - } + if (resctrl_arch_mon_capable()) { + mon_d = &dom->resctrl_mon_dom; + mon_dom_empty = mpam_resctrl_offline_domain_hdr(cpu, &mon_d->hdr); + if (mon_dom_empty) + resctrl_offline_mon_domain(&res->resctrl_res, &mon_d->hdr); + } - if (ctrl_dom_empty && mon_dom_empty) - kfree(dom); + if (ctrl_dom_empty && mon_dom_empty) + kfree(dom); + } } } diff --git a/fs/resctrl/internal.h b/fs/resctrl/internal.h index 3c02cab6c94e8..2e27efe08c27d 100644 --- a/fs/resctrl/internal.h +++ b/fs/resctrl/internal.h @@ -386,9 +386,9 @@ int alloc_rmid(u32 closid); void free_rmid(u32 closid, u32 rmid); -int resctrl_l3_mon_resource_init(void); +int resctrl_mon_init(void); -void resctrl_l3_mon_resource_exit(void); +void resctrl_mon_exit(void); void mon_event_count(void *info); @@ -475,6 +475,11 @@ ssize_t resctrl_io_alloc_cbm_write(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off); u32 resctrl_io_alloc_closid(struct rdt_resource *r); +int mbm_MB_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v); + +ssize_t mbm_MB_assignments_write(struct kernfs_open_file *of, char *buf, size_t nbytes, + loff_t off); + #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp); diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c index 94da0360952d5..b664fb16b4c68 100644 --- a/fs/resctrl/monitor.c +++ b/fs/resctrl/monitor.c @@ -450,7 +450,7 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) struct mbm_state *m; u64 tval = 0; - if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) { + if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, rr->r->rid)) { rr->err = -EIO; return -EINVAL; } @@ -547,6 +547,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) { switch (rr->r->rid) { case RDT_RESOURCE_L3: + case RDT_RESOURCE_MBA: WARN_ON_ONCE(rr->evt->any_cpu); if (rr->hdr) return __l3_mon_event_count(rdtgrp, rr); @@ -592,7 +593,7 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) struct rdt_l3_mon_domain *d; struct mbm_state *m; - if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) + if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, rr->r->rid)) return; d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr); m = get_mbm_state(d, closid, rmid, rr->evt->evtid); @@ -1002,7 +1003,7 @@ void free_rmid_lru_list(void) */ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = { MON_EVENT(QOS_L3_OCCUP_EVENT_ID, "llc_occupancy", RDT_RESOURCE_L3, false), - MON_EVENT(QOS_L3_MBM_TOTAL_EVENT_ID, "mbm_total_bytes", RDT_RESOURCE_L3, false), + MON_EVENT(QOS_L3_MBM_TOTAL_EVENT_ID, "mbm_total_bytes", RDT_RESOURCE_MBA, false), MON_EVENT(QOS_L3_MBM_LOCAL_EVENT_ID, "mbm_local_bytes", RDT_RESOURCE_L3, false), MON_EVENT(PMT_EVENT_ENERGY, "core_energy", RDT_RESOURCE_PERF_PKG, true), MON_EVENT(PMT_EVENT_ACTIVITY, "activity", RDT_RESOURCE_PERF_PKG, true), @@ -1631,9 +1632,9 @@ int resctrl_available_mbm_cntrs_show(struct kernfs_open_file *of, return ret; } -int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v) +static int mbm_assignments_show(struct kernfs_open_file *of, struct seq_file *s, + void *v, struct rdt_resource *r) { - struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3); struct rdt_l3_mon_domain *d; struct rdtgroup *rdtgrp; struct mon_evt *mevt; @@ -1679,6 +1680,18 @@ int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, voi return ret; } +int mbm_L3_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v) +{ + return mbm_assignments_show(of, s, v, + resctrl_arch_get_resource(RDT_RESOURCE_L3)); +} + +int mbm_MB_assignments_show(struct kernfs_open_file *of, struct seq_file *s, void *v) +{ + return mbm_assignments_show(of, s, v, + resctrl_arch_get_resource(RDT_RESOURCE_MBA)); +} + /* * mbm_get_mon_event_by_name() - Return the mon_evt entry for the matching * event name. @@ -1773,10 +1786,10 @@ static int resctrl_parse_mbm_assignment(struct rdt_resource *r, struct rdtgroup return -EINVAL; } -ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, - size_t nbytes, loff_t off) +static ssize_t mbm_assignments_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off, + struct rdt_resource *r) { - struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3); struct rdtgroup *rdtgrp; char *token, *event; int ret = 0; @@ -1818,6 +1831,20 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, return ret ?: nbytes; } +ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + return mbm_assignments_write(of, buf, nbytes, off, + resctrl_arch_get_resource(RDT_RESOURCE_L3)); +} + +ssize_t mbm_MB_assignments_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + return mbm_assignments_write(of, buf, nbytes, off, + resctrl_arch_get_resource(RDT_RESOURCE_MBA)); +} + static int closid_num_dirty_rmid_alloc(struct rdt_resource *r) { if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) { @@ -1858,7 +1885,7 @@ static void closid_num_dirty_rmid_free(void) } /** - * resctrl_l3_mon_resource_init() - Initialise global monitoring structures. + * resctrl_mon_resource_init() - Initialise global monitoring structures. * * Allocate and initialise global monitor resources that do not belong to a * specific domain. i.e. the closid_num_dirty_rmid[] used to find the CLOSID @@ -1870,27 +1897,21 @@ static void closid_num_dirty_rmid_free(void) * * Return: 0 for success, or -ENOMEM. */ -int resctrl_l3_mon_resource_init(void) +static void resctrl_mon_resource_init(struct rdt_resource *r) { - struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3); - int ret; + unsigned long fflags; - if (!r->mon_capable) - return 0; - - ret = closid_num_dirty_rmid_alloc(r); - if (ret) - return ret; + fflags = (r->rid == RDT_RESOURCE_MBA) ? RFTYPE_RES_MB :RFTYPE_RES_CACHE; if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) { mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID].configurable = true; resctrl_file_fflags_init("mbm_total_bytes_config", - RFTYPE_MON_INFO | RFTYPE_RES_CACHE); + RFTYPE_MON_INFO | fflags); } if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_LOCAL_EVENT_ID)) { mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID].configurable = true; resctrl_file_fflags_init("mbm_local_bytes_config", - RFTYPE_MON_INFO | RFTYPE_RES_CACHE); + RFTYPE_MON_INFO | fflags); } if (resctrl_is_mon_event_enabled(QOS_L3_MBM_LOCAL_EVENT_ID)) @@ -1908,19 +1929,43 @@ int resctrl_l3_mon_resource_init(void) NON_TEMP_WRITE_TO_LOCAL_MEM); r->mon.mbm_assign_on_mkdir = true; resctrl_file_fflags_init("num_mbm_cntrs", - RFTYPE_MON_INFO | RFTYPE_RES_CACHE); + RFTYPE_MON_INFO | fflags); resctrl_file_fflags_init("available_mbm_cntrs", - RFTYPE_MON_INFO | RFTYPE_RES_CACHE); + RFTYPE_MON_INFO | fflags); resctrl_file_fflags_init("event_filter", RFTYPE_ASSIGN_CONFIG); resctrl_file_fflags_init("mbm_assign_on_mkdir", RFTYPE_MON_INFO | - RFTYPE_RES_CACHE); - resctrl_file_fflags_init("mbm_L3_assignments", RFTYPE_MON_BASE); + fflags); + if (r->rid == RDT_RESOURCE_MBA) + resctrl_file_fflags_init("mbm_MB_assignments", RFTYPE_MON_BASE); + else + resctrl_file_fflags_init("mbm_L3_assignments", RFTYPE_MON_BASE); + resctrl_file_fflags_init("mbm_assign_mode", RFTYPE_MON_INFO | + fflags); } +} + +int resctrl_mon_init(void) +{ + struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3); + int ret; + + if (!r->mon_capable) + return 0; + + ret = closid_num_dirty_rmid_alloc(r); + if (ret) + return ret; + + resctrl_mon_resource_init(r); + + r = resctrl_arch_get_resource(RDT_RESOURCE_MBA); + if (r) + resctrl_mon_resource_init(r); return 0; } -void resctrl_l3_mon_resource_exit(void) +void resctrl_mon_exit(void) { struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3); diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index a749e36ca0971..83bb40e686064 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2188,6 +2188,13 @@ static struct rftype res_common_files[] = { .seq_show = mbm_L3_assignments_show, .write = mbm_L3_assignments_write, }, + { + .name = "mbm_MB_assignments", + .mode = 0644, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = mbm_MB_assignments_show, + .write = mbm_MB_assignments_write, + }, { .name = "mbm_assign_mode", .mode = 0644, @@ -4592,10 +4599,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h if (resctrl_mounted && resctrl_arch_mon_capable()) rmdir_mondata_subdir_allrdtgrp(r, hdr); - if (r->rid != RDT_RESOURCE_L3) - goto out_unlock; - - if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) + if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid)) goto out_unlock; d = container_of(hdr, struct rdt_l3_mon_domain, hdr); @@ -4701,10 +4705,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr mutex_lock(&rdtgroup_mutex); - if (r->rid != RDT_RESOURCE_L3) - goto mkdir; - - if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3)) + if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid)) goto out_unlock; d = container_of(hdr, struct rdt_l3_mon_domain, hdr); @@ -4721,7 +4722,6 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID)) INIT_DELAYED_WORK(&d->cqm_limbo, cqm_handle_limbo); -mkdir: err = 0; /* * If the filesystem is not mounted then only the default resource group @@ -4827,13 +4827,13 @@ int resctrl_init(void) io_alloc_init(); - ret = resctrl_l3_mon_resource_init(); + ret = resctrl_mon_init(); if (ret) return ret; ret = sysfs_create_mount_point(fs_kobj, "resctrl"); if (ret) { - resctrl_l3_mon_resource_exit(); + resctrl_mon_exit(); return ret; } @@ -4868,7 +4868,7 @@ int resctrl_init(void) cleanup_mountpoint: sysfs_remove_mount_point(fs_kobj, "resctrl"); - resctrl_l3_mon_resource_exit(); + resctrl_mon_exit(); return ret; } @@ -4931,6 +4931,6 @@ void resctrl_exit(void) * it can be used to umount resctrl. */ - resctrl_l3_mon_resource_exit(); + resctrl_mon_exit(); free_rmid_lru_list(); } From 39c973a9670aea9de6b8b0118b9c9b61f525c32a Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Wed, 4 Mar 2026 11:53:32 -0600 Subject: [PATCH 375/464] NVIDIA: SAUCE: resctrl/mpam: reset RIS by applying explicit default config BugLink: https://bugs.launchpad.net/bugs/2154527 Reset an RIS by building a default mpam_config and applying it via mpam_reprogram_ris_partid(), like any other config. - mpam_init_reset_cfg(): set features and default values only for controls supported by the RIS (cpor_part, mbw_part, mbw_max, mbw_prop, cmax_cmax, cmax_cmin). Use full masks for CPBM/MBW_PBM and MPAMCFG_* defaults for MBW_MAX, CMAX, CMIN. - mpam_reprogram_ris_partid(): apply cfg for all supported controls (no separate reset path). Signed-off-by: Shanker Donthineni (forward ported from commit e0b6de09b2a78f7aa12400ee756e5e6564118578 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - reset_cpbm and reset_mbw_pbm are not used. no need to define them; - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; - Since mpam_init_reset_cfg() has been removed from upstream, reset_cfg is init in mpam_reprogram_ris_partid(&reset_cfg); ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 2 ++ drivers/resctrl/mpam_resctrl.c | 18 ++---------------- 2 files changed, 4 insertions(+), 16 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 91d27a7dc2f86..9f1818a5b6e90 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -2685,6 +2685,8 @@ static void mpam_reset_component_cfg(struct mpam_component *comp) comp->cfg[i].mbw_pbm = GENMASK(cprops->mbw_pbm_bits - 1, 0); if (cprops->bwa_wd) comp->cfg[i].mbw_max = GENMASK(15, 16 - cprops->bwa_wd); + if (cprops->cmax_wd) + comp->cfg[i].cmax = MPAMCFG_CMAX_CMAX; } } diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index df53406cbe3c5..42b139bd58159 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -893,14 +893,7 @@ static u32 get_mba_granularity(struct mpam_props *cprops) */ static u32 fract16_to_percent(u16 fract, u8 wd) { - u32 val = fract; - - val >>= 16 - wd; - val += 1; - val *= MAX_MBA_BW; - val = DIV_ROUND_CLOSEST(val, 1 << wd); - - return val; + return DIV_ROUND_CLOSEST((fract + 1) * 100, 65536); } /* @@ -915,14 +908,7 @@ static u32 fract16_to_percent(u16 fract, u8 wd) */ static u16 percent_to_fract16(u8 pc, u8 wd) { - u32 val = pc; - - val <<= wd; - val = DIV_ROUND_CLOSEST(val, MAX_MBA_BW); - val = max(val, 1) - 1; - val <<= 16 - wd; - - return val; + return pc ? (((pc * 65536) / 100) - 1) : 0; } static u32 mbw_max_to_percent(u16 mbw_max, struct mpam_props *cprops) From 07a69fd2432ddefb910a91d39ec0bd7c6e107631 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Tue, 6 May 2025 21:02:21 -0500 Subject: [PATCH 376/464] NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Fix MPAM for indentity-mappings BugLink: https://bugs.launchpad.net/bugs/2154527 There is no struct arm_smmu_domain context for domains configured with identity mappings. Use the device to obtain the necessary information to program PARTID and PMGID. Signed-off-by: Shanker Donthineni (cherry picked from commit 23bcbda8e63535e6090ea00540e19742650edf59 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 73 +++++++-------------- drivers/iommu/iommu.c | 52 ++++++--------- include/linux/iommu.h | 4 +- 3 files changed, 45 insertions(+), 84 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 9f6154bed59c6..c913fb55de0c7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3819,94 +3819,67 @@ static int arm_smmu_def_domain_type(struct device *dev) return 0; } -static int arm_smmu_group_set_mpam(struct iommu_group *group, u16 partid, +static int arm_smmu_group_set_mpam(struct device *dev, u16 partid, u8 pmg) { int i; u32 sid; - unsigned long flags; struct arm_smmu_ste *step; - struct iommu_domain *domain; struct arm_smmu_device *smmu; struct arm_smmu_master *master; struct arm_smmu_cmdq_batch cmds; - struct arm_smmu_domain *smmu_domain; struct arm_smmu_cmdq_ent cmd = { .opcode = CMDQ_OP_CFGI_STE, .cfgi = { .leaf = true, }, }; - struct arm_smmu_master_domain *master_domain; - domain = iommu_get_domain_for_group(group); - smmu_domain = to_smmu_domain(domain); - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_MPAM)) + master = dev_iommu_priv_get(dev); + if (!(master->smmu->features & ARM_SMMU_FEAT_MPAM)) return -EIO; - smmu = smmu_domain->smmu; + smmu = master->smmu; arm_smmu_cmdq_batch_init(smmu, &cmds, &cmd); - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master_domain, &smmu_domain->devices, - devices_elm) { - master = master_domain->master; - - for (i = 0; i < master->num_streams; i++) { - sid = master->streams[i].id; - step = arm_smmu_get_step_for_sid(smmu, sid); - - /* These need locking if the VMSPtr is ever used */ - step->data[4] = FIELD_PREP(STRTAB_STE_4_PARTID, partid); - step->data[5] = FIELD_PREP(STRTAB_STE_5_PMG, pmg); + for (i = 0; i < master->num_streams; i++) { + sid = master->streams[i].id; + step = arm_smmu_get_step_for_sid(smmu, sid); - cmd.cfgi.sid = sid; - arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); - } + /* These need locking if the VMSPtr is ever used */ + step->data[4] = FIELD_PREP(STRTAB_STE_4_PARTID, partid); + step->data[5] = FIELD_PREP(STRTAB_STE_5_PMG, pmg); - master->partid = partid; - master->pmg = pmg; + cmd.cfgi.sid = sid; + arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + + master->partid = partid; + master->pmg = pmg; arm_smmu_cmdq_batch_submit(smmu, &cmds); return 0; } -static int arm_smmu_group_get_mpam(struct iommu_group *group, u16 *partid, +static int arm_smmu_group_get_mpam(struct device *dev, u16 *partid, u8 *pmg) { - int err = -EINVAL; - unsigned long flags; - struct iommu_domain *domain; struct arm_smmu_master *master; - struct arm_smmu_domain *smmu_domain; - struct arm_smmu_master_domain *master_domain; - domain = iommu_get_domain_for_group(group); - smmu_domain = to_smmu_domain(domain); - if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_MPAM)) + master = dev_iommu_priv_get(dev); + if (!(master->smmu->features & ARM_SMMU_FEAT_MPAM)) return -EIO; if (!partid && !pmg) return 0; - spin_lock_irqsave(&smmu_domain->devices_lock, flags); - list_for_each_entry(master_domain, &smmu_domain->devices, - devices_elm) { - master = master_domain->master; - if (master) { - if (partid) - *partid = master->partid; - if (pmg) - *pmg = master->pmg; - err = 0; - } - } - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); + if (partid) + *partid = master->partid; + if (pmg) + *pmg = master->pmg; - return err; + return 0; } static const struct iommu_ops arm_smmu_ops = { diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 817d78c8568cb..877eaef1ca7f5 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -4274,25 +4274,19 @@ int iommu_group_set_qos_params(struct iommu_group *group, { const struct iommu_ops *ops; struct group_device *device; - int ret; + int ret = -ENODEV; mutex_lock(&group->mutex); - device = list_first_entry_or_null(&group->devices, typeof(*device), - list); - if (!device) { - ret = -ENODEV; - goto out_unlock; - } - - ops = dev_iommu_ops(device->dev); - if (!ops->set_group_qos_params) { - ret = -EOPNOTSUPP; - goto out_unlock; + for_each_group_device(group, device) { + ops = dev_iommu_ops(device->dev); + if (!ops->set_group_qos_params) { + ret = -EOPNOTSUPP; + break; + } + ret = ops->set_group_qos_params(device->dev, partition, perf_mon_grp); + if (ret < 0) + break; } - - ret = ops->set_group_qos_params(group, partition, perf_mon_grp); - -out_unlock: mutex_unlock(&group->mutex); return ret; @@ -4312,25 +4306,19 @@ int iommu_group_get_qos_params(struct iommu_group *group, { const struct iommu_ops *ops; struct group_device *device; - int ret; + int ret = -ENODEV; mutex_lock(&group->mutex); - device = list_first_entry_or_null(&group->devices, typeof(*device), - list); - if (!device) { - ret = -ENODEV; - goto out_unlock; - } - - ops = dev_iommu_ops(device->dev); - if (!ops->get_group_qos_params) { - ret = -EOPNOTSUPP; - goto out_unlock; + for_each_group_device(group, device) { + ops = dev_iommu_ops(device->dev); + if (!ops->get_group_qos_params) { + ret = -EOPNOTSUPP; + break; + } + ret = ops->get_group_qos_params(device->dev, partition, perf_mon_grp); + if (!ret) + break; } - - ret = ops->get_group_qos_params(group, partition, perf_mon_grp); - -out_unlock: mutex_unlock(&group->mutex); return ret; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index a9f2a03439495..4e61d8cef35a6 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -705,9 +705,9 @@ struct iommu_ops { const struct iommu_user_data *user_data); /* Per group IOMMU features */ - int (*get_group_qos_params)(struct iommu_group *group, u16 *partition, + int (*get_group_qos_params)(struct device *dev, u16 *partition, u8 *perf_mon_grp); - int (*set_group_qos_params)(struct iommu_group *group, u16 partition, + int (*set_group_qos_params)(struct device *dev, u16 partition, u8 perf_mon_grp); const struct iommu_domain_ops *default_domain_ops; From 7985aeb0a62adc2cf52c2c4320514567cd6bd57a Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:42 +0000 Subject: [PATCH 377/464] NVIDIA: VR: SAUCE: resctrl: add MB_HLIM resource ID and schema type BugLink: https://bugs.launchpad.net/bugs/2154527 Add RDT_RESOURCE_MB_HLIM and RESCTRL_SCHEMA_MB_HLIM for per-domain MBW maximum hard-limit control on ARM MPAM. Document the schema in kernel-doc. Extend resctrl_get_default_ctrl() for RESCTRL_SCHEMA_MB_HLIM (default 0). (cherry picked from commit 9fc8c60cb42b519bb85c37ede8d44b847b18ae40 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/resctrl.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 4bec4543ef6a5..985074fa5ddb6 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -57,6 +57,7 @@ enum resctrl_res_level { RDT_RESOURCE_PERF_PKG, RDT_RESOURCE_L3_MAX, RDT_RESOURCE_L2_MAX, + RDT_RESOURCE_MB_HLIM, /* Must be the last */ RDT_NUM_RESOURCES, @@ -289,12 +290,15 @@ enum resctrl_scope { * @RESCTRL_SCHEMA_PERCENT: The schema is a percentage. * @RESCTRL_SCHEMA_MBPS: The schema ia a MBps value. * @RESCTRL_SCHEMA__AMD_MBA: The schema value is MBA for AMD platforms. + * @RESCTRL_SCHEMA_MB_HLIM: Per-domain MBW max hard limit (0/1), ARM MPAM only + * when MPAMF_MBW_IDR.MAX_LIM is 0b00 (HARDLIM RW). */ enum resctrl_schema_fmt { RESCTRL_SCHEMA_BITMAP, RESCTRL_SCHEMA_PERCENT, RESCTRL_SCHEMA_MBPS, RESCTRL_SCHEMA__AMD_MBA, + RESCTRL_SCHEMA_MB_HLIM, }; /** @@ -429,6 +433,8 @@ static inline u32 resctrl_get_resource_default_ctrl(struct rdt_resource *r) case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: return r->membw.max_bw; + case RESCTRL_SCHEMA_MB_HLIM: + return 0; } return WARN_ON_ONCE(1); From 19e21a6c11d69c6c6811cd06382fb3a3fbf40507 Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:42 +0000 Subject: [PATCH 378/464] NVIDIA: VR: SAUCE: resctrl: wire MB_HLIM schemata parsing and group setup BugLink: https://bugs.launchpad.net/bugs/2154527 Add mb_hlim parsing and validation (0/1), wire RESCTRL_SCHEMA_MB_HLIM into schemata and control-group setup, align MB_HLIM with MBA for exclusive mode and pseudo-locking, and default new groups to hardlim off. (cherry picked from commit 6911e81a21d23c2622c9dbf6e7b374d67dd4e672 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/ctrlmondata.c | 33 ++++++++++++++++++++++++++++++++- fs/resctrl/rdtgroup.c | 23 ++++++++++++++++++++++- include/linux/resctrl.h | 2 ++ 3 files changed, 56 insertions(+), 2 deletions(-) diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c index 8d00aeacc337c..138321730f73b 100644 --- a/fs/resctrl/ctrlmondata.c +++ b/fs/resctrl/ctrlmondata.c @@ -90,6 +90,33 @@ static int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s, return 0; } +static bool hlim_validate(char *buf, u32 *data) +{ + int ret = kstrtou32(buf, 10, data); + + if (ret || (*data != 0 && *data != 1)) { + rdt_last_cmd_printf("Invalid MB_HLIM value %s (expect 0 or 1)\n", buf); + return false; + } + return true; +} + +static int parse_mb_hlim(struct rdt_parse_data *data, struct resctrl_schema *s, + struct rdt_ctrl_domain *d) +{ + struct resctrl_staged_config *cfg; + u32 v; + + if (!hlim_validate(data->buf, &v)) + return -EINVAL; + + cfg = &d->staged_config[s->conf_type]; + cfg->new_ctrl = v; + cfg->have_new_ctrl = true; + + return 0; +} + /* * Check whether a cache bit mask is valid. * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID: @@ -224,13 +251,17 @@ static int parse_line(char *line, struct resctrl_schema *s, case RESCTRL_SCHEMA__AMD_MBA: parse_ctrlval = &parse_bw; break; + case RESCTRL_SCHEMA_MB_HLIM: + parse_ctrlval = &parse_mb_hlim; + break; } if (WARN_ON_ONCE(!parse_ctrlval)) return -EINVAL; if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP && - (r->rid == RDT_RESOURCE_MBA || r->rid == RDT_RESOURCE_SMBA)) { + (r->rid == RDT_RESOURCE_MBA || r->rid == RDT_RESOURCE_SMBA || + r->rid == RDT_RESOURCE_MB_HLIM)) { rdt_last_cmd_puts("Cannot pseudo-lock MBA resource\n"); return -EINVAL; } diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 83bb40e686064..151fb69f68d32 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1514,7 +1514,8 @@ static bool rdtgroup_mode_test_exclusive(struct rdtgroup *rdtgrp) list_for_each_entry(s, &resctrl_schema_all, list) { r = s->res; - if (r->rid == RDT_RESOURCE_MBA || r->rid == RDT_RESOURCE_SMBA) + if (r->rid == RDT_RESOURCE_MBA || r->rid == RDT_RESOURCE_SMBA || + r->rid == RDT_RESOURCE_MB_HLIM) continue; has_cache = true; list_for_each_entry(d, &r->ctrl_domains, hdr.list) { @@ -1811,6 +1812,9 @@ static int resctrl_schema_format_show(struct kernfs_open_file *of, case RESCTRL_SCHEMA__AMD_MBA: seq_puts(seq, "platform\n"); break; + case RESCTRL_SCHEMA_MB_HLIM: + seq_puts(seq, "0/1\n"); + break; } return 0; @@ -2595,6 +2599,7 @@ static u32 fflags_from_schema(struct resctrl_schema *s) fflags |= RFTYPE_SCHEMA_MBPS; break; case RESCTRL_SCHEMA__AMD_MBA: + case RESCTRL_SCHEMA_MB_HLIM: /* No standard files are exposed */ break; } @@ -2948,6 +2953,7 @@ static int schemata_list_add(struct rdt_resource *r, enum resctrl_conf_type type case RESCTRL_SCHEMA_PERCENT: case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: + case RESCTRL_SCHEMA_MB_HLIM: s->fmt_str = "%d=%u"; break; } @@ -3908,6 +3914,19 @@ static void rdtgroup_init_mba(struct rdt_resource *r, u32 closid) } } +/* Initialize MB_HLIM resource with default hardlim off (0). */ +static void rdtgroup_init_mb_hlim(struct resctrl_schema *s) +{ + struct resctrl_staged_config *cfg; + struct rdt_ctrl_domain *d; + + list_for_each_entry(d, &s->res->ctrl_domains, hdr.list) { + cfg = &d->staged_config[s->conf_type]; + cfg->new_ctrl = 0; + cfg->have_new_ctrl = true; + } +} + /* Initialize the RDT group's allocations. */ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) { @@ -3924,6 +3943,8 @@ static int rdtgroup_init_alloc(struct rdtgroup *rdtgrp) rdtgroup_init_mba(r, rdtgrp->closid); if (is_mba_sc(r)) continue; + } else if (r->rid == RDT_RESOURCE_MB_HLIM) { + rdtgroup_init_mb_hlim(s); } else { ret = rdtgroup_init_cat(s, rdtgrp->closid); if (ret < 0) diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 985074fa5ddb6..46a6f2c16deb4 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -454,6 +454,8 @@ static inline u32 resctrl_get_schema_default_ctrl(struct resctrl_schema *s) case RESCTRL_SCHEMA_MBPS: case RESCTRL_SCHEMA__AMD_MBA: return s->membw.max_bw; + case RESCTRL_SCHEMA_MB_HLIM: + return 0; } return WARN_ON_ONCE(1); From 3f89f038d12e6f947b78c60ff826d07824a7a130 Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:48 +0000 Subject: [PATCH 379/464] NVIDIA: VR: SAUCE: resctrl/mpam: probe MPAMF_MBW_IDR MAX_LIM and hardlim_rw BugLink: https://bugs.launchpad.net/bugs/2154527 Read mbw_max_lim from MPAMF_MBW_IDR.MAX_LIM when MBW_MAX is present, derive mpam_feat_mbw_max_hardlim_rw when both soft and hard limiting are supported, and merge mbw_max_lim and the feature across MSCs. Add mpam_props_sync_mbw_max_hardlim_rw() and propagate merged state in __props_mismatch(). (cherry picked from commit ced4d460ce56aa8b18d8ab3f19eeb76beeb25322 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 40 ++++++++++++++++++++++++++++++++- drivers/resctrl/mpam_internal.h | 7 ++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 9f1818a5b6e90..8f33c287a95c8 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -795,6 +795,19 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) return (!can_set || !can_clear); } +/* Align mpam_feat_mbw_max_hardlim_rw with MPAMF_MBW_IDR.MAX_LIM and mbw_max. */ +static void mpam_props_sync_mbw_max_hardlim_rw(struct mpam_props *props) +{ + if (!mpam_has_feature(mpam_feat_mbw_max, props)) { + mpam_clear_feature(mpam_feat_mbw_max_hardlim_rw, props); + return; + } + if (props->mbw_max_lim == 0) + mpam_set_feature(mpam_feat_mbw_max_hardlim_rw, props); + else + mpam_clear_feature(mpam_feat_mbw_max_hardlim_rw, props); +} + static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) { int err; @@ -842,6 +855,8 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) { u32 mbw_features = mpam_read_partsel_reg(msc, MBW_IDR); + props->mbw_max_lim = 0; + /* portion bitmap resolution */ props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features); if (props->mbw_pbm_bits && @@ -856,14 +871,18 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris) */ props->bwa_wd = min(props->bwa_wd, 16); - if (props->bwa_wd && FIELD_GET(MPAMF_MBW_IDR_HAS_MAX, mbw_features)) + if (props->bwa_wd && FIELD_GET(MPAMF_MBW_IDR_HAS_MAX, mbw_features)) { mpam_set_feature(mpam_feat_mbw_max, props); + props->mbw_max_lim = FIELD_GET(MPAMF_MBW_IDR_MAX_LIM, mbw_features); + } if (props->bwa_wd && FIELD_GET(MPAMF_MBW_IDR_HAS_MIN, mbw_features)) mpam_set_feature(mpam_feat_mbw_min, props); if (props->bwa_wd && FIELD_GET(MPAMF_MBW_IDR_HAS_PROP, mbw_features)) mpam_set_feature(mpam_feat_mbw_prop, props); + + mpam_props_sync_mbw_max_hardlim_rw(props); } /* Priority partitioning */ @@ -2255,12 +2274,31 @@ static void __props_mismatch(struct mpam_props *parent, if (alias && !mpam_has_bwa_wd_feature(parent) && mpam_has_bwa_wd_feature(child)) { parent->bwa_wd = child->bwa_wd; + parent->mbw_max_lim = child->mbw_max_lim; + if (mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, child)) + mpam_set_feature(mpam_feat_mbw_max_hardlim_rw, parent); + else + mpam_clear_feature(mpam_feat_mbw_max_hardlim_rw, parent); } else if (MISMATCHED_HELPER(parent, child, mpam_has_bwa_wd_feature, bwa_wd, alias)) { pr_debug("took the min bwa_wd\n"); parent->bwa_wd = min(parent->bwa_wd, child->bwa_wd); } + if (CAN_MERGE_FEAT(parent, child, mpam_feat_mbw_max, alias)) { + parent->mbw_max_lim = child->mbw_max_lim; + if (mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, child)) + mpam_set_feature(mpam_feat_mbw_max_hardlim_rw, parent); + else + mpam_clear_feature(mpam_feat_mbw_max_hardlim_rw, parent); + } else if (MISMATCHED_FEAT(parent, child, mpam_feat_mbw_max, + mbw_max_lim, alias)) { + pr_debug("%s mbw_max_lim mismatch, clearing mbw_max\n", __func__); + mpam_clear_feature(mpam_feat_mbw_max, parent); + parent->mbw_max_lim = 0; + mpam_props_sync_mbw_max_hardlim_rw(parent); + } + if (alias && !mpam_has_cmax_wd_feature(parent) && mpam_has_cmax_wd_feature(child)) { parent->cmax_wd = child->cmax_wd; } else if (MISMATCHED_HELPER(parent, child, mpam_has_cmax_wd_feature, diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index a08901adc4e09..fd7531afd21c4 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -206,6 +206,7 @@ enum mpam_device_features { mpam_feat_mbw_part, mpam_feat_mbw_min, mpam_feat_mbw_max, + mpam_feat_mbw_max_hardlim_rw, mpam_feat_mbw_prop, mpam_feat_intpri_part, mpam_feat_intpri_part_0_low, @@ -237,6 +238,11 @@ struct mpam_props { u16 dspri_wd; u16 num_csu_mon; u16 num_mbwu_mon; + /* + * MPAMF_MBW_IDR.MAX_LIM [1:0] when mpam_feat_mbw_max; else 0. + * 0 = soft+hard, 1 = soft only, 2 = hard only, 3 = reserved. + */ + u8 mbw_max_lim; /* * Kunit tests use memset() to set up feature combinations that should be @@ -628,6 +634,7 @@ static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { } /* MPAMF_MBW_IDR - MPAM features memory bandwidth partitioning ID register */ #define MPAMF_MBW_IDR_BWA_WD GENMASK(5, 0) +#define MPAMF_MBW_IDR_MAX_LIM GENMASK(9, 8) #define MPAMF_MBW_IDR_HAS_MIN BIT(10) #define MPAMF_MBW_IDR_HAS_MAX BIT(11) #define MPAMF_MBW_IDR_HAS_PBM BIT(12) From 665df640ed8c98f6d8ab043ab390f148d1bddfdd Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:42 +0000 Subject: [PATCH 380/464] NVIDIA: VR: SAUCE: resctrl/mpam: track MBW max hard-limit in config BugLink: https://bugs.launchpad.net/bugs/2154527 Add mbw_max_hardlim to mpam_config. When reprogramming MBW_MAX, OR in HARDLIM (MPAMCFG_MBW_MAX bit 31) when either MBW_MAX or HARDLIM_RW features are active. Merge HARDLIM in mpam_update_config(). (forward ported from commit 114894b316e4f01d12979c66f58177b5af1b144d https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_devices.c`; - Resolve minor conflicts in `drivers/resctrl/mpam_internal.h`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_devices.c | 15 ++++++++++++--- drivers/resctrl/mpam_internal.h | 1 + 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c index 8f33c287a95c8..58d5b2f10eeb1 100644 --- a/drivers/resctrl/mpam_devices.c +++ b/drivers/resctrl/mpam_devices.c @@ -1638,10 +1638,17 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid, } if (mpam_has_feature(mpam_feat_mbw_max, rprops)) { - if (mpam_has_feature(mpam_feat_mbw_max, cfg)) - mpam_write_partsel_reg(msc, MBW_MAX, cfg->mbw_max); - else + if (mpam_has_feature(mpam_feat_mbw_max, cfg) || + mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, cfg)) { + u32 mbw_val = cfg->mbw_max; + + if (mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, cfg) && + cfg->mbw_max_hardlim) + mbw_val |= MPAMCFG_MBW_MAX_HARDLIM; + mpam_write_partsel_reg(msc, MBW_MAX, mbw_val); + } else { mpam_write_partsel_reg(msc, MBW_MAX, MPAMCFG_MBW_MAX_MAX); + } } if (mpam_has_feature(mpam_feat_mbw_prop, rprops)) @@ -3136,6 +3143,8 @@ static bool mpam_update_config(struct mpam_config *cfg, maybe_update_config(cfg, mpam_feat_cmax_cmax, newcfg, cmax, has_changes); maybe_update_config(cfg, mpam_feat_mbw_part, newcfg, mbw_pbm, has_changes); maybe_update_config(cfg, mpam_feat_mbw_max, newcfg, mbw_max, has_changes); + maybe_update_config(cfg, mpam_feat_mbw_max_hardlim_rw, newcfg, + mbw_max_hardlim, has_changes); return has_changes; } diff --git a/drivers/resctrl/mpam_internal.h b/drivers/resctrl/mpam_internal.h index fd7531afd21c4..d5b733ec8fe7f 100644 --- a/drivers/resctrl/mpam_internal.h +++ b/drivers/resctrl/mpam_internal.h @@ -359,6 +359,7 @@ struct mpam_config { u32 cpbm; u32 mbw_pbm; u16 mbw_max; + bool mbw_max_hardlim; u16 cmax; u16 cmin; From 21e116c6a1af39f52ef8ce57e1d64d4e6a8ac040 Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:42 +0000 Subject: [PATCH 381/464] NVIDIA: VR: SAUCE: resctrl/mpam: bind MB_HLIM resource to MBA MPAM class BugLink: https://bugs.launchpad.net/bugs/2154527 When mpam_feat_mbw_max_hardlim_rw is present, share the MBA MPAM class with RDT_RESOURCE_MB_HLIM. Wire resctrl_arch_get_config and resctrl_arch_update_one for MB_HLIM, and tear down MBA and MB_HLIM controls together on class removal. Handle RDT_RESOURCE_MB_HLIM in fflags_from_resource() so creating info/MB_HLIM at resctrl mount does not hit the unhandled-rid WARN. (forward ported from commit 04e2ea34385215db0e00716ad4553c15084382f6 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 53 ++++++++++++++++++++++++++++++++++ fs/resctrl/rdtgroup.c | 2 ++ 2 files changed, 55 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 42b139bd58159..32e3bde805c8d 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1196,6 +1196,13 @@ static void mpam_resctrl_pick_mba(void) candidate_class->level); res = &mpam_resctrl_controls[RDT_RESOURCE_MBA]; res->class = candidate_class; + if (mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, + &candidate_class->props)) { + struct mpam_resctrl_res *mbh = + &mpam_resctrl_controls[RDT_RESOURCE_MB_HLIM]; + + mbh->class = candidate_class; + } } } @@ -1487,6 +1494,14 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) r->ctrl_scope = RESCTRL_L3_CACHE; } + break; + case RDT_RESOURCE_MB_HLIM: + if (!mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, cprops)) + break; + r->alloc_capable = true; + r->schema_fmt = RESCTRL_SCHEMA_MB_HLIM; + r->ctrl_scope = RESCTRL_L3_CACHE; + r->name = "MB_HLIM"; break; case RDT_RESOURCE_MBA: r->schema_fmt = RESCTRL_SCHEMA_PERCENT; @@ -1636,6 +1651,33 @@ static int mpam_resctrl_monitor_init(struct mpam_resctrl_mon *mon, return 0; } +/* MB_HLIM schemata read: 0/1 per domain for current closid. */ +static u32 mpam_read_mbw_max_hardlim(struct rdt_resource *r, struct rdt_ctrl_domain *dom, + u32 closid, enum resctrl_conf_type type) +{ + struct mpam_resctrl_dom *m_dom; + struct mpam_config *cfg; + u32 partid; + + if (!mpam_is_enabled() || r->rid != RDT_RESOURCE_MB_HLIM) + return 0; + + partid = resctrl_get_config_index(closid, type); + if (partid >= resctrl_arch_get_num_closid(r)) + return 0; + + m_dom = container_of(dom, struct mpam_resctrl_dom, resctrl_ctrl_dom); + if (!m_dom->ctrl_comp || !m_dom->ctrl_comp->cfg) + return 0; + + cfg = &m_dom->ctrl_comp->cfg[partid]; + if (!mpam_has_feature(mpam_feat_mbw_max, cfg) && + !mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, cfg)) + return 0; + + return cfg->mbw_max_hardlim ? 1 : 0; +} + u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, u32 closid, enum resctrl_conf_type type) { @@ -1675,6 +1717,8 @@ u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d, case RDT_RESOURCE_L3_MAX: configured_by = mpam_feat_cmax_cmax; break; + case RDT_RESOURCE_MB_HLIM: + return mpam_read_mbw_max_hardlim(r, d, closid, type); case RDT_RESOURCE_MBA: if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { configured_by = mpam_feat_mbw_max; @@ -1757,6 +1801,15 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d, mpam_set_feature(mpam_feat_mbw_max, &cfg); break; } + return -EINVAL; + case RDT_RESOURCE_MB_HLIM: + if (mpam_has_feature(mpam_feat_mbw_max_hardlim_rw, cprops) && + mpam_has_feature(mpam_feat_mbw_max, cprops)) { + cfg.mbw_max_hardlim = cfg_val != 0; + mpam_set_feature(mpam_feat_mbw_max_hardlim_rw, &cfg); + mpam_set_feature(mpam_feat_mbw_max, &cfg); + break; + } fallthrough; default: return -EINVAL; diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 151fb69f68d32..628af1f29c0bd 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2573,6 +2573,8 @@ static unsigned long fflags_from_resource(struct rdt_resource *r) return RFTYPE_RES_MB; case RDT_RESOURCE_PERF_PKG: return RFTYPE_RES_PERF_PKG; + case RDT_RESOURCE_MB_HLIM: + return 0; } return 0; From b7ebb1310be97d17d1d6505df956ef81e17cd54a Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:47 +0000 Subject: [PATCH 382/464] NVIDIA: VR: SAUCE: resctrl: add membw.mb_max_lim and arch_has_mb_max_lim BugLink: https://bugs.launchpad.net/bugs/2154527 Add mb_max_lim and arch_has_mb_max_lim to struct resctrl_membw with kernel-doc: MPAM MAX_LIM encoding (MPAMF_MBW_IDR), invalid elsewhere unless arch sets arch_has_mb_max_lim. (cherry picked from commit 7939f1e41019d139cc11f2b93d6e8bd7ef4663d1 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/resctrl.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h index 46a6f2c16deb4..2287b7a3ee342 100644 --- a/include/linux/resctrl.h +++ b/include/linux/resctrl.h @@ -251,11 +251,15 @@ enum membw_throttle_mode { * @min_bw: Minimum memory bandwidth percentage user can request * @max_bw: Maximum memory bandwidth value, used as the reset value * @bw_gran: Granularity at which the memory bandwidth is allocated + * @mb_max_lim: MPAM MAX_LIM encoding (MPAMF_MBW_IDR); invalid elsewhere + * @arch_has_mb_max_lim:True if mb_max_lim is supported */ struct resctrl_membw { u32 min_bw; u32 max_bw; u32 bw_gran; + u8 mb_max_lim; + bool arch_has_mb_max_lim; }; /** From 9a44413401a8d7f8ee97d74e801ce942632451de Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:47 +0000 Subject: [PATCH 383/464] NVIDIA: VR: SAUCE: resctrl/mpam: populate MBA mb_max_lim from MPAM probe BugLink: https://bugs.launchpad.net/bugs/2154527 When mpam_feat_mbw_max is present, set membw.mb_max_lim from cprops->mbw_max_lim and membw.arch_has_mb_max_lim so generic resctrl can expose max_lim. (forward ported from commit 05b9bc18310115182b70ec063229f20d601dc32c https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `drivers/resctrl/mpam_resctrl.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 32e3bde805c8d..1b225b9a2696c 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -1512,6 +1512,11 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) r->membw.max_bw = MAX_MBA_BW; r->membw.bw_gran = get_mba_granularity(cprops); + if (mpam_has_feature(mpam_feat_mbw_max, cprops)) { + r->membw.mb_max_lim = cprops->mbw_max_lim; + r->membw.arch_has_mb_max_lim = true; + } + r->name = "MB"; r->alloc_capable = true; break; From eb2a397ffd999826e6613223eb6abed31cbdf0f8 Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:47 +0000 Subject: [PATCH 384/464] NVIDIA: VR: SAUCE: resctrl: add MBA max_lim sysfs and visibility from init BugLink: https://bugs.launchpad.net/bugs/2154527 Add rdt_mb_max_lim_show() and the max_lim entry in res_common_files[]. Add mb_max_lim_init() to call resctrl_file_fflags_init("max_lim", ...) when arch_has_mb_max_lim, and invoke it from resctrl_init() after io_alloc_init(). (forward ported from commit be13cad9cd1530470c9c7d96cb3665f7da8e1873 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) [fenghuay: - Resolve minor conflicts in `fs/resctrl/rdtgroup.c`; ] Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/resctrl/rdtgroup.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 628af1f29c0bd..ed507dda9180a 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -1303,6 +1303,17 @@ static int rdt_delay_linear_show(struct kernfs_open_file *of, return 0; } +static int rdt_mb_max_lim_show(struct kernfs_open_file *of, + struct seq_file *seq, void *v) +{ + struct resctrl_schema *s = rdt_kn_parent_priv(of->kn); + struct rdt_resource *r = s->res; + + seq_printf(seq, "%d\n", r->membw.mb_max_lim); + + return 0; +} + static int max_threshold_occ_show(struct kernfs_open_file *of, struct seq_file *seq, void *v) { @@ -2131,6 +2142,12 @@ static struct rftype res_common_files[] = { .seq_show = rdt_delay_linear_show, .fflags = RFTYPE_CTRL_INFO | RFTYPE_RES_MB, }, + { + .name = "max_lim", + .mode = 0444, + .kf_ops = &rdtgroup_kf_single_ops, + .seq_show = rdt_mb_max_lim_show, + }, /* * Platform specific which (if any) capabilities are provided by * thread_throttle_mode. Defer "fflags" initialization to platform @@ -2379,6 +2396,17 @@ static void io_alloc_init(void) } } +/* The resctrl file "max_lim" is added using MB resource if visible. */ +static void mb_max_lim_init(void) +{ + struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_MBA); + + if (!r->membw.arch_has_mb_max_lim) + return; + + resctrl_file_fflags_init("max_lim", RFTYPE_CTRL_INFO | RFTYPE_RES_MB); +} + void resctrl_file_fflags_init(const char *config, unsigned long fflags) { struct rftype *rft; @@ -4850,6 +4878,8 @@ int resctrl_init(void) io_alloc_init(); + mb_max_lim_init(); + ret = resctrl_mon_init(); if (ret) return ret; From 177f1f9aabbd2180025043646005a5b16bd43f76 Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Mon, 6 Apr 2026 01:34:48 +0000 Subject: [PATCH 385/464] NVIDIA: VR: SAUCE: Documentation: resctrl: document max_lim and MB_HLIM for MPAM MBA BugLink: https://bugs.launchpad.net/bugs/2154527 Document the MBA max_lim sysfs file, MB_HLIM schemata (0/1 per domain), and how they relate to MPAM MBW_MAX, HARDLIM, and MPAMF_MBW_IDR.MAX_LIM. Add schema_format for mb_hlim under the MB allocation info directory. max_lim is exposed as a single decimal integer (MPAMF_MBW_IDR.MAX_LIM [1:0], 0-3), matching rdt_mb_max_lim_show(). MB_HLIM appears when the probe treats HARDLIM as read/write, which this series ties to max_lim reading zero (see mpam_props_sync_mbw_max_hardlim_rw()). (cherry picked from commit 93e1b6a2b3729e1e2db818036829c498378b4f16 https://github.com/NVIDIA/NV-Kernels 24.04_linux-nvidia-6.17-next) Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/filesystems/resctrl.rst | 34 +++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst index e9ff59c2e57e8..26739383c0156 100644 --- a/Documentation/filesystems/resctrl.rst +++ b/Documentation/filesystems/resctrl.rst @@ -247,6 +247,19 @@ with respect to allocation: non-linear. This field is purely informational only. +"max_lim": + Read-only. On ARM MPAM systems where MBA exposes MBW_MAX, this + file contains a single decimal integer: the + ``MPAMF_MBW_IDR.MAX_LIM`` field [1:0] (values ``0``–``3``) as probed for the MBA resource. + The file appears only when the platform supports MBA MBW_MAX and + the MAX_LIM value is available; otherwise it is not listed. + + The Arm MPAM architecture defines the meaning of each MAX_LIM + encoding. In this kernel, when ``max_lim`` reads ``0``, the + driver treats the ``HARDLIM`` bit of ``MPAMCFG_MBW_MAX`` as + read/write and an optional ``MB_HLIM`` line may appear in + ``schemata``. When ``max_lim`` is nonzero, ``MB_HLIM`` is omitted. + "thread_throttle_mode": Indicator on Intel systems of how tasks running on threads of a physical core are throttled in cases where they @@ -963,6 +976,27 @@ Memory bandwidth domain is L3 cache. MB:=bw_MiBps0;=bw_MiBps1;... +MBW maximum hard limit (ARM MPAM) +--------------------------------- +On some ARM systems, resctrl memory bandwidth allocation uses MPAM +maximum bandwidth (MBW_MAX). When ``max_lim`` reads ``0`` (see ``max_lim`` +under the ``MB`` allocation ``info`` directory), an additional schemata +line selects the ``HARDLIM`` bit for ``MPAMCFG_MBW_MAX`` independently of +the numeric limit on the ``MB`` line. + +The line uses the same cache/domain indices as ``MB``. Each value must +be ``0`` or ``1``: ``0`` clears HARDLIM (soft-limit behaviour for the +max), ``1`` sets HARDLIM (hard limit). When ``max_lim`` is nonzero or +``MB_HLIM`` is not supported for the platform, the line is omitted from +``schemata``. + +Format: +:: + + MB_HLIM:=0|1;=0|1;... + +The corresponding ``schema_format`` entry under ``info`` is ``mb_hlim``. + Slow Memory Bandwidth Allocation (SMBA) --------------------------------------- AMD hardware supports Slow Memory Bandwidth Allocation (SMBA). From 9d97d90c12bc968a3fc3c7ba12fef992a937922e Mon Sep 17 00:00:00 2001 From: Fenghua Yu Date: Wed, 13 May 2026 23:30:03 +0000 Subject: [PATCH 386/464] NVIDIA: SAUCE: Fix mbm_L3_assign and mon_local_bytes BugLink: https://bugs.launchpad.net/bugs/2154527 Add local bytes counter in mpam_resctrl_counters[] to fix missing mbm_local_bytes monitoring on Grace. Add mon->assigned_counters check to enable mbm_L3_assignments config file on Grace. Signed-off-by: Fenghua Yu Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/resctrl/mpam_resctrl.c | 34 +++++++++++++--------------------- 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c index 1b225b9a2696c..e96df761fd3bf 100644 --- a/drivers/resctrl/mpam_resctrl.c +++ b/drivers/resctrl/mpam_resctrl.c @@ -46,7 +46,7 @@ static struct mpam_resctrl_res mpam_resctrl_controls[RDT_NUM_RESOURCES]; * to those supported by MPAM. * Class pointer may be NULL. */ -#define MPAM_MAX_EVENT QOS_L3_MBM_TOTAL_EVENT_ID +#define MPAM_MAX_EVENT QOS_L3_MBM_LOCAL_EVENT_ID static struct mpam_resctrl_mon mpam_resctrl_counters[MPAM_MAX_EVENT + 1]; #define for_each_mpam_resctrl_mon(mon, eventid) \ @@ -175,21 +175,6 @@ static void resctrl_reset_task_closids(void) read_unlock(&tasklist_lock); } -static struct mpam_resctrl_mon *mpam_resctrl_mon_from_res(struct mpam_resctrl_res *res) -{ - struct mpam_resctrl_mon *mon; - enum resctrl_event_id eventid; - - if (!res->class) - return NULL; - - for_each_mpam_resctrl_mon(mon, eventid) { - if (mon->class == res->class) - return mon; - } - return NULL; -} - static struct mpam_resctrl_res *mpam_resctrl_res_from_mon(struct mpam_resctrl_mon *mon) { struct mpam_resctrl_res *res; @@ -1423,15 +1408,22 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain * bool resctrl_arch_mbm_cntr_assign_enabled(struct rdt_resource *r) { struct mpam_resctrl_res *res; - struct mpam_resctrl_mon *mon; + enum resctrl_event_id evt; res = container_of(r, struct mpam_resctrl_res, resctrl_res); - mon = mpam_resctrl_mon_from_res(res); - if (!mon) - return false; + /* OCCUP shares the L3 class but has no MBWU assigned_counters. */ + for (evt = QOS_L3_MBM_TOTAL_EVENT_ID; evt <= QOS_L3_MBM_LOCAL_EVENT_ID; + evt++) { + struct mpam_resctrl_mon *mon = &mpam_resctrl_counters[evt]; - return mon->assigned_counters ? true : false; + if (!mon->assigned_counters) + continue; + if (mpam_resctrl_res_from_mon(mon) == res) + return true; + } + + return false; } int resctrl_arch_mbm_cntr_assign_set(struct rdt_resource *r, bool enable) From fbbf87bcd7fc03bce1de6c559d1e10411075e537 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:08:58 +0530 Subject: [PATCH 387/464] NVIDIA: VR: SAUCE: cxl: Add cxl_get_hdm_info() for HDM decoder metadata BugLink: https://bugs.launchpad.net/bugs/2152222 cxl_probe_component_regs() finds the HDM decoder block during device probe and caches its location, but does not record the decoder count and does not expose the result outside drivers/cxl/. vfio-cxl needs the decoder count and the byte offset and size of the HDM block without re-running the probe sequence. Record decoder_cnt in rmap->count when parsing the HDM capability in cxl_probe_component_regs(), extend struct cxl_reg_map with a count member, and add cxl_get_hdm_info() to return offset, size, and count from the cached map. Export under the CXL namespace; stub to -EOPNOTSUPP when CONFIG_CXL_BUS is off. Co-developed-by: Zhi Wang Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit fd317b86093e9 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [kobak: Added the target-local private drivers/cxl/cxl.h cxl_get_hdm_info() prototype because drivers/cxl/core/pci.c includes the private CXL header in addition to the public include/cxl/cxl.h declaration.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/pci.c | 29 +++++++++++++++++++++++++++++ drivers/cxl/core/regs.c | 1 + drivers/cxl/cxl.h | 2 ++ include/cxl/cxl.h | 15 +++++++++++++++ 4 files changed, 47 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 6944f0f74790b..c248bf52799bc 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -454,6 +454,35 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, } EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, "CXL"); +/** + * cxl_get_hdm_info - Get HDM decoder register block location and count + * @cxlds: CXL device state (must have component regs enumerated via + * cxl_probe_component_regs()) + * @count: number of HDM decoders in the block (from HDM Capability bits [3:0]) + * @offset: byte offset of HDM decoder block within the component register BAR + * @size: size in bytes of the HDM decoder block + * + * Return: 0 on success. -ENODEV if the HDM decoder block is not present. + */ +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u8 *count, + resource_size_t *offset, resource_size_t *size) +{ + struct cxl_reg_map *hdm = &cxlds->reg_map.component_map.hdm_decoder; + + if (WARN_ON(!count || !offset || !size)) + return -EINVAL; + + if (!hdm->valid) + return -ENODEV; + + *count = hdm->count; + *offset = hdm->offset; + *size = hdm->size; + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_get_hdm_info, "CXL"); + #define CXL_DOE_TABLE_ACCESS_REQ_CODE 0x000000ff #define CXL_DOE_TABLE_ACCESS_REQ_CODE_READ 0 #define CXL_DOE_TABLE_ACCESS_TABLE_TYPE 0x0000ff00 diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index 20c2d9fbcfe7d..e828df0629d02 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -85,6 +85,7 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, decoder_cnt = cxl_hdm_decoder_count(hdr); length = 0x20 * decoder_cnt + 0x10; rmap = &map->hdm_decoder; + rmap->count = decoder_cnt; break; } case CXL_CM_CAP_CAP_ID_RAS: diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index c7bffa399581e..e1433ff43f12f 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -722,6 +722,8 @@ int cxl_port_setup_regs(struct cxl_port *port, resource_size_t component_reg_phys); struct cxl_dev_state; +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u8 *count, + resource_size_t *offset, resource_size_t *size); int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds, struct cxl_endpoint_dvsec_info *info); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 1c496c1e846c2..15b7835d76d83 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -134,6 +134,7 @@ struct cxl_reg_map { int id; unsigned long offset; unsigned long size; + u8 count; }; struct cxl_component_reg_map { @@ -315,4 +316,18 @@ static inline bool cxl_region_contains_soft_reserve(struct resource *res) return false; } #endif +#ifdef CONFIG_CXL_BUS + +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u8 *count, + resource_size_t *offset, resource_size_t *size); + +#else + +static inline +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u8 *count, + resource_size_t *offset, resource_size_t *size) +{ return -EOPNOTSUPP; } + +#endif /* CONFIG_CXL_BUS */ + #endif /* __CXL_CXL_H__ */ From 72e8bffd8c7da77859b64dada425b17acacf1eb9 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:08:59 +0530 Subject: [PATCH 388/464] NVIDIA: VR: SAUCE: cxl: Declare cxl_probe_component_regs in public header BugLink: https://bugs.launchpad.net/bugs/2152222 vfio-cxl lives outside drivers/cxl/ but still needs to locate the component register block and fill cxl_component_reg_map. BOS already has cxl_find_regblock() in include/cxl/pci.h, but cxl_probe_component_regs() was still private to drivers/cxl/cxl.h. Declare cxl_probe_component_regs() in include/cxl/pci.h next to the existing register-block helpers so VFIO CXL can use the parsed component register map. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit e02c1b7ac02a7 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Move cxl_probe_component_regs() to include/cxl/pci.h instead of include/cxl/cxl.h to align with existing Srirangan/Alejandro convention; skip cxl_find_regblock() move as it is already in include/cxl/pci.h; add struct cxl_component_reg_map forward declaration] [kobak: Kept the target's private drivers/cxl/cxl.h declarations while adding the public include/cxl/pci.h header expected by VFIO CXL.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/cxl/pci.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/cxl/pci.h b/include/cxl/pci.h index edbf980c283f1..5c5fac2a91151 100644 --- a/include/cxl/pci.h +++ b/include/cxl/pci.h @@ -16,10 +16,13 @@ enum cxl_regloc_type { struct pci_dev; struct cxl_register_map; +struct cxl_component_reg_map; int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); +void cxl_probe_component_regs(struct device *dev, void __iomem *base, + struct cxl_component_reg_map *map); int cxl_setup_regs(struct cxl_register_map *map); #endif From 610806a7e5e9e5a890bd4ac74aa74a2f1a3a0324 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:00 +0530 Subject: [PATCH 389/464] NVIDIA: VR: SAUCE: cxl: Move component/HDM register defines to uapi/cxl/cxl_regs.h BugLink: https://bugs.launchpad.net/bugs/2152222 VFIO and other code outside the CXL core needs the same offset/mask constants the core uses for the component register block and HDM decoders. Pull them into a new include/uapi/cxl/cxl_regs.h (GPL-2.0 WITH Linux-syscall-note) and include it from include/cxl/cxl.h. Use uapi-friendly __GENMASK helpers for masks and _BITUL() for single-bit flags because UAPI headers cannot depend on kernel-internal BIT(). Section comments in the new file reference CXL spec r4.0 numbering. For UAPI change, replaced the SZ_64K with actual size as the macro will not be available for userspace programs. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 52ead24ed8ade6b664e65fbc5514147a6022263e from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Remove defines from include/cxl/cxl.h instead of drivers/cxl/cxl.h as they were already moved there by Srirangan's SAUCE commit, Add #include needed by __GENMASK() in uapi header] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/cxl/cxl.h | 43 +-------------------------- include/uapi/cxl/cxl_regs.h | 58 +++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 42 deletions(-) create mode 100644 include/uapi/cxl/cxl_regs.h diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 15b7835d76d83..716371ba38243 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -10,6 +10,7 @@ #include #include #include +#include /** * enum cxl_devtype - delineate type-2 from a generic type-3 device @@ -76,48 +77,6 @@ struct cxl_regs { #define CXL_CM_CAP_CAP_ID_HDM 0x5 #define CXL_CM_CAP_CAP_HDM_VERSION 1 -/* CXL 2.0 8.2.4 CXL Component Register Layout and Definition */ -#define CXL_COMPONENT_REG_BLOCK_SIZE SZ_64K - -/* CXL 2.0 8.2.5 CXL.cache and CXL.mem Registers */ -#define CXL_CM_OFFSET 0x1000 -#define CXL_CM_CAP_HDR_OFFSET 0x0 -#define CXL_CM_CAP_HDR_ID_MASK GENMASK(15, 0) -#define CM_CAP_HDR_CAP_ID 1 -#define CXL_CM_CAP_HDR_VERSION_MASK GENMASK(19, 16) -#define CM_CAP_HDR_CAP_VERSION 1 -#define CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK GENMASK(23, 20) -#define CM_CAP_HDR_CACHE_MEM_VERSION 1 -#define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK GENMASK(31, 24) -#define CXL_CM_CAP_PTR_MASK GENMASK(31, 20) - -/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ -#define CXL_HDM_DECODER_CAP_OFFSET 0x0 -#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0) -#define CXL_HDM_DECODER_TARGET_COUNT_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER_INTERLEAVE_11_8 BIT(8) -#define CXL_HDM_DECODER_INTERLEAVE_14_12 BIT(9) -#define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY BIT(11) -#define CXL_HDM_DECODER_INTERLEAVE_16_WAY BIT(12) -#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 -#define CXL_HDM_DECODER_ENABLE BIT(1) -#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) -#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14) -#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18) -#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c) -#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20) -#define CXL_HDM_DECODER0_CTRL_IG_MASK GENMASK(3, 0) -#define CXL_HDM_DECODER0_CTRL_IW_MASK GENMASK(7, 4) -#define CXL_HDM_DECODER0_CTRL_LOCK BIT(8) -#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) -#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) -#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11) -#define CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12) -#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24) -#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28) -#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i) -#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i) - /* HDM decoder control register constants CXL 3.0 8.2.5.19.7 */ #define CXL_DECODER_MIN_GRANULARITY 256 #define CXL_DECODER_MAX_ENCODED_IG 6 diff --git a/include/uapi/cxl/cxl_regs.h b/include/uapi/cxl/cxl_regs.h new file mode 100644 index 0000000000000..c821ef7ec2bb7 --- /dev/null +++ b/include/uapi/cxl/cxl_regs.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * CXL Standard defines + * + * Hardware register offsets and bit-field masks for the CXL Component + * Register block, as defined by the CXL Specification r4.0. + */ + +#ifndef _UAPI_CXL_REGS_H_ +#define _UAPI_CXL_REGS_H_ + +#include /* __BITS_PER_LONG; needed by __GENMASK() */ +#include /* _BITUL(), _BITULL() */ +#include /* __GENMASK() */ + +/* CXL 4.0 8.2.3 CXL Component Register Layout and Definition */ +#define CXL_COMPONENT_REG_BLOCK_SIZE 0x00010000 + +/* CXL 4.0 8.2.4 CXL.cache and CXL.mem Registers*/ +#define CXL_CM_OFFSET 0x1000 +#define CXL_CM_CAP_HDR_OFFSET 0x0 +#define CXL_CM_CAP_HDR_ID_MASK __GENMASK(15, 0) +#define CM_CAP_HDR_CAP_ID 1 +#define CXL_CM_CAP_HDR_VERSION_MASK __GENMASK(19, 16) +#define CM_CAP_HDR_CAP_VERSION 1 +#define CXL_CM_CAP_HDR_CACHE_MEM_VERSION_MASK __GENMASK(23, 20) +#define CM_CAP_HDR_CACHE_MEM_VERSION 1 +#define CXL_CM_CAP_HDR_ARRAY_SIZE_MASK __GENMASK(31, 24) +#define CXL_CM_CAP_PTR_MASK __GENMASK(31, 20) + +/* CXL HDM Decoder Capability Structure */ +#define CXL_HDM_DECODER_CAP_OFFSET 0x0 +#define CXL_HDM_DECODER_COUNT_MASK __GENMASK(3, 0) +#define CXL_HDM_DECODER_TARGET_COUNT_MASK __GENMASK(7, 4) +#define CXL_HDM_DECODER_INTERLEAVE_11_8 _BITUL(8) +#define CXL_HDM_DECODER_INTERLEAVE_14_12 _BITUL(9) +#define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY _BITUL(11) +#define CXL_HDM_DECODER_INTERLEAVE_16_WAY _BITUL(12) +#define CXL_HDM_DECODER_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_ENABLE _BITUL(1) +#define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) +#define CXL_HDM_DECODER0_BASE_HIGH_OFFSET(i) (0x20 * (i) + 0x14) +#define CXL_HDM_DECODER0_SIZE_LOW_OFFSET(i) (0x20 * (i) + 0x18) +#define CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(i) (0x20 * (i) + 0x1c) +#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_IG_MASK __GENMASK(3, 0) +#define CXL_HDM_DECODER0_CTRL_IW_MASK __GENMASK(7, 4) +#define CXL_HDM_DECODER0_CTRL_LOCK _BITUL(8) +#define CXL_HDM_DECODER0_CTRL_COMMIT _BITUL(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED _BITUL(10) +#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR _BITUL(11) +#define CXL_HDM_DECODER0_CTRL_HOSTONLY _BITUL(12) +#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24) +#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28) +#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i) +#define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i) + +#endif /* _UAPI_CXL_REGS_H_ */ From e1e4016e206766a158856bfa0056dd2c54abe38c Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:01 +0530 Subject: [PATCH 390/464] NVIDIA: VR: SAUCE: cxl: Split cxl_await_range_active() from media-ready wait BugLink: https://bugs.launchpad.net/bugs/2152222 Before accessing CXL device memory after reset/power-on, the driver must ensure media is ready. Not every CXL device implements the CXL Memory Device register group (many Type-2 devices do not). cxl_await_media_ready() reads cxlds->regs.memdev. Access to the memory device registers on a Type-2 device may result in kernel panic. Split the HDM DVSEC range-active poll out of cxl_await_media_ready() into a new function, cxl_await_range_active(). Type-2 devices often lack the CXLMDEV status register, so they need the range check without the memdev read. cxl_await_media_ready() now calls cxl_await_range_active() for the DVSEC poll, then reads the memory device status as before. Co-developed-by: Zhi Wang Reviewed-by: Dave Jiang Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 023bae337329a533c86481d829ea1bbc9ea7aa21 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Add cxl_await_range_active() declaration to include/cxl/pci.h unconditionally instead of include/cxl/cxl.h with CONFIG_CXL_BUS guards, consistent with existing convention] [kobak: Folded the private drivers/cxl/cxl.h cxl_await_range_active() prototype into this helper commit because drivers/cxl/core/pci.c includes the private CXL header.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/pci.c | 35 ++++++++++++++++++++++++++++++----- drivers/cxl/cxl.h | 1 + include/cxl/pci.h | 2 ++ 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index c248bf52799bc..3b6c878f822c0 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -147,16 +147,24 @@ static int cxl_dvsec_mem_range_active(struct cxl_dev_state *cxlds, int id) return 0; } -/* - * Wait up to @media_ready_timeout for the device to report memory - * active. +/** + * cxl_await_range_active - Wait for all HDM DVSEC memory ranges to be active + * @cxlds: CXL device state (DVSEC and HDM count must be valid) + * + * For each HDM decoder range reported in the CXL DVSEC capability, waits for + * the range to report MEM INFO VALID (up to 1s per range), then MEM ACTIVE + * (up to media_ready_timeout seconds per range, default 60s). Used by + * cxl_await_media_ready() and by callers that only need range readiness + * without checking the memory device status register. + * + * Return: 0 if all ranges become valid and active, -ETIMEDOUT if a timeout + * occurs, or a negative errno from config read on failure. */ -int cxl_await_media_ready(struct cxl_dev_state *cxlds) +int cxl_await_range_active(struct cxl_dev_state *cxlds) { struct pci_dev *pdev = to_pci_dev(cxlds->dev); int d = cxlds->cxl_dvsec; int rc, i, hdm_count; - u64 md_status; u16 cap; rc = pci_read_config_word(pdev, @@ -177,6 +185,23 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds) return rc; } + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_await_range_active, "CXL"); + +/* + * Wait up to @media_ready_timeout for the device to report memory + * active. + */ +int cxl_await_media_ready(struct cxl_dev_state *cxlds) +{ + u64 md_status; + int rc; + + rc = cxl_await_range_active(cxlds); + if (rc) + return rc; + md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET); if (!CXLMDEV_READY(md_status)) return -EIO; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index e1433ff43f12f..fd5a90f501dcf 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -722,6 +722,7 @@ int cxl_port_setup_regs(struct cxl_port *port, resource_size_t component_reg_phys); struct cxl_dev_state; +int cxl_await_range_active(struct cxl_dev_state *cxlds); int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u8 *count, resource_size_t *offset, resource_size_t *size); int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds, diff --git a/include/cxl/pci.h b/include/cxl/pci.h index 5c5fac2a91151..de1e99403bc7a 100644 --- a/include/cxl/pci.h +++ b/include/cxl/pci.h @@ -17,6 +17,7 @@ enum cxl_regloc_type { struct pci_dev; struct cxl_register_map; struct cxl_component_reg_map; +struct cxl_dev_state; int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); @@ -24,5 +25,6 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map); +int cxl_await_range_active(struct cxl_dev_state *cxlds); int cxl_setup_regs(struct cxl_register_map *map); #endif From ec0b56abb00769abe3c9f4555b7e0b0346f7c491 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:02 +0530 Subject: [PATCH 391/464] NVIDIA: VR: SAUCE: cxl: Record BIR and BAR offset in cxl_register_map BugLink: https://bugs.launchpad.net/bugs/2152222 The Register Locator DVSEC (CXL 4.0 8.1.9) describes register blocks by BAR index (BIR) and offset within the BAR. CXL core currently only stores the resolved HPA (resource + offset) in struct cxl_register_map, so callers that need to use pci_iomap() or report the BAR to userspace must reverse-engineer the BAR from the HPA. Add bar_index and bar_offset to struct cxl_register_map and fill them in cxl_decode_regblock() when the regblock is BAR-backed (BIR 0-5). Add cxl_regblock_get_bar_info() so callers (e.g. vfio-cxl) can get BAR index and offset directly and use pci_iomap() instead of ioremap(HPA). Return -EINVAL if the map is not BAR-backed. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 947749bd1b8d0308311553dbb7ed3db38be55907 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Add cxl_regblock_get_bar_info() declaration to include/cxl/pci.h unconditionally instead of include/cxl/cxl.h with CONFIG_CXL_BUS guards, consistent with existing convention, Add BIR range validation (reject BIR >= PCI_STD_NUM_BARS) plus a bar_index bounds check in cxl_regblock_get_bar_info()] [kobak: Added the target-local private drivers/cxl/cxl.h cxl_regblock_get_bar_info() prototype; struct cxl_register_map carries bar_index/bar_offset in include/cxl/cxl.h.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/regs.c | 49 +++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ include/cxl/cxl.h | 9 ++++++++ include/cxl/pci.h | 2 ++ 4 files changed, 62 insertions(+) diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index e828df0629d02..be9149dd90780 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -278,6 +278,19 @@ static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi, u64 offset = ((u64)reg_hi << 32) | (reg_lo & PCI_DVSEC_CXL_REG_LOCATOR_BLOCK_OFF_LOW); + /* + * The BIR field is 3 bits wide (CXL spec); values 6 and 7 are + * reserved. PCI only defines BAR 0-5, and pci_resource_*() on a + * higher index reads past the resource array. Reject those here + * so callers do not get garbage. + */ + if (bar >= PCI_STD_NUM_BARS) { + dev_warn(&pdev->dev, + "Reserved BIR %d in Register Locator entry (type %d)\n", + bar, reg_type); + return false; + } + if (offset > pci_resource_len(pdev, bar)) { dev_warn(&pdev->dev, "BAR%d: %pr: too small (offset: %pa, type: %d)\n", bar, @@ -288,9 +301,44 @@ static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi, map->reg_type = reg_type; map->resource = pci_resource_start(pdev, bar) + offset; map->max_size = pci_resource_len(pdev, bar) - offset; + map->bar_index = bar; + map->bar_offset = offset; return true; } +/** + * cxl_regblock_get_bar_info() - Get BAR index and offset for a BAR-backed + * regblock + * @map: Register map from cxl_find_regblock() or cxl_find_regblock_instance() + * @bar_index: Output BAR index (0-5). Optional, may be NULL. + * @bar_offset: Output offset within the BAR. Optional, may be NULL. + * + * When the register block was found via the Register Locator DVSEC and + * lives in a PCI BAR (BIR 0-5), this returns the BAR index and the offset + * within that BAR. + * + * Return: 0 if the regblock is BAR-backed (bar_index <= 5), -EINVAL otherwise. + */ +int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index, + resource_size_t *bar_offset) +{ + if (!map || map->bar_index == 0xff) + return -EINVAL; + /* + * Guard callers against stale or out-of-range bar_index. Only BAR + * indices 0..5 are valid PCI BARs; anything else means the map was + * not BAR-backed or was filled from a reserved BIR. + */ + if (map->bar_index >= PCI_STD_NUM_BARS) + return -EINVAL; + if (bar_index) + *bar_index = map->bar_index; + if (bar_offset) + *bar_offset = map->bar_offset; + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_regblock_get_bar_info, "CXL"); + /* * __cxl_find_regblock_instance() - Locate a register block or count instances by type / index * Use CXL_INSTANCES_COUNT for @index if counting instances. @@ -309,6 +357,7 @@ static int __cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_ty *map = (struct cxl_register_map) { .host = &pdev->dev, + .bar_index = 0xFF, .resource = CXL_RESOURCE_NONE, }; diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index fd5a90f501dcf..5357868600494 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -160,6 +160,8 @@ int cxl_map_pmu_regs(struct cxl_register_map *map, struct cxl_pmu_regs *regs); int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type); int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map, unsigned int index); +int cxl_regblock_get_bar_info(const struct cxl_register_map *map, + u8 *bar_index, resource_size_t *bar_offset); struct cxl_dport; int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 716371ba38243..6fd6efaaef06a 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -118,9 +118,16 @@ struct cxl_pmu_reg_map { * @resource: physical resource base of the register block * @max_size: maximum mapping size to perform register search * @reg_type: see enum cxl_regloc_type + * @bar_index: PCI BAR index (0-5) when regblock is BAR-backed; 0xFF otherwise + * @bar_offset: offset within the BAR; only valid when bar_index <= 5 * @component_map: cxl_reg_map for component registers * @device_map: cxl_reg_maps for device registers * @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units + * + * When the register block is described by the Register Locator DVSEC with + * a BAR Indicator (BIR 0-5), bar_index and bar_offset are set so callers can + * use pci_iomap(pdev, bar_index, size) and base + bar_offset instead of + * ioremap(resource). */ struct cxl_register_map { struct device *host; @@ -128,6 +135,8 @@ struct cxl_register_map { resource_size_t resource; resource_size_t max_size; u8 reg_type; + u8 bar_index; + resource_size_t bar_offset; union { struct cxl_component_reg_map component_map; struct cxl_device_reg_map device_map; diff --git a/include/cxl/pci.h b/include/cxl/pci.h index de1e99403bc7a..38bd7c73575a8 100644 --- a/include/cxl/pci.h +++ b/include/cxl/pci.h @@ -26,5 +26,7 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, void cxl_probe_component_regs(struct device *dev, void __iomem *base, struct cxl_component_reg_map *map); int cxl_await_range_active(struct cxl_dev_state *cxlds); +int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index, + resource_size_t *bar_offset); int cxl_setup_regs(struct cxl_register_map *map); #endif From f814c22068ad3f108c5b1cd206e089a0646a5b33 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:03 +0530 Subject: [PATCH 392/464] NVIDIA: VR: SAUCE: vfio: UAPI for CXL-capable PCI device assignment BugLink: https://bugs.launchpad.net/bugs/2152222 Vendor GPUs and accelerators can expose CXL.mem (HDM-D or HDM-DB) without using PCI class code 0x0502. VMMs need a stable way to learn DPA sizing, firmware commit state, and where the extra VFIO regions live. Add VFIO_DEVICE_FLAGS_CXL (bit 9) and VFIO_DEVICE_INFO_CAP_CXL (cap ID 6). The capability struct carries: hdm_regs_bar_index PCI BAR containing the component register block hdm_regs_offset byte offset within that BAR to the CXL.mem area (comp_reg_offset + CXL_CM_OFFSET) dpa_region_index VFIO region index for the DPA window comp_regs_region_index VFIO region index for the emulated COMP_REGS HDM decoder count and the HDM block offset within COMP_REGS are intentionally absent; both are derivable from the CXL Capability Array at COMP_REGS offset 0. Locate cap ID 0x5 (HDM) and read bits[31:20] of its entry for the byte offset. Then read bits[3:0] of the HDM Decoder Capability register for the count: count = (field == 0) ? 1 : field * 2. Two flags accompany the capability: VFIO_CXL_CAP_FIRMWARE_COMMITTED A decoder covering @dpa_size bytes was programmed and committed by platform firmware before device open. The VMM can use the DPA region immediately without re-committing. VFIO_CXL_CAP_CACHE_CAPABLE The device is HDM-DB (CXL.mem + CXL.cache). HDM-DB requires a Write-Back Invalidation sequence before FLR to flush dirty cache lines; HDM-D (CXL.mem only) does not. QEMU uses this flag to schedule WBI and to report Back-Invalidation capability accurately in the virtual CXL topology. Mirrors the Cache_Capable bit from the CXL DVSEC Capability register. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (cherry-picked from commit c0f4d247a0e714f22dcd9fe6679132edf1b6a846 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/uapi/linux/vfio.h | 86 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index bb7b89330d359..996765978ce7f 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -215,6 +215,16 @@ struct vfio_device_info { #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6) /* vfio-fsl-mc device */ #define VFIO_DEVICE_FLAGS_CAPS (1 << 7) /* Info supports caps */ #define VFIO_DEVICE_FLAGS_CDX (1 << 8) /* vfio-cdx device */ +/* + * Vendor-specific CXL device with CXL.mem capability (HDM-D or HDM-DB + * decoder, PCI class code != PCI_CLASS_MEMORY_CXL). Covers CXL Type-2 + * accelerators and non-class-code Type-3 variants. When set, + * VFIO_DEVICE_FLAGS_PCI is also set (same device is a PCI device). The + * capability chain (VFIO_DEVICE_FLAGS_CAPS) contains VFIO_DEVICE_INFO_CAP_CXL + * describing HDM decoders, region indices, decoder layout, and CXL-specific + * options. + */ +#define VFIO_DEVICE_FLAGS_CXL (1 << 9) /* Device supports CXL */ __u32 num_regions; /* Max region index + 1 */ __u32 num_irqs; /* Max IRQ index + 1 */ __u32 cap_offset; /* Offset within info struct of first cap */ @@ -257,6 +267,70 @@ struct vfio_device_info_cap_pci_atomic_comp { __u32 reserved; }; +/* + * VFIO_DEVICE_INFO_CAP_CXL - CXL Type-2 device capability + * + * Present in the device info capability chain when VFIO_DEVICE_FLAGS_CXL + * is set. Describes Host Managed Device Memory (HDM) layout and CXL + * memory options so that userspace (e.g. QEMU) can expose the CXL region + * and component registers correctly to the guest. + * + * The HDM decoder count and HDM decoder block offset within the COMP_REGS + * region are derivable from the COMP_REGS region itself. + * + * To find the HDM decoder block offset (hdm_decoder_offset), traverse the CXL + * Capability Array starting at COMP_REGS region offset 0: + * - Dword 0 bits[31:24] (CXL_CM_CAP_HDR_ARRAY_SIZE_MASK): number of + * capability entries. + * - Each subsequent dword at offset (cap * 4): bits[15:0] = cap ID + * (CXL_CM_CAP_HDR_ID_MASK), bits[31:20] = byte offset from COMP_REGS + * start to that capability's register block (CXL_CM_CAP_PTR_MASK). + * - Locate the entry with cap ID == CXL_CM_CAP_CAP_ID_HDM (0x5); the + * extracted bits[31:20] value is directly the byte offset + * hdm_decoder_offset (no further scaling required). + * + * To find the HDM decoder count, pread the HDM Decoder Capability register + * at hdm_decoder_offset + CXL_HDM_DECODER_CAP_OFFSET within the + * COMP_REGS region; bits[3:0] (CXL_HDM_DECODER_COUNT_MASK) encode the count + * using the formula: count = (field == 0) ? 1 : field * 2. + */ +#define VFIO_DEVICE_INFO_CAP_CXL 6 +struct vfio_device_info_cap_cxl { + struct vfio_info_cap_header header; + __u8 hdm_regs_bar_index; /* PCI BAR containing HDM registers */ + __u8 reserved[3]; + __u32 flags; +/* Decoder was committed by host firmware/BIOS */ +#define VFIO_CXL_CAP_FIRMWARE_COMMITTED (1 << 0) +/* + * Device implements an HDM-DB decoder (CXL.cache + CXL.mem). Reflects + * the Cache_Capable bit (bit 0) in the CXL DVSEC Capability register. + * + * When clear: HDM-D decoder (CXL.mem only, no CXL.cache). FLR does not + * require a Write-Back Invalidation (WBI) sequence; the device holds no + * coherent copies of host memory. + * + * When set: HDM-DB decoder (CXL 3.0+). The kernel driver does not + * perform Write-Back Invalidation (WBI) automatically. The VMM must + * issue a WBI sequence before asserting FLR to flush dirty device cache + * lines and prevent coherency violations, and should advertise + * Back-Invalidation support in the virtual CXL topology. + */ +#define VFIO_CXL_CAP_CACHE_CAPABLE (1 << 1) + /* + * Byte offset within the BAR to the CXL.mem register area start + * (= comp_reg_offset + CXL_CM_OFFSET). This is where the CXL + * Capability Array Header lives. + */ + __u64 hdm_regs_offset; + /* + * Region indices for the two CXL VFIO device regions. + * Avoids forcing userspace to scan all regions by type/subtype. + */ + __u32 dpa_region_index; /* VFIO_REGION_SUBTYPE_CXL */ + __u32 comp_regs_region_index; /* VFIO_REGION_SUBTYPE_CXL_COMP_REGS */ +}; + /** * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, * struct vfio_region_info) @@ -370,6 +444,18 @@ struct vfio_region_info_cap_type { */ #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) +/* 1e98 vendor PCI sub-types (CXL Consortium) */ +/* + * CXL memory region. Use with region type + * (PCI_VENDOR_ID_CXL | VFIO_REGION_TYPE_PCI_VENDOR_TYPE). + * DPA memory region (fault+zap mmap) + */ +#define VFIO_REGION_SUBTYPE_CXL (1) +/* + * HDM decoder register emulation region (read/write only, no mmap). + */ +#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS (2) + /* sub-types for VFIO_REGION_TYPE_GFX */ #define VFIO_REGION_SUBTYPE_GFX_EDID (1) From b60556a67048b54d46de25195cc8927c4c4ecb18 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:04 +0530 Subject: [PATCH 393/464] NVIDIA: VR: SAUCE: vfio/pci: Add CXL state to vfio_pci_core_device BugLink: https://bugs.launchpad.net/bugs/2152222 Add struct vfio_pci_cxl_state and hang a pointer to it off vfio_pci_core_device. vdev->cxl stays NULL for non-CXL devices, so existing vfio-pci-core paths just pay a NULL check. The new struct embeds struct cxl_dev_state by value (CXL core uses container_of() against this field) and stores pointers to the cxl_memdev, root decoder, and endpoint decoder that the CXL core owns. cxl_region is not introduced here; it is added later when region management lands. The series builds the CXL Type-2 passthrough path inside vfio-pci-core rather than in a separate variant driver. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 87b80cc08c264 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve context mismatch in vfio_pci_core.h; add #include to vfio_cxl_priv.h for cxl_find_regblock/cxl_probe_component_regs declarations] [kobak: Preserved existing VFIO PCI DMABUF forward declarations while adding the CXL state forward declaration.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_priv.h | 29 ++++++++++++++++++++++++++++ include/linux/vfio_pci_core.h | 2 ++ 2 files changed, 31 insertions(+) create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_priv.h diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h new file mode 100644 index 0000000000000..0ea1d8ddbd492 --- /dev/null +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Common infrastructure for CXL Type-2 device variant drivers + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#ifndef __LINUX_VFIO_CXL_PRIV_H +#define __LINUX_VFIO_CXL_PRIV_H + +#include +#include +#include + +/* CXL device state embedded in vfio_pci_core_device */ +struct vfio_pci_cxl_state { + struct cxl_dev_state cxlds; + struct cxl_memdev *cxlmd; + struct cxl_root_decoder *cxlrd; + struct cxl_endpoint_decoder *cxled; + resource_size_t hdm_reg_offset; + size_t hdm_reg_size; + resource_size_t comp_reg_offset; + size_t comp_reg_size; + u8 hdm_count; + u8 comp_reg_bar; +}; + +#endif /* __LINUX_VFIO_CXL_PRIV_H */ diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 2ebba746c18f7..6e8885f79d264 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -29,6 +29,7 @@ struct vfio_pci_core_device; struct vfio_pci_region; struct p2pdma_provider; struct dma_buf_attachment; +struct vfio_pci_cxl_state; struct vfio_pci_eventfd { struct eventfd_ctx *ctx; @@ -137,6 +138,7 @@ struct vfio_pci_core_device { struct mutex ioeventfds_lock; struct list_head ioeventfds_list; struct vfio_pci_vf_token *vf_token; + struct vfio_pci_cxl_state *cxl; struct list_head sriov_pfs_item; struct vfio_pci_core_device *sriov_pf_core_dev; struct notifier_block nb; From 9b89aeee35420ed39160311c8f37c93249b948b7 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:05 +0530 Subject: [PATCH 394/464] NVIDIA: VR: SAUCE: vfio/pci: Add CONFIG_VFIO_CXL_CORE and stub CXL hooks BugLink: https://bugs.launchpad.net/bugs/2152222 Introduce the Kconfig option CONFIG_VFIO_CXL_CORE and the necessary build rules to compile CXL.mem passthrough infrastructure for vendor-specific CXL devices into the vfio-pci-core module. The new option depends on VFIO_PCI_CORE, CXL_BUS and CXL_MEM. Wire up the detection and cleanup entry-point stubs in vfio_pci_core_register_device() and vfio_pci_core_unregister_device() so that subsequent patches can fill in the CXL-specific logic without touching the vfio-pci-core flow again. The vfio_cxl_core.c file added here is an empty skeleton; the actual CXL detection and initialisation code is introduced in the following patch to keep this build-system patch reviewable on its own. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 336a1448463a from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve context mismatches in Kconfig, Makefile, and vfio_pci_priv.h due to missing upstream xe/dmabuf support in NV-Kernels base] [kobak: Preserved existing VFIO PCI DMABUF declarations while adding VFIO CXL stubs.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/Kconfig | 2 ++ drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/cxl/Kconfig | 9 ++++++ drivers/vfio/pci/cxl/vfio_cxl_core.c | 41 ++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 4 +++ drivers/vfio/pci/vfio_pci_priv.h | 14 ++++++++++ 6 files changed, 71 insertions(+) create mode 100644 drivers/vfio/pci/cxl/Kconfig create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_core.c diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig index 1e82b44bda1a0..b981a7c164ca9 100644 --- a/drivers/vfio/pci/Kconfig +++ b/drivers/vfio/pci/Kconfig @@ -68,6 +68,8 @@ source "drivers/vfio/pci/virtio/Kconfig" source "drivers/vfio/pci/nvgrace-gpu/Kconfig" +source "drivers/vfio/pci/cxl/Kconfig" + source "drivers/vfio/pci/qat/Kconfig" source "drivers/vfio/pci/xe/Kconfig" diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index e0a0757dd1d2b..ecb0eacbc089b 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o +vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o diff --git a/drivers/vfio/pci/cxl/Kconfig b/drivers/vfio/pci/cxl/Kconfig new file mode 100644 index 0000000000000..fad53300fecfb --- /dev/null +++ b/drivers/vfio/pci/cxl/Kconfig @@ -0,0 +1,9 @@ +config VFIO_CXL_CORE + bool "VFIO CXL core" + depends on VFIO_PCI_CORE && CXL_BUS && CXL_MEM + help + Extends vfio-pci-core with CXL.mem passthrough for vendor-specific + CXL devices (CXL_DEVTYPE_DEVMEM) that implement HDM-D or HDM-DB + decoders without the standard CXL memory expander class code + (PCI_CLASS_MEMORY_CXL). Covers CXL Type-2 accelerators and + non-class-code Type-3 variants (e.g. compressed memory devices). diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c new file mode 100644 index 0000000000000..d12afec82ecdb --- /dev/null +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * VFIO CXL Core - CXL.mem passthrough for vendor-specific CXL devices + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + * + * This module extends vfio-pci-core to pass through CXL.mem regions for + * vendor-specific CXL devices (CXL_DEVTYPE_DEVMEM) that implement HDM-D or + * HDM-DB decoders but do not report the standard CXL memory expander class + * code (PCI_CLASS_MEMORY_CXL, 0x0502). This covers both CXL Type-2 + * accelerators (with CXL.cache) and non-class-code Type-3 variants (e.g. + * compressed memory devices) which cannot be paravirtualized by the host + * CXL subsystem and require direct DPA region access from the guest. + */ + +#include +#include +#include +#include + +#include "../vfio_pci_priv.h" +#include "vfio_cxl_priv.h" + +/** + * vfio_pci_cxl_detect_and_init - Detect and initialize a vendor-specific + * CXL.mem device + * @vdev: VFIO PCI device + * + * Called from vfio_pci_core_register_device(). Detects CXL DVSEC capability + * and initializes CXL features. On failure vdev->cxl remains NULL and the + * device operates as a standard PCI device. + */ +void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) +{ +} + +void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) +{ +} + +MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index e34c7e1ba1c81..c5d7cfabf5155 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2183,6 +2183,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) if (ret) goto out_vf; + vfio_pci_cxl_detect_and_init(vdev); + vfio_pci_probe_power_state(vdev); /* @@ -2226,6 +2228,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) vfio_pci_vf_uninit(vdev); vfio_pci_vga_uninit(vdev); + vfio_pci_cxl_cleanup(vdev); + if (!disable_idle_d3) pm_runtime_get_noresume(&vdev->pdev->dev); diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index 27ac280f00b97..d7df5538dcde8 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -133,4 +133,18 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, } #endif +#if IS_ENABLED(CONFIG_VFIO_CXL_CORE) + +void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev); +void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev); + +#else + +static inline void +vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { } +static inline void +vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { } + +#endif /* CONFIG_VFIO_CXL_CORE */ + #endif From 96b4dd026a20622d40880facb767936eb048a583 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:06 +0530 Subject: [PATCH 395/464] NVIDIA: VR: SAUCE: vfio/cxl: Detect CXL DVSEC and probe HDM block BugLink: https://bugs.launchpad.net/bugs/2152222 Detect a vendor-specific CXL device at vfio-pci bind time and probe its HDM decoder register block. vfio_cxl_create_device_state() allocates per-device state via devm, reads the DVSEC length from PCI_DVSEC_HEADER1, and records MEM_CAPABLE and CACHE_CAPABLE from the CXL DVSEC. vfio_cxl_setup_regs() locates the component register block, claims and maps that BAR window, calls cxl_probe_component_regs() to find the HDM block, then unmaps and releases the window on all paths. vfio_pci_cxl_detect_and_init() enables PCI memory decoding for the probe, chains these setup steps, disables the device again, and leaves vdev->cxl NULL on failure so the device falls back to plain vfio-pci. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (cherry-picked from commit 939ebb73d430f185c3e8ba55f26ba6888452b0d7 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Use pci_get_dsn() instead of pdev->dev.id for cxlds serial; expand comment explaining why] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 222 +++++++++++++++++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_priv.h | 12 ++ 2 files changed, 234 insertions(+) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index d12afec82ecdb..8b83f6619aa09 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -21,6 +21,163 @@ #include "../vfio_pci_priv.h" #include "vfio_cxl_priv.h" +/* + * vfio_cxl_create_device_state - Allocate and validate CXL device state + * + * Returns a pointer to the allocated vfio_pci_cxl_state on success, or + * ERR_PTR on failure. The allocation uses devm; the caller must call + * devm_kfree(&pdev->dev, cxl) on any subsequent setup failure to release + * the resource before device unbind. Using devm_kfree() to undo a devm + * allocation early is explicitly supported by the devres API. + * + * The caller assigns vdev->cxl only after all setup steps succeed, preventing + * partially-initialised state from being visible through vdev->cxl on any + * failure path. + */ +static struct vfio_pci_cxl_state * +vfio_cxl_create_device_state(struct pci_dev *pdev, u16 dvsec) +{ + struct vfio_pci_cxl_state *cxl; + u16 cap_word; + u32 hdr1; + + /* + * Freed automatically when pdev->dev is released. Use the PCI Device + * Serial Number capability for cxlds->serial; pdev->dev.id is the + * generic-device sibling counter (typically 0) and surfaces as a bogus + * serial in sysfs and CXL tracepoints. + */ + cxl = devm_cxl_dev_state_create(&pdev->dev, + CXL_DEVTYPE_DEVMEM, + pci_get_dsn(pdev), dvsec, + struct vfio_pci_cxl_state, + cxlds, false); + if (!cxl) + return ERR_PTR(-ENOMEM); + + pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1, &hdr1); + cxl->dvsec_len = PCI_DVSEC_HEADER1_LEN(hdr1); + + pci_read_config_word(pdev, dvsec + CXL_DVSEC_CAPABILITY_OFFSET, + &cap_word); + + /* + * Only handle vendor devices (class != 0x0502) with Mem_Capable set. + * CACHE_CAPABLE is forwarded to the VMM so it knows whether a WBI + * sequence is needed before FLR. + */ + if (!FIELD_GET(CXL_DVSEC_MEM_CAPABLE, cap_word) || + (pdev->class >> 8) == PCI_CLASS_MEMORY_CXL) { + devm_kfree(&pdev->dev, cxl); + return ERR_PTR(-ENODEV); + } + + cxl->cache_capable = FIELD_GET(CXL_DVSEC_CACHE_CAPABLE, cap_word); + + return cxl; +} + +static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl) +{ + struct cxl_register_map *map = &cxl->cxlds.reg_map; + resource_size_t offset, bar_offset, size; + struct pci_dev *pdev = vdev->pdev; + void __iomem *base; + int ret; + u8 count; + u8 bar; + + if (WARN_ON_ONCE(!pci_is_enabled(pdev))) + return -EINVAL; + + /* Find component register block via Register Locator DVSEC */ + ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, map); + if (ret) + return ret; + + /* + * Request the region and map. This is a transient mapping + * used only to probe register capabilities; released immediately + * after cxl_probe_component_regs() returns. + */ + if (!request_mem_region(map->resource, map->max_size, "vfio-cxl-probe")) + return -EBUSY; + + base = ioremap(map->resource, map->max_size); + if (!base) { + ret = -ENOMEM; + goto failed_release; + } + + /* Probe component register capabilities */ + cxl_probe_component_regs(&pdev->dev, base, &map->component_map); + + /* Check if HDM decoder was found */ + if (!map->component_map.hdm_decoder.valid) { + ret = -ENODEV; + goto failed_unmap; + } + + pci_dbg(pdev, "vfio_cxl: HDM decoder at offset=0x%lx, size=0x%lx\n", + map->component_map.hdm_decoder.offset, + map->component_map.hdm_decoder.size); + + /* Get HDM register info */ + ret = cxl_get_hdm_info(&cxl->cxlds, &count, &offset, &size); + if (ret) + goto failed_unmap; + + if (!count || !size) { + ret = -ENODEV; + goto failed_unmap; + } + + cxl->hdm_count = count; + /* + * cxl_get_hdm_info() returns rmap->offset = CXL_CM_OFFSET + + * (see cxl_probe_component_regs() which does base += CXL_CM_OFFSET before + * reading caps and stores CXL_CM_OFFSET + cap_ptr as the offset). + * Subtract CXL_CM_OFFSET so hdm_reg_offset is relative to the CXL.mem + * register area start, which is where comp_reg_virt[0] is anchored. + * The physical BAR address for hdm_iobase is recovered by adding + * CXL_CM_OFFSET back in vfio_cxl_setup_virt_regs(). + */ + cxl->hdm_reg_offset = offset - CXL_CM_OFFSET; + cxl->hdm_reg_size = size; + + ret = cxl_regblock_get_bar_info(map, &bar, &bar_offset); + if (ret) + goto failed_unmap; + + cxl->comp_reg_bar = bar; + cxl->comp_reg_offset = bar_offset; + cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE; + + iounmap(base); + release_mem_region(map->resource, map->max_size); + + return 0; + +failed_unmap: + iounmap(base); +failed_release: + release_mem_region(map->resource, map->max_size); + + return ret; +} + +/* + * Free CXL state early on probe failure. devm_kfree() on a live devres + * allocation removes it from the list immediately, so the normal devres + * teardown at unbind time won't double-free it. + */ +static void vfio_cxl_dev_state_free(struct pci_dev *pdev, + struct vfio_pci_cxl_state *cxl) +{ + devm_kfree(&pdev->dev, cxl); +} + /** * vfio_pci_cxl_detect_and_init - Detect and initialize a vendor-specific * CXL.mem device @@ -32,10 +189,75 @@ */ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { + struct pci_dev *pdev = vdev->pdev; + struct vfio_pci_cxl_state *cxl; + u16 dvsec; + int ret; + + if (!pcie_is_cxl(pdev)) + return; + + dvsec = pci_find_dvsec_capability(pdev, + PCI_VENDOR_ID_CXL, + PCI_DVSEC_CXL_DEVICE); + if (!dvsec) + return; + + /* + * CXL DVSEC found: any failure from here is a hard probe error on + * a confirmed CXL-capable device, not a silent non-CXL fallback. + * Warn the operator so misconfiguration is visible. + */ + cxl = vfio_cxl_create_device_state(pdev, dvsec); + if (IS_ERR(cxl)) { + if (PTR_ERR(cxl) != -ENODEV) + pci_warn(pdev, + "vfio-cxl: CXL device state allocation failed: %ld\n", + PTR_ERR(cxl)); + return; + } + + /* + * Required for ioremap of the component register block and + * calls to cxl_probe_component_regs(). + */ + ret = pci_enable_device_mem(pdev); + if (ret) { + pci_warn(pdev, + "vfio-cxl: pci_enable_device_mem failed: %d\n", ret); + goto free_cxl; + } + + ret = vfio_cxl_setup_regs(vdev, cxl); + if (ret) { + pci_warn(pdev, + "vfio-cxl: HDM register probing failed: %d\n", ret); + pci_disable_device(pdev); + goto free_cxl; + } + + pci_disable_device(pdev); + + /* + * Register probing succeeded. Assign vdev->cxl now so that + * all subsequent helpers can access state via vdev->cxl. + * All failure paths below clear vdev->cxl before calling + * vfio_cxl_dev_state_free(). + */ + vdev->cxl = cxl; + + return; + +free_cxl: + vfio_cxl_dev_state_free(pdev, cxl); } void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + if (!cxl) + return; } MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index 0ea1d8ddbd492..bb03f9363d98f 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -22,8 +22,20 @@ struct vfio_pci_cxl_state { size_t hdm_reg_size; resource_size_t comp_reg_offset; size_t comp_reg_size; + u16 dvsec_len; u8 hdm_count; u8 comp_reg_bar; + bool cache_capable; }; +/* + * CXL DVSEC for CXL Devices - register offsets within the DVSEC + * (CXL 4.0 8.1.3). + * Offsets are relative to the DVSEC capability base (cxl->dvsec). + */ +#define CXL_DVSEC_CAPABILITY_OFFSET 0xa +#define CXL_DVSEC_MEM_CAPABLE BIT(2) +/* CXL DVSEC Capability register bit 0: device supports CXL.cache (HDM-DB) */ +#define CXL_DVSEC_CACHE_CAPABLE BIT(0) + #endif /* __LINUX_VFIO_CXL_PRIV_H */ From 9515cafc1e4934bd7d58670ca5a27d0742e3babb Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:07 +0530 Subject: [PATCH 396/464] NVIDIA: VR: SAUCE: vfio/pci: Export config access helpers BugLink: https://bugs.launchpad.net/bugs/2152222 Promote vfio_raw_config_write() and vfio_raw_config_read() to non-static so that the CXL DVSEC write handler in the next patch can call them. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (cherry-picked from commit 07d7141447024a424c16206383e15bcddf8dfb9f from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/vfio_pci_config.c | 12 ++++++------ drivers/vfio/pci/vfio_pci_priv.h | 8 ++++++++ 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index b4e39253f98da..61e7cdf7bec86 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -270,9 +270,9 @@ static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos, } /* Raw access skips any kind of virtualization */ -static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos, - int count, struct perm_bits *perm, - int offset, __le32 val) +int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 val) { int ret; @@ -283,9 +283,9 @@ static int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos, return count; } -static int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos, - int count, struct perm_bits *perm, - int offset, __le32 *val) +int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 *val) { int ret; diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index d7df5538dcde8..1082ba43bafe9 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -37,6 +37,14 @@ int vfio_pci_set_irqs_ioctl(struct vfio_pci_core_device *vdev, uint32_t flags, ssize_t vfio_pci_config_rw(struct vfio_pci_core_device *vdev, char __user *buf, size_t count, loff_t *ppos, bool iswrite); +int vfio_raw_config_write(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 val); + +int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 *val); + ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf, size_t count, loff_t *ppos, bool iswrite); From 41045be95c5b72d60199fa621c1ed206887d6814 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:08 +0530 Subject: [PATCH 397/464] NVIDIA: VR: SAUCE: vfio/cxl: Introduce HDM decoder register emulation framework BugLink: https://bugs.launchpad.net/bugs/2152222 Add HDM decoder register emulation for CXL devices assigned to a guest. New file vfio_cxl_emu.c allocates comp_reg_virt[] covering the full component register block (CXL_COMPONENT_REG_BLOCK_SIZE), snapshots it from MMIO after probe, and registers a VFIO device region (VFIO_REGION_SUBTYPE_CXL_COMP_REGS) with read/write ops but no mmap, so every access hits the emulated buffer and write dispatchers. vfio_cxl_setup_virt_regs() is called from the tail of vfio_cxl_setup_regs(); vfio_cxl_clean_virt_regs() runs on cleanup. HDM decoder register defines come from include/uapi/cxl/cxl_regs.h. Bits with no hardware equivalent stay in vfio_cxl_priv.h. hdm_decoder_n_ctrl_write() allows the guest to clear the LOCK bit. A firmware-committed decoder arrives with LOCK=1; the guest driver must clear it before reprogramming BASE and SIZE with the VM's GPA. Such a write clears the bit in the shadow while preserving all other fields. Co-developed-by: Zhi Wang Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 4ab495542be13ab04ac277ff8731d4233d661b97 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve Makefile context mismatch due to missing upstream dmabuf support in NV-Kernels base, Add CTRL LOCK enforcement in BASE_LO/SIZE_LO writes, BI bit masking for non-cache-capable devices, pass max_size to vfio_cxl_setup_virt_regs() for bounds check, add vfio_pci_cxl_cleanup() in registration error path] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/Makefile | 2 +- drivers/vfio/pci/cxl/vfio_cxl_core.c | 5 + drivers/vfio/pci/cxl/vfio_cxl_emu.c | 462 +++++++++++++++++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_priv.h | 48 +++ drivers/vfio/pci/vfio_pci_core.c | 9 + include/uapi/cxl/cxl_regs.h | 5 + 6 files changed, 530 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_emu.c diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index ecb0eacbc089b..bef916495eae7 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o -vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o +vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 8b83f6619aa09..72033fda7954d 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -154,8 +154,11 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev, cxl->comp_reg_offset = bar_offset; cxl->comp_reg_size = CXL_COMPONENT_REG_BLOCK_SIZE; + ret = vfio_cxl_setup_virt_regs(vdev, cxl, base, map->max_size); iounmap(base); release_mem_region(map->resource, map->max_size); + if (ret) + return ret; return 0; @@ -258,6 +261,8 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) if (!cxl) return; + + vfio_cxl_clean_virt_regs(cxl); } MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c new file mode 100644 index 0000000000000..5626ab51a053e --- /dev/null +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -0,0 +1,462 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include + +#include "../vfio_pci_priv.h" +#include "vfio_cxl_priv.h" + +/* + * comp_reg_virt[] shadow layout: + * Covers the full CXL.mem register area (starting at CXL_CM_OFFSET + * within the component register block). Index 0 is the CXL Capability + * Array Header; the HDM decoder block starts at index + * hdm_reg_offset / sizeof(__le32). + * + * Register layout within the HDM block (CXL spec 4.0 8.2.4.20 CXL HDM Decoder + * Capability Structure): + * 0x00: HDM Decoder Capability + * 0x04: HDM Decoder Global Control + * 0x08: (reserved) + * 0x0c: (reserved) + * For each decoder N (N=0..hdm_count-1), at base 0x10 + N*0x20: + * +0x00: BASE_LO + * +0x04: BASE_HI + * +0x08: SIZE_LO + * +0x0c: SIZE_HI + * +0x10: CTRL + * +0x14: TARGET_LIST_LO + * +0x18: TARGET_LIST_HI + * +0x1c: (reserved) + */ + +static inline __le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32 hdm_off) +{ + /* + * hdm_off is a byte offset within the HDM decoder block. + * comp_reg_virt covers the CXL.mem register area starting at + * CXL_CM_OFFSET within the component register block. + * hdm_reg_offset is CXL.mem-relative, so adding hdm_reg_offset + * gives the correct index into comp_reg_virt[]. + */ + return &cxl->comp_reg_virt[(cxl->hdm_reg_offset + hdm_off) / + sizeof(__le32)]; +} + +static ssize_t virt_hdm_rev_reg_write(struct vfio_pci_core_device *vdev, + const __le32 *val32, u64 offset, u64 size) +{ + /* Discard writes on reserved registers. */ + return size; +} + +static ssize_t hdm_decoder_n_lo_write(struct vfio_pci_core_device *vdev, + const __le32 *val32, u64 offset, u64 size) +{ + u32 new_val = le32_to_cpu(*val32); + u32 dec_idx, ctrl_off, ctrl; + + if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD)) + return -EINVAL; + + /* + * Honor the CTRL LOCK bit the same way BASE_HI/SIZE_HI do: once the + * guest sets LOCK, BASE_LO/SIZE_LO must remain frozen in shadow. + */ + dec_idx = ((u32)offset - CXL_HDM_DECODER_FIRST_BLOCK_OFFSET) / + CXL_HDM_DECODER_BLOCK_STRIDE; + ctrl_off = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET + + dec_idx * CXL_HDM_DECODER_BLOCK_STRIDE + + CXL_HDM_DECODER_N_CTRL_OFFSET; + ctrl = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, ctrl_off)); + if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) + return size; + + /* Bits [27:0] are reserved. */ + new_val &= ~CXL_HDM_DECODER_BASE_LO_RESERVED_MASK; + + *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val); + + return size; +} + +static ssize_t hdm_decoder_global_ctrl_write(struct vfio_pci_core_device *vdev, + const __le32 *val32, u64 size) +{ + u32 hdm_gcap; + u32 new_val = le32_to_cpu(*val32); + + if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD)) + return -EINVAL; + + /* Bit [31:2] are reserved. */ + new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK; + + /* Poison On Decode Error Enable (bit 0) is RO=0 if not supported. */ + hdm_gcap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, + CXL_HDM_DECODER_CAP_OFFSET)); + if (!(hdm_gcap & CXL_HDM_DECODER_POISON_ON_DECODE_ERR)) + new_val &= ~CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT; + + *hdm_reg_ptr(vdev->cxl, CXL_HDM_DECODER_CTRL_OFFSET) = + cpu_to_le32(new_val); + + return size; +} + +/** + * hdm_decoder_n_ctrl_write - Write handler for HDM decoder CTRL register. + * @vdev: VFIO PCI core device + * @val32: New register value supplied by userspace (little-endian) + * @offset: Byte offset within the HDM block for this decoder's CTRL register + * @size: Access size in bytes; must equal CXL_REG_SIZE_DWORD + * + * The COMMIT bit (bit 9) is the key: setting it requests the hardware to + * lock the decoder. The emulated COMMITTED bit (bit 10) mirrors COMMIT + * immediately to allow QEMU's notify_change to detect the transition and + * map/unmap the DPA MemoryRegion in the guest address space. + * + * Note: the actual hardware HDM decoder programming (writing the real + * BASE/SIZE with host physical addresses) happens in the QEMU notify_change + * callback BEFORE this write reaches the hardware. This ordering is + * correct because vfio_region_write() calls notify_change() first. + * + * Return: @size on success, %-EINVAL if @size is not %CXL_REG_SIZE_DWORD. + */ +static ssize_t hdm_decoder_n_ctrl_write(struct vfio_pci_core_device *vdev, + const __le32 *val32, u64 offset, u64 size) +{ + u32 hdm_gcap; + u32 ro_mask = CXL_HDM_DECODER_CTRL_RO_BITS_MASK; + u32 rev_mask = CXL_HDM_DECODER_CTRL_RESERVED_MASK; + u32 new_val = le32_to_cpu(*val32); + u32 cur_val; + + if (WARN_ON_ONCE(size != CXL_REG_SIZE_DWORD)) + return -EINVAL; + + cur_val = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, offset)); + if (cur_val & CXL_HDM_DECODER0_CTRL_LOCK) { + if (new_val & CXL_HDM_DECODER0_CTRL_LOCK) + return size; + + /* LOCK_0 only: preserve all other bits, clear LOCK */ + *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32( + cur_val & ~CXL_HDM_DECODER0_CTRL_LOCK); + return size; + } + + hdm_gcap = le32_to_cpu(*hdm_reg_ptr(vdev->cxl, + CXL_HDM_DECODER_CAP_OFFSET)); + ro_mask |= CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO; + rev_mask |= CXL_HDM_DECODER_CTRL_DEVICE_RESERVED; + + if (!(hdm_gcap & CXL_HDM_DECODER_UIO_CAPABLE)) + rev_mask |= CXL_HDM_DECODER_CTRL_UIO_RESERVED; + + /* + * BI (bit 13) is RsvdP for devices without CXL.cache. HDM-D decoders + * on a CXL.mem-only device must not see BI set in shadow. + */ + if (!vdev->cxl->cache_capable) + rev_mask |= CXL_HDM_DECODER_CTRL_BI_RESERVED; + + new_val &= ~rev_mask; + cur_val &= ro_mask; + new_val = (new_val & ~ro_mask) | cur_val; + + /* + * Mirror COMMIT to COMMITTED immediately in the emulated state. + */ + if (new_val & CXL_HDM_DECODER0_CTRL_COMMIT) + new_val |= CXL_HDM_DECODER0_CTRL_COMMITTED; + else + new_val &= ~CXL_HDM_DECODER0_CTRL_COMMITTED; + + *hdm_reg_ptr(vdev->cxl, offset) = cpu_to_le32(new_val); + + return size; +} + +/* + * Dispatch table for COMP_REGS region writes. Indexed by byte offset within + * the HDM decoder block. Returns the appropriate write handler. + * + * Layout: + * 0x00 HDM Decoder Capability (RO) + * 0x04 HDM Global Control (RW with reserved masking) + * 0x08-0x0f (reserved) (ignored) + * Per decoder N, base = 0x10 + N*0x20: + * base+0x00 BASE_LO (RW, [27:0] reserved) + * base+0x04 BASE_HI (RW) + * base+0x08 SIZE_LO (RW, [27:0] reserved) + * base+0x0c SIZE_HI (RW) + * base+0x10 CTRL (RW, complex rules) + * base+0x14 TARGET_LIST_LO (ignored for Type-2) + * base+0x18 TARGET_LIST_HI (ignored for Type-2) + * base+0x1c (reserved) (ignored) + */ +static ssize_t comp_regs_dispatch_write(struct vfio_pci_core_device *vdev, + u32 off, const __le32 *val32, u32 size) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u32 dec_base, dec_off; + + /* HDM Decoder Capability (0x00): RO */ + if (off == CXL_HDM_DECODER_CAP_OFFSET) + return size; + + /* HDM Global Control (0x04) */ + if (off == CXL_HDM_DECODER_CTRL_OFFSET) + return hdm_decoder_global_ctrl_write(vdev, val32, size); + + /* + * Offsets 0x08-0x0f are reserved per CXL 4.0 Table 8-115. + * Per-decoder registers start at 0x10, stride 0x20 + */ + if (off < CXL_HDM_DECODER_FIRST_BLOCK_OFFSET) + return size; /* reserved gap */ + + dec_base = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET; + /* + * Reject accesses beyond the last implemented HDM decoder. + * Without this check an out-of-bounds offset would silently + * corrupt comp_reg_virt[] memory past the end of the allocation. + */ + if ((off - dec_base) / CXL_HDM_DECODER_BLOCK_STRIDE >= cxl->hdm_count) + return size; + + dec_off = (off - dec_base) % CXL_HDM_DECODER_BLOCK_STRIDE; + + switch (dec_off) { + case CXL_HDM_DECODER_N_BASE_LOW_OFFSET: /* BASE_LO */ + case CXL_HDM_DECODER_N_SIZE_LOW_OFFSET: /* SIZE_LO */ + return hdm_decoder_n_lo_write(vdev, val32, off, size); + case CXL_HDM_DECODER_N_BASE_HIGH_OFFSET: /* BASE_HI */ + case CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET: /* SIZE_HI */ + { + /* Full 32-bit write, no reserved bits; frozen when COMMIT_LOCK set */ + u32 ctrl_off = off - dec_off + CXL_HDM_DECODER_N_CTRL_OFFSET; + u32 ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, ctrl_off)); + + if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) + return size; + *hdm_reg_ptr(cxl, off) = *val32; + return size; + } + case CXL_HDM_DECODER_N_CTRL_OFFSET: /* CTRL */ + return hdm_decoder_n_ctrl_write(vdev, val32, off, size); + case CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET: + case CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET: + case CXL_HDM_DECODER_N_REV_OFFSET: + return virt_hdm_rev_reg_write(vdev, val32, off, size); + default: + return size; + } +} + +/* + * vfio_cxl_comp_regs_rw - regops rw handler for + * VFIO_REGION_SUBTYPE_CXL_COMP_REGS. + * + * Reads return the emulated HDM state (comp_reg_virt[]). + * Writes go through comp_regs_dispatch_write() for bit-field enforcement. + * Only 4-byte aligned 4-byte accesses are supported (hardware requirement). + */ +static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev, + char __user *buf, size_t count, + loff_t *ppos, bool iswrite) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + size_t done = 0; + + if (!count) + return 0; + + /* Clamp to total region size: cap array prefix + HDM block */ + if (pos >= cxl->hdm_reg_offset + cxl->hdm_reg_size) + return -EINVAL; + count = min(count, + (size_t)(cxl->hdm_reg_offset + cxl->hdm_reg_size - pos)); + + while (done < count) { + u32 sz = count - done; + u32 off = pos + done; + __le32 v; + + /* Enforce exactly 4-byte, 4-byte-aligned accesses */ + if (sz != CXL_REG_SIZE_DWORD || (off & 0x3)) + return done ? (ssize_t)done : -EINVAL; + + if (iswrite) { + if (off < cxl->hdm_reg_offset) { + /* Cap array area is read-only; discard writes */ + done += sizeof(v); + continue; + } + if (copy_from_user(&v, buf + done, sizeof(v))) + return done ? (ssize_t)done : -EFAULT; + comp_regs_dispatch_write(vdev, + off - cxl->hdm_reg_offset, + &v, sizeof(v)); + } else { + /* Read from extended buffer _ covers cap array and HDM */ + v = cxl->comp_reg_virt[off / sizeof(__le32)]; + if (copy_to_user(buf + done, &v, sizeof(v))) + return done ? (ssize_t)done : -EFAULT; + } + done += sizeof(v); + } + + *ppos += done; + return done; +} + +static void vfio_cxl_comp_regs_release(struct vfio_pci_core_device *vdev, + struct vfio_pci_region *region) +{ + /* comp_reg_virt is freed in vfio_cxl_clean_virt_regs() */ +} + +static const struct vfio_pci_regops vfio_cxl_comp_regs_ops = { + .rw = vfio_cxl_comp_regs_rw, + .release = vfio_cxl_comp_regs_release, +}; + +/* + * vfio_cxl_setup_virt_regs - Allocate emulated HDM register state. + * + * Allocates comp_reg_virt as a compact __le32 array covering only + * hdm_reg_size bytes of HDM decoder registers. The initial values + * are read from hardware via the BAR ioremap established by the caller. + * + * DVSEC state is accessed via vdev->vconfig (see the following patch). + */ +int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl, + void __iomem *cap_base, + resource_size_t max_size) +{ + size_t total_size, nregs, i; + + if (WARN_ON(!cxl->hdm_reg_size)) + return -EINVAL; + + total_size = cxl->hdm_reg_offset + cxl->hdm_reg_size; + + /* + * The caller's map covers [comp_reg_offset, comp_reg_offset+max_size) + * inside the BAR; the HDM block ends at CXL_CM_OFFSET + total_size + * relative to that map. Reject HDM blocks that walk past the + * advertised map size; pci_resource_len() would happily allow a stale + * BAR-wide window and the subsequent readl()s would run off the + * ioremap range. + */ + if (CXL_CM_OFFSET + total_size > max_size) + return -ENODEV; + + nregs = total_size / sizeof(__le32); + cxl->comp_reg_virt = kcalloc(nregs, sizeof(__le32), GFP_KERNEL); + if (!cxl->comp_reg_virt) + return -ENOMEM; + + /* + * Snapshot the CXL.mem register area from the caller's mapping. + * cap_base maps the component register block from comp_reg_offset. + * The CXL.mem registers start at CXL_CM_OFFSET (= 0x1000) within that + * block; reading from cap_base + CXL_CM_OFFSET ensures comp_reg_virt[0] + * holds the CXL Capability Array Header required by guest drivers. + */ + for (i = 0; i < nregs; i++) + cxl->comp_reg_virt[i] = + cpu_to_le32(readl(cap_base + CXL_CM_OFFSET + + i * sizeof(__le32))); + + /* + * Establish persistent mapping; kept alive until + * vfio_cxl_clean_virt_regs(). + */ + cxl->hdm_iobase = ioremap(pci_resource_start(vdev->pdev, + cxl->comp_reg_bar) + + cxl->comp_reg_offset + CXL_CM_OFFSET + + cxl->hdm_reg_offset, + cxl->hdm_reg_size); + if (!cxl->hdm_iobase) { + kfree(cxl->comp_reg_virt); + cxl->comp_reg_virt = NULL; + return -ENOMEM; + } + + return 0; +} + +/* + * Called with memory_lock write side held (from vfio_cxl_reactivate_region). + * Uses the pre-established hdm_iobase, no ioremap() under the lock, + * which would deadlock on PREEMPT_RT where ioremap() can sleep. + */ +void vfio_cxl_reinit_comp_regs(struct vfio_pci_cxl_state *cxl) +{ + size_t i, nregs; + u32 n; + + if (!cxl || !cxl->comp_reg_virt || !cxl->hdm_iobase) + return; + + nregs = cxl->hdm_reg_size / sizeof(__le32); + + for (i = 0; i < nregs; i++) + *hdm_reg_ptr(cxl, i * sizeof(__le32)) = + cpu_to_le32(readl(cxl->hdm_iobase + + i * sizeof(__le32))); + + /* + * For firmware-committed decoders, clear COMMIT_LOCK (bit 8) and zero + * BASE in comp_reg_virt[] so QEMU can write the correct guest GPA via + * setup_locked_hdm() before guest DPA access begins. + * + * Check the COMMITTED bit (bit 10) directly from the freshly-snapshotted + * ctrl register rather than relying on cxl->precommitted. At probe time + * this function is called before cxl->precommitted is set (it is set + * after vfio_cxl_read_committed_decoder_size() succeeds), so using + * cxl->precommitted here would silently skip the LOCK clearing and leave + * the hardware HPA in comp_reg_virt[]. + */ + for (n = 0; n < cxl->hdm_count; n++) { + u32 ctrl_off = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET + + n * CXL_HDM_DECODER_BLOCK_STRIDE + + CXL_HDM_DECODER_N_CTRL_OFFSET; + u32 base_lo_off = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET + + n * CXL_HDM_DECODER_BLOCK_STRIDE + + CXL_HDM_DECODER_N_BASE_LOW_OFFSET; + u32 base_hi_off = CXL_HDM_DECODER_FIRST_BLOCK_OFFSET + + n * CXL_HDM_DECODER_BLOCK_STRIDE + + CXL_HDM_DECODER_N_BASE_HIGH_OFFSET; + u32 ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, ctrl_off)); + + if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)) + continue; + + if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK) { + *hdm_reg_ptr(cxl, ctrl_off) = + cpu_to_le32(ctrl & + ~CXL_HDM_DECODER0_CTRL_LOCK); + *hdm_reg_ptr(cxl, base_lo_off) = 0; + *hdm_reg_ptr(cxl, base_hi_off) = 0; + } + } +} + +void vfio_cxl_clean_virt_regs(struct vfio_pci_cxl_state *cxl) +{ + if (cxl->hdm_iobase) { + iounmap(cxl->hdm_iobase); + cxl->hdm_iobase = NULL; + } + kfree(cxl->comp_reg_virt); + cxl->comp_reg_virt = NULL; +} diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index bb03f9363d98f..8017d92e483f5 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -22,12 +22,53 @@ struct vfio_pci_cxl_state { size_t hdm_reg_size; resource_size_t comp_reg_offset; size_t comp_reg_size; + __le32 *comp_reg_virt; + void __iomem *hdm_iobase; u16 dvsec_len; u8 hdm_count; u8 comp_reg_bar; bool cache_capable; }; +/* Register access sizes */ +#define CXL_REG_SIZE_WORD 2 +#define CXL_REG_SIZE_DWORD 4 + +/* HDM Decoder - register offsets (CXL 4.0 Table 8-115) */ +#define CXL_HDM_DECODER_GLOBAL_CTRL_OFFSET 0x4 +#define CXL_HDM_DECODER_FIRST_BLOCK_OFFSET 0x10 +#define CXL_HDM_DECODER_BLOCK_STRIDE 0x20 +#define CXL_HDM_DECODER_N_BASE_LOW_OFFSET 0x0 +#define CXL_HDM_DECODER_N_BASE_HIGH_OFFSET 0x4 +#define CXL_HDM_DECODER_N_SIZE_LOW_OFFSET 0x8 +#define CXL_HDM_DECODER_N_SIZE_HIGH_OFFSET 0xc +#define CXL_HDM_DECODER_N_CTRL_OFFSET 0x10 +#define CXL_HDM_DECODER_N_TARGET_LIST_LOW_OFFSET 0x14 +#define CXL_HDM_DECODER_N_TARGET_LIST_HIGH_OFFSET 0x18 +#define CXL_HDM_DECODER_N_REV_OFFSET 0x1c + +/* + * HDM Decoder N Control emulation masks. + * + * Single-bit hardware definitions are in as + * CXL_HDM_DECODER0_CTRL_* (bits 0-14) and CXL_HDM_DECODER_*_CAP. + * The masks below express emulation policy for a CXL.mem device. + */ +#define CXL_HDM_DECODER_CTRL_RO_BITS_MASK (BIT(10) | BIT(11)) +#define CXL_HDM_DECODER_CTRL_RESERVED_MASK (BIT(15) | GENMASK(31, 28)) +#define CXL_HDM_DECODER_CTRL_DEVICE_BITS_RO BIT(12) +#define CXL_HDM_DECODER_CTRL_DEVICE_RESERVED (GENMASK(19, 16) | GENMASK(23, 20)) +#define CXL_HDM_DECODER_CTRL_UIO_RESERVED (BIT(14) | GENMASK(27, 24)) +/* + * bit 13 (BI) is RsvdP for devices without CXL.cache (Cache_Capable=0). + * HDM-D (CXL.mem only) decoders must not have BI set by the guest. + */ +#define CXL_HDM_DECODER_CTRL_BI_RESERVED BIT(13) +#define CXL_HDM_DECODER_BASE_LO_RESERVED_MASK GENMASK(27, 0) + +#define CXL_HDM_DECODER_GLOBAL_CTRL_RESERVED_MASK GENMASK(31, 2) +#define CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT BIT(0) + /* * CXL DVSEC for CXL Devices - register offsets within the DVSEC * (CXL 4.0 8.1.3). @@ -38,4 +79,11 @@ struct vfio_pci_cxl_state { /* CXL DVSEC Capability register bit 0: device supports CXL.cache (HDM-DB) */ #define CXL_DVSEC_CACHE_CAPABLE BIT(0) +int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl, + void __iomem *cap_base, + resource_size_t max_size); +void vfio_cxl_clean_virt_regs(struct vfio_pci_cxl_state *cxl); +void vfio_cxl_reinit_comp_regs(struct vfio_pci_cxl_state *cxl); + #endif /* __LINUX_VFIO_CXL_PRIV_H */ diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index c5d7cfabf5155..2510986c459ec 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2209,6 +2209,15 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) return 0; out_power: + /* + * vfio_pci_cxl_detect_and_init() may have assigned vdev->cxl and + * allocated comp_reg_virt[] / hdm_iobase / region state above. The + * normal teardown via vfio_pci_core_unregister_device() will not run + * if registration failed, so release the CXL state here. No-op when + * vdev->cxl is NULL (non-CXL device or detect skipped). + */ + vfio_pci_cxl_cleanup(vdev); + if (!disable_idle_d3) pm_runtime_get_noresume(dev); diff --git a/include/uapi/cxl/cxl_regs.h b/include/uapi/cxl/cxl_regs.h index c821ef7ec2bb7..aa9fea9b88f7f 100644 --- a/include/uapi/cxl/cxl_regs.h +++ b/include/uapi/cxl/cxl_regs.h @@ -34,8 +34,13 @@ #define CXL_HDM_DECODER_TARGET_COUNT_MASK __GENMASK(7, 4) #define CXL_HDM_DECODER_INTERLEAVE_11_8 _BITUL(8) #define CXL_HDM_DECODER_INTERLEAVE_14_12 _BITUL(9) +#define CXL_HDM_DECODER_POISON_ON_DECODE_ERR _BITUL(10) #define CXL_HDM_DECODER_INTERLEAVE_3_6_12_WAY _BITUL(11) #define CXL_HDM_DECODER_INTERLEAVE_16_WAY _BITUL(12) +#define CXL_HDM_DECODER_UIO_CAPABLE _BITUL(13) +#define CXL_HDM_DECODER_UIO_COUNT_MASK __GENMASK(19, 16) +#define CXL_HDM_DECODER_MEMDATA_NXM _BITUL(20) +#define CXL_HDM_DECODER_COHERENCY_MODELS_MASK __GENMASK(22, 21) #define CXL_HDM_DECODER_CTRL_OFFSET 0x4 #define CXL_HDM_DECODER_ENABLE _BITUL(1) #define CXL_HDM_DECODER0_BASE_LOW_OFFSET(i) (0x20 * (i) + 0x10) From e266bd361400f726838f9e5d470de06a0beea72f Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:09 +0530 Subject: [PATCH 398/464] NVIDIA: VR: SAUCE: vfio/cxl: Wait for HDM ranges and create memdev BugLink: https://bugs.launchpad.net/bugs/2152222 After HDM registers are mapped, call cxl_await_range_active() so we only proceed when DVSEC ranges report active, avoiding access to the memdev register group that Type-2 devices may lack. This wait is required before re-snapshotting component registers: firmware commits final HDM decoder values such as SIZE_HIGH only after MEM_ACTIVE. Once cxl_await_range_active() confirms that state, re-read component regs with vfio_cxl_reinit_comp_regs() so those committed values land in comp_reg_virt. Read committed decoder size from hardware, set capacity via cxl_set_capacity(), and devm_cxl_add_memdev(). Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (cherry-picked from commit 537d8a2414cf52e363f0d82ea34b98647e8af2f7 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Line offset adjustments only (cascading from 0011 changes)] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 56 ++++++++++++++++++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_emu.c | 42 +++++++++++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_priv.h | 4 ++ 3 files changed, 102 insertions(+) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 72033fda7954d..3279d2a89feb8 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -170,6 +170,22 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_cxl_create_memdev(struct vfio_pci_cxl_state *cxl, + resource_size_t capacity) +{ + int ret; + + ret = cxl_set_capacity(&cxl->cxlds, capacity); + if (ret) + return ret; + + cxl->cxlmd = devm_cxl_add_memdev(&cxl->cxlds, NULL); + if (IS_ERR(cxl->cxlmd)) + return PTR_ERR(cxl->cxlmd); + + return 0; +} + /* * Free CXL state early on probe failure. devm_kfree() on a live devres * allocation removes it from the list immediately, so the normal devres @@ -194,6 +210,7 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { struct pci_dev *pdev = vdev->pdev; struct vfio_pci_cxl_state *cxl; + resource_size_t capacity = 0; u16 dvsec; int ret; @@ -239,8 +256,44 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) goto free_cxl; } + cxl->cxlds.media_ready = !cxl_await_range_active(&cxl->cxlds); + if (!cxl->cxlds.media_ready) { + pci_warn(pdev, "CXL media not ready\n"); + pci_disable_device(pdev); + goto regs_failed; + } + + /* + * Take the single authoritative HDM decoder snapshot now that + * MEM_ACTIVE is confirmed and BAR memory is still enabled. Using + * readl() per-dword ensures correct MMIO serialisation and captures + * the final firmware-written values for all fields including SIZE_HIGH, + * which firmware commits to the BAR at MEM_ACTIVE time. + */ + vfio_cxl_reinit_comp_regs(cxl); + pci_disable_device(pdev); + capacity = vfio_cxl_read_committed_decoder_size(vdev, cxl); + if (capacity == 0) { + /* + * TODO: Add handling for devices which do not have + * firmware pre-committed decoders + */ + pci_info(pdev, "Uncommitted region size must be configured via sysfs before bind\n"); + goto regs_failed; + } + + cxl->dpa_size = capacity; + + pci_dbg(pdev, "Device capacity: %llu MB\n", capacity >> 20); + + ret = vfio_cxl_create_memdev(cxl, capacity); + if (ret) { + pci_warn(pdev, "Failed to create memdev\n"); + goto regs_failed; + } + /* * Register probing succeeded. Assign vdev->cxl now so that * all subsequent helpers can access state via vdev->cxl. @@ -251,6 +304,9 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) return; +regs_failed: + vfio_cxl_clean_virt_regs(cxl); + free_cxl: vfio_cxl_dev_state_free(pdev, cxl); } diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c index 5626ab51a053e..5cd0dfdcb7148 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_emu.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -394,6 +394,48 @@ int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev, return 0; } +/* + * vfio_cxl_read_committed_decoder_size - Extract committed DPA capacity from + * comp_reg_virt[]. + * + * Called from probe context after vfio_cxl_reinit_comp_regs() has taken the + * post-MEM_ACTIVE readl() snapshot and patched SIZE_HIGH/SIZE_LOW from DVSEC. + * comp_reg_virt[] is already correct at this point; no hardware access needed. + * + * Returns the committed DPA capacity in bytes, or 0 if the decoder is not + * committed. + */ +resource_size_t +vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl) +{ + struct pci_dev *pdev = vdev->pdev; + resource_size_t capacity; + u32 ctrl, sz_hi, sz_lo; + + if (WARN_ON(!cxl || !cxl->comp_reg_virt)) + return 0; + + ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, CXL_HDM_DECODER0_CTRL_OFFSET(0))); + sz_hi = le32_to_cpu(*hdm_reg_ptr(cxl, CXL_HDM_DECODER0_SIZE_HIGH_OFFSET(0))); + sz_lo = le32_to_cpu(*hdm_reg_ptr(cxl, CXL_HDM_DECODER0_SIZE_LOW_OFFSET(0))); + + if (!(ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED)) { + pci_dbg(pdev, + "vfio_cxl: decoder0 not committed: ctrl=0x%08x\n", + ctrl); + return 0; + } + + capacity = ((resource_size_t)sz_hi << 32) | (sz_lo & GENMASK(31, 28)); + + pci_dbg(pdev, + "vfio_cxl: decoder0 committed: sz_hi=0x%08x sz_lo=0x%08x capacity=0x%llx\n", + sz_hi, sz_lo, (unsigned long long)capacity); + + return capacity; +} + /* * Called with memory_lock write side held (from vfio_cxl_reactivate_region). * Uses the pre-established hdm_iobase, no ioremap() under the lock, diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index 8017d92e483f5..a9b244f1624c8 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -23,6 +23,7 @@ struct vfio_pci_cxl_state { resource_size_t comp_reg_offset; size_t comp_reg_size; __le32 *comp_reg_virt; + size_t dpa_size; void __iomem *hdm_iobase; u16 dvsec_len; u8 hdm_count; @@ -85,5 +86,8 @@ int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev, resource_size_t max_size); void vfio_cxl_clean_virt_regs(struct vfio_pci_cxl_state *cxl); void vfio_cxl_reinit_comp_regs(struct vfio_pci_cxl_state *cxl); +resource_size_t +vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl); #endif /* __LINUX_VFIO_CXL_PRIV_H */ From 68fe9376935b25e1a1c610cac4ef834f8d4e423e Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:10 +0530 Subject: [PATCH 399/464] NVIDIA: VR: SAUCE: vfio/cxl: CXL region management support BugLink: https://bugs.launchpad.net/bugs/2152222 Region Management makes use of APIs provided by CXL_CORE as below: CREATE_REGION flow: 1. Validate request (size, decoder availability) 2. Allocate HPA via cxl_get_hpa_freespace() 3. Allocate DPA via cxl_request_dpa() 4. Create region via cxl_create_region() - commits HDM decoder 5. Get HPA range via cxl_get_region_range() DESTROY_REGION flow: 1. Detach decoder via cxl_decoder_detach() 2. Free DPA via cxl_dpa_free() 3. Release root decoder via cxl_put_root_decoder() Use DEFINE_FREE scope helpers so error paths unwind cleanly. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 799c46dc149544101e6293d3f600a4ba115a270f from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Add borrowed-reference comment for precommitted decoders, init region to NULL, do not unregister precommitted regions in teardown] [kobak: Restored BOS CXL helper providers/exports and vfio-pci-core CXL namespace import so the region-management backport builds against BOS CXL core.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/hdm.c | 97 +++++++++++ drivers/cxl/core/region.c | 251 +++++++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + drivers/vfio/pci/cxl/vfio_cxl_core.c | 122 +++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_priv.h | 8 + drivers/vfio/pci/vfio_pci_core.c | 1 + include/cxl/cxl.h | 29 ++++ 7 files changed, 509 insertions(+) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 64d96b3310e73..603cc746aa497 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -565,6 +565,7 @@ int cxl_dpa_free(struct cxl_endpoint_decoder *cxled) devm_cxl_dpa_release(cxled); return 0; } +EXPORT_SYMBOL_NS_GPL(cxl_dpa_free, "CXL"); int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled, enum cxl_partition_mode mode) @@ -596,6 +597,64 @@ int cxl_dpa_set_part(struct cxl_endpoint_decoder *cxled, return 0; } +static int find_free_decoder(struct device *dev, const void *data) +{ + struct cxl_endpoint_decoder *cxled; + struct cxl_port *port; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled = to_cxl_endpoint_decoder(dev); + port = cxled_to_port(cxled); + + return cxled->cxld.id == (port->hdm_end + 1); +} + +static struct cxl_endpoint_decoder * +cxl_find_free_decoder(struct cxl_memdev *cxlmd) +{ + struct cxl_port *endpoint = cxlmd->endpoint; + struct device *dev; + + guard(rwsem_read)(&cxl_rwsem.dpa); + dev = device_find_child(&endpoint->dev, NULL, find_free_decoder); + if (!dev) + return NULL; + + return to_cxl_endpoint_decoder(dev); +} + +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd, + enum cxl_partition_mode mode, + resource_size_t alloc) +{ + struct cxl_endpoint_decoder *cxled; + int rc; + + if (!IS_ALIGNED(alloc, SZ_256M)) + return ERR_PTR(-EINVAL); + + cxled = cxl_find_free_decoder(cxlmd); + if (!cxled) + return ERR_PTR(-ENODEV); + + rc = cxl_dpa_set_part(cxled, mode); + if (rc) + goto err_put; + + rc = cxl_dpa_alloc(cxled, alloc); + if (rc) + goto err_put; + + return cxled; + +err_put: + put_device(&cxled->cxld.dev); + return ERR_PTR(rc); +} +EXPORT_SYMBOL_NS_GPL(cxl_request_dpa, "CXL"); + static int __cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size) { struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); @@ -679,6 +738,44 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, u64 size) return devm_add_action_or_reset(&port->dev, cxl_dpa_release, cxled); } +static int find_committed_endpoint_decoder(struct device *dev, const void *data) +{ + struct cxl_endpoint_decoder *cxled; + struct cxl_port *port; + + if (!is_endpoint_decoder(dev)) + return 0; + + cxled = to_cxl_endpoint_decoder(dev); + port = cxled_to_port(cxled); + + return cxled->cxld.id == port->hdm_end; +} + +struct cxl_endpoint_decoder *cxl_get_committed_decoder(struct cxl_memdev *cxlmd, + struct cxl_region **cxlr) +{ + struct cxl_port *endpoint = cxlmd->endpoint; + struct cxl_endpoint_decoder *cxled; + struct device *cxled_dev; + + if (!endpoint) + return NULL; + + guard(rwsem_read)(&cxl_rwsem.dpa); + cxled_dev = device_find_child(&endpoint->dev, NULL, + find_committed_endpoint_decoder); + if (!cxled_dev) + return NULL; + + cxled = to_cxl_endpoint_decoder(cxled_dev); + *cxlr = cxled->cxld.region; + + put_device(cxled_dev); + return cxled; +} +EXPORT_SYMBOL_NS_GPL(cxl_get_committed_decoder, "CXL"); + static void cxld_set_interleave(struct cxl_decoder *cxld, u32 *ctrl) { u16 eig; diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 455a2c090589b..e820fb63b95aa 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -710,6 +710,141 @@ static int free_hpa(struct cxl_region *cxlr) return 0; } +struct cxlrd_max_context { + struct device * const *host_bridges; + int interleave_ways; + unsigned long flags; + resource_size_t max_hpa; + struct cxl_root_decoder *cxlrd; +}; + +static int find_max_hpa(struct device *dev, void *data) +{ + struct cxlrd_max_context *ctx = data; + struct cxl_switch_decoder *cxlsd; + struct cxl_root_decoder *cxlrd; + struct resource *res, *prev; + struct cxl_decoder *cxld; + resource_size_t free = 0; + resource_size_t max; + int found = 0; + + if (!is_root_decoder(dev)) + return 0; + + cxlrd = to_cxl_root_decoder(dev); + cxlsd = &cxlrd->cxlsd; + cxld = &cxlsd->cxld; + + if ((cxld->flags & ctx->flags) != ctx->flags) { + dev_dbg(dev, "flags not matching: %08lx vs %08lx\n", + cxld->flags, ctx->flags); + return 0; + } + + for (int i = 0; i < ctx->interleave_ways; i++) { + for (int j = 0; j < ctx->interleave_ways; j++) { + if (ctx->host_bridges[i] == cxlsd->target[j]->dport_dev) { + found++; + break; + } + } + } + + if (found != ctx->interleave_ways) { + dev_dbg(dev, + "Not enough host bridges. Found %d for %d interleave ways requested\n", + found, ctx->interleave_ways); + return 0; + } + + lockdep_assert_held_read(&cxl_rwsem.region); + res = cxlrd->res->child; + + if (!res) + max = resource_size(cxlrd->res); + else + max = 0; + + for (prev = NULL; res; prev = res, res = res->sibling) { + if (!prev && res->start == cxlrd->res->start && + res->end == cxlrd->res->end) { + max = resource_size(cxlrd->res); + break; + } + if (prev && !resource_size(prev)) + continue; + + if (!prev && res->start > cxlrd->res->start) { + free = res->start - cxlrd->res->start; + max = max(free, max); + } + if (prev && res->start > prev->end + 1) { + free = res->start - prev->end + 1; + max = max(free, max); + } + } + + if (prev && prev->end + 1 < cxlrd->res->end + 1) { + free = cxlrd->res->end + 1 - prev->end + 1; + max = max(free, max); + } + + dev_dbg(&cxlrd->cxlsd.cxld.dev, "found %pa bytes of free space\n", &max); + if (max > ctx->max_hpa) { + if (ctx->cxlrd) + put_device(&ctx->cxlrd->cxlsd.cxld.dev); + get_device(&cxlrd->cxlsd.cxld.dev); + ctx->cxlrd = cxlrd; + ctx->max_hpa = max; + } + return 0; +} + +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd, + int interleave_ways, + unsigned long flags, + resource_size_t *max_avail_contig) +{ + struct cxlrd_max_context ctx = { + .flags = flags, + .interleave_ways = interleave_ways, + }; + struct cxl_port *root_port; + struct cxl_port *endpoint; + + endpoint = cxlmd->endpoint; + if (!endpoint) { + dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n"); + return ERR_PTR(-ENXIO); + } + + ctx.host_bridges = &endpoint->host_bridge; + + struct cxl_root *root __free(put_cxl_root) = find_cxl_root(endpoint); + if (!root) { + dev_dbg(&endpoint->dev, "endpoint is not related to a root port\n"); + return ERR_PTR(-ENXIO); + } + + root_port = &root->port; + scoped_guard(rwsem_read, &cxl_rwsem.region) + device_for_each_child(&root_port->dev, &ctx, find_max_hpa); + + if (!ctx.cxlrd) + return ERR_PTR(-ENOMEM); + + *max_avail_contig = ctx.max_hpa; + return ctx.cxlrd; +} +EXPORT_SYMBOL_NS_GPL(cxl_get_hpa_freespace, "CXL"); + +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd) +{ + put_device(&cxlrd->cxlsd.cxld.dev); +} +EXPORT_SYMBOL_NS_GPL(cxl_put_root_decoder, "CXL"); + static ssize_t size_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { @@ -2520,6 +2655,27 @@ static void cxl_region_release_action(struct cxl_region *cxlr) unregister_region(cxlr); } +void cxl_unregister_region(struct cxl_region *cxlr) +{ + cxl_region_release_action(cxlr); +} +EXPORT_SYMBOL_NS_GPL(cxl_unregister_region, "CXL"); + +int cxl_get_region_range(struct cxl_region *region, struct range *range) +{ + if (WARN_ON_ONCE(!region)) + return -ENODEV; + + if (!region->params.res) + return -ENOSPC; + + range->start = region->params.res->start; + range->end = region->params.res->end; + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_get_region_range, "CXL"); + static struct lock_class_key cxl_region_key; static struct cxl_region *cxl_region_alloc(struct cxl_root_decoder *cxlrd, int id) @@ -3992,6 +4148,101 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, return cxlr; } +DEFINE_FREE(cxl_region_release, struct cxl_region *, + if (!IS_ERR_OR_NULL(_T)) cxl_region_release_action(_T)) + +static struct cxl_region * +__construct_new_region(struct cxl_root_decoder *cxlrd, + struct cxl_endpoint_decoder **cxled, int ways) +{ + struct cxl_memdev *cxlmd = cxled_to_memdev(cxled[0]); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld; + struct cxl_region_params *p; + resource_size_t size = 0; + int rc, i, part = READ_ONCE(cxled[0]->part); + + if (part < 0 || part >= cxlds->nr_partitions) { + dev_err(cxlmd->dev.parent, + "%s:%s: invalid partition index %d (max %u)\n", + dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev), + part, cxlds->nr_partitions); + return ERR_PTR(-ENXIO); + } + + struct cxl_region *cxlr __free(cxl_region_release) = + __create_region(cxlrd, cxlds->part[part].mode, + atomic_read(&cxlrd->region_id), + cxled[0]->cxld.target_type); + if (IS_ERR(cxlr)) + return cxlr; + + guard(rwsem_write)(&cxl_rwsem.region); + + p = &cxlr->params; + if (p->state >= CXL_CONFIG_INTERLEAVE_ACTIVE) { + dev_err(cxlmd->dev.parent, + "%s:%s: %s unexpected region state\n", + dev_name(&cxlmd->dev), dev_name(&cxled[0]->cxld.dev), + __func__); + return ERR_PTR(-EBUSY); + } + + if (ways < 1) + return ERR_PTR(-EINVAL); + + p->interleave_ways = ways; + p->interleave_granularity = cxld->interleave_granularity; + + scoped_guard(rwsem_read, &cxl_rwsem.dpa) { + for (i = 0; i < ways; i++) { + if (!cxled[i]->dpa_res) + return ERR_PTR(-EINVAL); + size += resource_size(cxled[i]->dpa_res); + } + + rc = alloc_hpa(cxlr, size); + if (rc) + return ERR_PTR(rc); + + for (i = 0; i < ways; i++) { + rc = cxl_region_attach(cxlr, cxled[i], 0); + if (rc) + return ERR_PTR(rc); + } + } + + rc = cxl_region_decode_commit(cxlr); + if (rc) + return ERR_PTR(rc); + + p->state = CXL_CONFIG_COMMIT; + + return no_free_ptr(cxlr); +} + +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd, + struct cxl_endpoint_decoder **cxled, + int ways) +{ + struct cxl_region *cxlr; + + mutex_lock(&cxlrd->range_lock); + cxlr = __construct_new_region(cxlrd, cxled, ways); + mutex_unlock(&cxlrd->range_lock); + if (IS_ERR(cxlr)) + return cxlr; + + if (device_attach(&cxlr->dev) <= 0) { + dev_err(&cxlr->dev, "failed to create region\n"); + cxl_region_release_action(cxlr); + return ERR_PTR(-ENODEV); + } + + return cxlr; +} +EXPORT_SYMBOL_NS_GPL(cxl_create_region, "CXL"); + static struct cxl_region * cxl_find_region_by_range(struct cxl_root_decoder *cxlrd, struct range *hpa_range) diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 5357868600494..c52676d604976 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -13,6 +13,7 @@ #include #include #include +#include extern const struct nvdimm_security_ops *cxl_security_ops; diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 3279d2a89feb8..3e2cb2e67f8b0 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -21,6 +21,13 @@ #include "../vfio_pci_priv.h" #include "vfio_cxl_priv.h" +/* + * Scope-based cleanup wrappers for the CXL resource APIs + */ +DEFINE_FREE(cxl_put_root_decoder, struct cxl_root_decoder *, if (!IS_ERR_OR_NULL(_T)) cxl_put_root_decoder(_T)) +DEFINE_FREE(cxl_dpa_free, struct cxl_endpoint_decoder *, if (!IS_ERR_OR_NULL(_T)) cxl_dpa_free(_T)) +DEFINE_FREE(cxl_unregister_region, struct cxl_region *, if (!IS_ERR_OR_NULL(_T)) cxl_unregister_region(_T)) + /* * vfio_cxl_create_device_state - Allocate and validate CXL device state * @@ -170,6 +177,115 @@ static int vfio_cxl_setup_regs(struct vfio_pci_core_device *vdev, return ret; } +int vfio_cxl_create_cxl_region(struct vfio_pci_cxl_state *cxl, + resource_size_t size) +{ + resource_size_t max_size; + + struct cxl_root_decoder *cxlrd __free(cxl_put_root_decoder) = + cxl_get_hpa_freespace(cxl->cxlmd, 1, + CXL_DECODER_F_RAM | CXL_DECODER_F_TYPE2, + &max_size); + if (IS_ERR(cxlrd)) + return PTR_ERR(cxlrd); + + /* Insufficient HPA space; cxlrd freed automatically by __free() */ + if (max_size < size) + return -ENOSPC; + + struct cxl_endpoint_decoder *cxled __free(cxl_dpa_free) = + cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM, size); + if (IS_ERR(cxled)) + return PTR_ERR(cxled); + + struct cxl_region *region __free(cxl_unregister_region) = + cxl_create_region(cxlrd, &cxled, 1); + if (IS_ERR(region)) + return PTR_ERR(region); + + /* All operations succeeded; transfer ownership to cxl state */ + cxl->cxlrd = no_free_ptr(cxlrd); + cxl->cxled = no_free_ptr(cxled); + cxl->region = no_free_ptr(region); + + return 0; +} + +void vfio_cxl_destroy_cxl_region(struct vfio_pci_cxl_state *cxl) +{ + if (!cxl->region) + return; + + /* + * Precommitted regions are obtained via cxl_get_committed_decoder() as + * a borrowed reference owned by the cxl core; do not unregister or + * free the decoder objects from here. Only vfio_cxl_create_cxl_region() + * owns the region and decoders. + */ + if (!cxl->precommitted) { + cxl_unregister_region(cxl->region); + cxl_dpa_free(cxl->cxled); + cxl_put_root_decoder(cxl->cxlrd); + } + + cxl->region = NULL; + cxl->cxled = NULL; + cxl->cxlrd = NULL; +} + +static int vfio_cxl_create_region_helper(struct vfio_pci_core_device *vdev, + struct vfio_pci_cxl_state *cxl, + resource_size_t capacity) +{ + struct pci_dev *pdev = vdev->pdev; + struct range range; + int ret; + + if (cxl->precommitted) { + struct cxl_endpoint_decoder *cxled; + /* + * cxl_get_committed_decoder() does not write *region on every + * failure path (e.g. when cxlmd->endpoint is NULL or no decoder + * is committed). Initialise to NULL so the !cxl->region check + * below catches it regardless of stack-init mode. + */ + struct cxl_region *region = NULL; + + cxled = cxl_get_committed_decoder(cxl->cxlmd, ®ion); + if (IS_ERR(cxled)) + return PTR_ERR(cxled); + cxl->cxled = cxled; + cxl->region = region; + } else { + ret = vfio_cxl_create_cxl_region(cxl, capacity); + if (ret) + return ret; + } + + if (!cxl->region) { + pci_err(pdev, "Failed to create CXL region\n"); + ret = -ENODEV; + goto failed; + } + + ret = cxl_get_region_range(cxl->region, &range); + if (ret) + goto failed; + + cxl->region_hpa = range.start; + cxl->region_size = range_len(&range); + + pci_dbg(pdev, "CXL region: HPA 0x%llx size %lu MB\n", + cxl->region_hpa, cxl->region_size >> 20); + + return 0; + +failed: + vfio_cxl_destroy_cxl_region(cxl); + + return ret; +} + static int vfio_cxl_create_memdev(struct vfio_pci_cxl_state *cxl, resource_size_t capacity) { @@ -284,6 +400,7 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) goto regs_failed; } + cxl->precommitted = true; cxl->dpa_size = capacity; pci_dbg(pdev, "Device capacity: %llu MB\n", capacity >> 20); @@ -294,6 +411,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) goto regs_failed; } + ret = vfio_cxl_create_region_helper(vdev, cxl, capacity); + if (ret) + goto regs_failed; + /* * Register probing succeeded. Assign vdev->cxl now so that * all subsequent helpers can access state via vdev->cxl. @@ -319,6 +440,7 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) return; vfio_cxl_clean_virt_regs(cxl); + vfio_cxl_destroy_cxl_region(cxl); } MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index a9b244f1624c8..e7d4f38d03be0 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -18,6 +18,10 @@ struct vfio_pci_cxl_state { struct cxl_memdev *cxlmd; struct cxl_root_decoder *cxlrd; struct cxl_endpoint_decoder *cxled; + struct cxl_region *region; + resource_size_t region_hpa; + size_t region_size; + void *region_vaddr; resource_size_t hdm_reg_offset; size_t hdm_reg_size; resource_size_t comp_reg_offset; @@ -29,6 +33,7 @@ struct vfio_pci_cxl_state { u8 hdm_count; u8 comp_reg_bar; bool cache_capable; + bool precommitted; }; /* Register access sizes */ @@ -89,5 +94,8 @@ void vfio_cxl_reinit_comp_regs(struct vfio_pci_cxl_state *cxl); resource_size_t vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev, struct vfio_pci_cxl_state *cxl); +int vfio_cxl_create_cxl_region(struct vfio_pci_cxl_state *cxl, + resource_size_t size); +void vfio_cxl_destroy_cxl_region(struct vfio_pci_cxl_state *cxl); #endif /* __LINUX_VFIO_CXL_PRIV_H */ diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2510986c459ec..c48da03089801 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2604,3 +2604,4 @@ module_exit(vfio_pci_core_cleanup); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR(DRIVER_AUTHOR); MODULE_DESCRIPTION(DRIVER_DESC); +MODULE_IMPORT_NS("CXL"); diff --git a/include/cxl/cxl.h b/include/cxl/cxl.h index 6fd6efaaef06a..0c41508828382 100644 --- a/include/cxl/cxl.h +++ b/include/cxl/cxl.h @@ -177,6 +177,18 @@ struct cxl_dpa_partition { #define CXL_NR_PARTITIONS_MAX 2 +/* + * cxl_decoder flags that define the type of memory / devices this decoder + * supports as well as configuration lock status. + */ +#define CXL_DECODER_F_RAM BIT(0) +#define CXL_DECODER_F_PMEM BIT(1) +#define CXL_DECODER_F_TYPE2 BIT(2) +#define CXL_DECODER_F_TYPE3 BIT(3) +#define CXL_DECODER_F_LOCK BIT(4) +#define CXL_DECODER_F_ENABLE BIT(5) +#define CXL_DECODER_F_MASK GENMASK(5, 0) + struct cxl_memdev_attach { int (*probe)(struct cxl_memdev *cxlmd); }; @@ -274,6 +286,23 @@ int cxl_set_capacity(struct cxl_dev_state *cxlds, u64 capacity); struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds, const struct cxl_memdev_attach *attach); struct cxl_region; +struct cxl_endpoint_decoder *cxl_get_committed_decoder(struct cxl_memdev *cxlmd, + struct cxl_region **cxlr); +int cxl_get_region_range(struct cxl_region *region, struct range *range); +void cxl_unregister_region(struct cxl_region *cxlr); +struct cxl_port; +struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd, + int interleave_ways, + unsigned long flags, + resource_size_t *max); +void cxl_put_root_decoder(struct cxl_root_decoder *cxlrd); +struct cxl_endpoint_decoder *cxl_request_dpa(struct cxl_memdev *cxlmd, + enum cxl_partition_mode mode, + resource_size_t alloc); +int cxl_dpa_free(struct cxl_endpoint_decoder *cxled); +struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd, + struct cxl_endpoint_decoder **cxled, + int ways); int cxl_memdev_attach_region(struct cxl_memdev *cxlmd, struct cxl_attach_region *attach); #ifdef CONFIG_CXL_REGION From 2a0bb4203a2d1a54264142dc352cf2d3ad7684e3 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:11 +0530 Subject: [PATCH 400/464] NVIDIA: VR: SAUCE: vfio/cxl: DPA VFIO region with demand fault mmap and reset zap BugLink: https://bugs.launchpad.net/bugs/2152222 Wire the CXL DPA range up as a VFIO demand-paged region so QEMU can mmap guest device memory directly. Faults call vmf_insert_pfn() to insert one PFN at a time rather than mapping the full range upfront. CXL region lifecycle: - The CXL memory region is registered with VFIO layer during vfio_pci_open_device - mmap() establishes the VMA with vm_ops but inserts no PTEs - Each guest page fault calls vfio_cxl_region_page_fault() which inserts a single PFN under the memory_lock read side - On device reset, vfio_cxl_zap_region_locked() sets region_active=false and calls unmap_mapping_range() to invalidate all DPA PTEs atomically while holding memory_lock for writing - Faults racing with reset see region_active==false and return VM_FAULT_SIGBUS - vfio_cxl_reactivate_region() restores region_active after successful hardware reset Also integrate the zap/reactivate calls into vfio_pci_ioctl_reset() so that FLR correctly invalidates DPA mappings and restores them on success. Co-developed-by: Zhi Wang Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit f5e419121227 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve context mismatches in vfio_pci_core.c and vfio_pci_priv.h due to missing upstream dmabuf support in NV-Kernels base, Add vdev back-pointer in cxl_state, hold memory_lock read-side in fault/rw paths, advance *ppos in region rw, add vfio_direct_config_read export and use it instead of vfio_raw_config_read in DVSEC fallback] [kobak: Preserved existing VFIO PCI DMABUF reset movement while adding CXL DPA zap/reactivation.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 232 ++++++++++++++++++++++++++- drivers/vfio/pci/cxl/vfio_cxl_emu.c | 2 +- drivers/vfio/pci/cxl/vfio_cxl_priv.h | 14 +- drivers/vfio/pci/vfio_pci_core.c | 11 ++ drivers/vfio/pci/vfio_pci_priv.h | 6 + 5 files changed, 262 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 3e2cb2e67f8b0..04f6f5cb47f6d 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -419,8 +419,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) * Register probing succeeded. Assign vdev->cxl now so that * all subsequent helpers can access state via vdev->cxl. * All failure paths below clear vdev->cxl before calling - * vfio_cxl_dev_state_free(). + * vfio_cxl_dev_state_free(). cxl->vdev is the back-pointer used + * by vm_fault and other helpers that only have the cxl state in hand. */ + cxl->vdev = vdev; vdev->cxl = cxl; return; @@ -443,4 +445,232 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) vfio_cxl_destroy_cxl_region(cxl); } +static vm_fault_t vfio_cxl_region_vm_fault(struct vm_fault *vmf) +{ + struct vfio_pci_region *region = vmf->vma->vm_private_data; + struct vfio_pci_cxl_state *cxl = region->data; + struct vfio_pci_core_device *vdev = cxl->vdev; + unsigned long pgoff; + unsigned long pfn; + vm_fault_t ret; + + /* + * Hold memory_lock read side across the region_active check and the + * vmf_insert_pfn so the reset path cannot run unmap_mapping_range + * between the two and leave a stale PTE pointing at the pre-reset HPA. + * vfio_cxl_prepare_reset holds the write side while it clears + * region_active and zaps existing PTEs. + */ + down_read(&vdev->memory_lock); + + if (!cxl->region_active) { + ret = VM_FAULT_SIGBUS; + goto out; + } + + pgoff = vmf->pgoff & + ((1UL << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + + if (pgoff >= (cxl->region_size >> PAGE_SHIFT)) { + ret = VM_FAULT_SIGBUS; + goto out; + } + + pfn = PHYS_PFN(cxl->region_hpa) + pgoff; + ret = vmf_insert_pfn(vmf->vma, vmf->address, pfn); + +out: + up_read(&vdev->memory_lock); + return ret; +} + +static const struct vm_operations_struct vfio_cxl_region_vm_ops = { + .fault = vfio_cxl_region_vm_fault, +}; + +static int vfio_cxl_region_mmap(struct vfio_pci_core_device *vdev, + struct vfio_pci_region *region, + struct vm_area_struct *vma) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u64 req_len, pgoff, end; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_MMAP)) + return -EINVAL; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_READ) && + (vma->vm_flags & VM_READ)) + return -EPERM; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_WRITE) && + (vma->vm_flags & VM_WRITE)) + return -EPERM; + + pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + + if (check_sub_overflow(vma->vm_end, vma->vm_start, &req_len) || + check_add_overflow(PFN_PHYS(pgoff), req_len, &end)) + return -EOVERFLOW; + + if (end > cxl->region_size) + return -EINVAL; + + vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); + + vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED | VM_IO | VM_PFNMAP | + VM_DONTEXPAND | VM_DONTDUMP); + + vma->vm_ops = &vfio_cxl_region_vm_ops; + vma->vm_private_data = region; + + return 0; +} + +/* + * vfio_cxl_zap_region_locked - Invalidate all DPA region PTEs. + * + * Must be called with vdev->memory_lock held for writing. Sets + * region_active=false before zapping so any subsequent I/O to the region + * sees the inactive state and returns an error rather than accessing + * stale mappings. + */ +void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + lockdep_assert_held_write(&vdev->memory_lock); + + if (!cxl) + return; + + WRITE_ONCE(cxl->region_active, false); +} + +/* + * vfio_cxl_reactivate_region - Re-enable DPA region after successful reset. + * + * Must be called with vdev->memory_lock held for writing. Re-reads the + * HDM decoder state from hardware (FLR cleared it) and sets region_active + * so that subsequent I/O to the region is permitted again. + */ +void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + lockdep_assert_held_write(&vdev->memory_lock); + + if (!cxl) + return; + /* + * Re-initialise the emulated HDM comp_reg_virt[] from hardware. + * After FLR the decoder registers read as zero; mirror that in + * the emulated state so QEMU sees a clean slate. + */ + vfio_cxl_reinit_comp_regs(cxl); + + /* + * Only re-enable the DPA mmap if the hardware has actually + * re-committed decoder 0 after FLR. Read the COMMITTED bit from the + * freshly-re-snapshotted comp_reg_virt[] so we check the post-FLR + * hardware state, not stale pre-reset state. + * + * If COMMITTED is 0 (slow firmware re-commit path), leave + * region_active=false. Guest faults will return VM_FAULT_SIGBUS + * until the decoder is re-committed and the region is re-enabled. + */ + if (cxl->precommitted && cxl->comp_reg_virt) { + /* + * Read CTRL via the full CXL.mem-relative index: hdm_reg_offset + * (now CXL.mem-relative) plus the within-HDM-block offset. + */ + u32 ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_CTRL_OFFSET(0))); + + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) + WRITE_ONCE(cxl->region_active, true); + } +} + +static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev, + char __user *buf, size_t count, loff_t *ppos, + bool iswrite) +{ + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS; + struct vfio_pci_cxl_state *cxl = core_dev->region[i].data; + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + ssize_t ret; + + if (!count || pos >= cxl->region_size) + return 0; + + /* + * Hold memory_lock read side across the region_active check and the + * user copy. vfio_cxl_prepare_reset() holds the write side while it + * clears region_active and unmaps the inode range; without the read + * side here, the copy could still touch cxl->region_vaddr after the + * reset has begun. Guard against access after a failed reset + * (region_active=false) or a release race (region_vaddr=NULL): either + * means the memremap'd window is no longer valid; touching it would + * produce a Synchronous External Abort. + */ + down_read(&core_dev->memory_lock); + + if (!cxl->region_active || !cxl->region_vaddr) { + ret = -EIO; + goto out; + } + + count = min(count, (size_t)(cxl->region_size - pos)); + + if (iswrite) { + if (copy_from_user(cxl->region_vaddr + pos, buf, count)) { + ret = -EFAULT; + goto out; + } + } else { + if (copy_to_user(buf, cxl->region_vaddr + pos, count)) { + ret = -EFAULT; + goto out; + } + } + + /* + * vfio_pci_rw() returns the region rw result verbatim and relies on + * the handler to advance *ppos. Without this, successive read/write + * syscalls on the DPA region keep operating at the same offset + * instead of advancing. + */ + *ppos += count; + ret = count; + +out: + up_read(&core_dev->memory_lock); + return ret; +} + +static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev, + struct vfio_pci_region *region) +{ + struct vfio_pci_cxl_state *cxl = region->data; + + /* + * Deactivate the region before removing user mappings so that any + * fault handler racing the release returns VM_FAULT_SIGBUS rather + * than inserting a PFN into an unmapped region. + */ + WRITE_ONCE(cxl->region_active, false); + + if (cxl->region_vaddr) { + memunmap(cxl->region_vaddr); + cxl->region_vaddr = NULL; + } +} + +static const struct vfio_pci_regops vfio_cxl_regops = { + .rw = vfio_cxl_region_rw, + .mmap = vfio_cxl_region_mmap, + .release = vfio_cxl_region_release, +}; + MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c index 5cd0dfdcb7148..1b65260c80ced 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_emu.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -33,7 +33,7 @@ * +0x1c: (reserved) */ -static inline __le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32 hdm_off) +__le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32 hdm_off) { /* * hdm_off is a byte offset within the HDM decoder block. diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index e7d4f38d03be0..088ea882a6567 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -12,9 +12,18 @@ #include #include -/* CXL device state embedded in vfio_pci_core_device */ +struct vfio_pci_core_device; + +/* + * CXL device state embedded in vfio_pci_core_device. + * + * cxlds must be the first field: devm_cxl_dev_state_create() asserts + * offsetof(cxlds) == 0 so CXL core's container_of() lookups land back + * on this struct. + */ struct vfio_pci_cxl_state { struct cxl_dev_state cxlds; + struct vfio_pci_core_device *vdev; struct cxl_memdev *cxlmd; struct cxl_root_decoder *cxlrd; struct cxl_endpoint_decoder *cxled; @@ -34,6 +43,7 @@ struct vfio_pci_cxl_state { u8 comp_reg_bar; bool cache_capable; bool precommitted; + bool region_active; }; /* Register access sizes */ @@ -98,4 +108,6 @@ int vfio_cxl_create_cxl_region(struct vfio_pci_cxl_state *cxl, resource_size_t size); void vfio_cxl_destroy_cxl_region(struct vfio_pci_cxl_state *cxl); +__le32 *hdm_reg_ptr(struct vfio_pci_cxl_state *cxl, u32 hdm_off); + #endif /* __LINUX_VFIO_CXL_PRIV_H */ diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index c48da03089801..b3f9ef7a1016f 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1231,6 +1231,9 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, vfio_pci_zap_and_down_write_memory_lock(vdev); + /* Zap CXL DPA region PTEs before hardware reset clears HDM state */ + vfio_cxl_zap_region_locked(vdev); + /* * This function can be invoked while the power state is non-D0. If * pci_try_reset_function() has been called while the power state is @@ -1246,6 +1249,14 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, ret = pci_try_reset_function(vdev->pdev); if (__vfio_pci_memory_enabled(vdev)) vfio_pci_dma_buf_move(vdev, false); + + /* + * Re-enable DPA region if reset succeeded; fault handler will + * re-insert PFNs on next access without requiring a new mmap. + */ + if (!ret) + vfio_cxl_reactivate_region(vdev); + up_write(&vdev->memory_lock); return ret; diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index 1082ba43bafe9..726063b6ff70e 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -145,6 +145,8 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev); void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev); +void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev); +void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev); #else @@ -152,6 +154,10 @@ static inline void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { } static inline void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { } +static inline void +vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { } +static inline void +vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { } #endif /* CONFIG_VFIO_CXL_CORE */ From efaaf31a0101ce1235bdaf09c5ad505c176a2c4b Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:12 +0530 Subject: [PATCH 401/464] NVIDIA: VR: SAUCE: vfio/cxl: Virtualize CXL DVSEC config writes BugLink: https://bugs.launchpad.net/bugs/2152222 CXL devices expose DVSEC registers in PCI configuration space. Several of them affect device behavior (CXL.io/CXL.mem/CXL.cache enables, lock state, range bases) and must be virtualised so the guest cannot disturb host-owned policy. Add CXL-aware read and write handlers that operate on vdev->vconfig: - DVSEC reads come back from the vconfig shadow that vfio_config_init() already populates via vfio_ecap_init(). - DVSEC writes go through per-register handlers (cxl_dvsec_*_write) which apply the spec-defined reserved-bit and lock-bit masking before updating the shadow. - The handlers are wired in via vdev->dvsec_readfn / dvsec_writefn, which the global ecap_perms[PCI_EXT_CAP_ID_DVSEC] dispatcher routes to when the device is a CXL device. Non-CXL devices with a DVSEC capability fall through to direct hardware access. Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 3ff6c19fc5176d2468e159d6d390b6d235b416e2 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve context mismatches in Makefile and vfio_pci_core.h due to missing upstream dmabuf/p2pdma forward declarations in NV-Kernels base, Carry Disable_Caching into Cache WBI hardware write, use vfio_direct_config_read fallback, add byte-aligned read/write routing for DVSEC registers, handle partial-byte W1C writes for STATUS/STATUS2, add PM_INIT_COMPLETION RW1CS handling] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/Makefile | 2 +- drivers/vfio/pci/cxl/vfio_cxl_config.c | 344 +++++++++++++++++++++++++ drivers/vfio/pci/cxl/vfio_cxl_core.c | 4 +- drivers/vfio/pci/cxl/vfio_cxl_priv.h | 43 +++- drivers/vfio/pci/vfio_pci_config.c | 50 +++- drivers/vfio/pci/vfio_pci_priv.h | 7 + include/linux/vfio_pci_core.h | 7 + include/uapi/cxl/cxl_regs.h | 98 +++++++ 8 files changed, 541 insertions(+), 14 deletions(-) create mode 100644 drivers/vfio/pci/cxl/vfio_cxl_config.c diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index bef916495eae7..7c86b7845e8f4 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o -vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o +vfio-pci-core-$(CONFIG_VFIO_CXL_CORE) += cxl/vfio_cxl_core.o cxl/vfio_cxl_emu.o cxl/vfio_cxl_config.o vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o vfio-pci-core-$(CONFIG_VFIO_PCI_DMABUF) += vfio_pci_dmabuf.o obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o diff --git a/drivers/vfio/pci/cxl/vfio_cxl_config.c b/drivers/vfio/pci/cxl/vfio_cxl_config.c new file mode 100644 index 0000000000000..35d35e2ded9bf --- /dev/null +++ b/drivers/vfio/pci/cxl/vfio_cxl_config.c @@ -0,0 +1,344 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * CXL DVSEC configuration space emulation for vfio-pci. + * + * Integrates into the existing vfio-pci-core ecap_perms[] framework using + * vdev->vconfig as the sole shadow buffer for DVSEC registers. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include +#include + +#include "../vfio_pci_priv.h" +#include "vfio_cxl_priv.h" + +static inline u16 _cxlds_get_dvsec(struct vfio_pci_cxl_state *cxl) +{ + return (u16)cxl->cxlds.cxl_dvsec; +} + +/* Helpers to access vdev->vconfig at a DVSEC-relative offset */ +static inline u16 dvsec_virt_read16(struct vfio_pci_core_device *vdev, + u16 off) +{ + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + + return get_unaligned_le16(vdev->vconfig + dvsec + off); +} + +static inline void dvsec_virt_write16(struct vfio_pci_core_device *vdev, + u16 off, u16 val) +{ + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + + put_unaligned_le16(val, vdev->vconfig + dvsec + off); +} + +static inline u32 dvsec_virt_read32(struct vfio_pci_core_device *vdev, + u16 off) +{ + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + + return get_unaligned_le32(vdev->vconfig + dvsec + off); +} + +static inline void dvsec_virt_write32(struct vfio_pci_core_device *vdev, + u16 off, u32 val) +{ + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + + put_unaligned_le32(val, vdev->vconfig + dvsec + off); +} + +static u32 dvsec_virt_merge_write32(struct vfio_pci_core_device *vdev, + u16 off, u16 byte_in_reg, + int count, __le32 val) +{ + u32 cur = dvsec_virt_read32(vdev, off); + u32 data = le32_to_cpu(val); + u32 mask; + + if (byte_in_reg + count > sizeof(u32)) + return cur; + + if (count == sizeof(u32)) + return data; + + mask = (1U << (count * 8)) - 1; + mask <<= byte_in_reg * 8; + + return (cur & ~mask) | ((data << (byte_in_reg * 8)) & mask); +} + +/* Individual DVSEC register write handlers */ + +static void cxl_dvsec_control_write(struct vfio_pci_core_device *vdev, + u16 new_val) +{ + u16 lock = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET); + u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET); + u16 rev_mask = CXL_CTRL_RESERVED_MASK; + + if (lock & CXL_DVSEC_LOCK_CONFIG_LOCK) + return; /* register is locked after first write */ + + if (!(cap3 & CXL_DVSEC_CAP3_P2P_MEM_CAPABLE)) + rev_mask |= CXL_CTRL_P2P_REV_MASK; + + new_val &= ~rev_mask; + new_val |= CXL_DVSEC_CTRL_IO_ENABLE; /* IO_Enable always returns 1 */ + + dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, new_val); +} + +static void cxl_dvsec_status_write(struct vfio_pci_core_device *vdev, + u16 new_val) +{ + u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS_OFFSET); + + /* + * VIRAL_STATUS (bit 14) is the only writable bit; all others are + * reserved and always zero. + */ + new_val = cur_val & ~(new_val & CXL_DVSEC_STATUS_VIRAL_STATUS); + dvsec_virt_write16(vdev, CXL_DVSEC_STATUS_OFFSET, new_val); +} + +static void cxl_dvsec_control2_write(struct vfio_pci_core_device *vdev, + u16 new_val) +{ + struct pci_dev *pdev = vdev->pdev; + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + u16 abs_off = dvsec + CXL_DVSEC_CONTROL2_OFFSET; + u16 cap2 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY2_OFFSET); + u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET); + u16 rev_mask = CXL_CTRL2_RESERVED_MASK; + + if (!(cap3 & CXL_DVSEC_CAP3_VOLATILE_HDM_CONFIGURABILITY)) + rev_mask |= CXL_CTRL2_VOLATILE_HDM_REV_MASK; + if (!(cap2 & CXL_DVSEC_CAP2_MOD_COMPLETION_CAPABLE)) + rev_mask |= CXL_CTRL2_MODIFIED_COMP_REV_MASK; + + new_val &= ~rev_mask; + + /* + * Cache WBI: forward to hardware. cxl_dev_reset() programs + * Disable_Caching first and then initiates Cache WBI with that bit + * still set; replicate that ordering for VMM-delegated WBI by carrying + * the just-written Disable_Caching value into the same hardware write. + * new_val is the post-merge 16-bit shadow value, so it already reflects + * a prior shadow-only Disable_Caching=1 followed by a WBI-only write. + */ + if (new_val & CXL_DVSEC_CTRL2_INITIATE_CACHE_WBI) { + u16 hw_val = CXL_DVSEC_CTRL2_INITIATE_CACHE_WBI; + + if (new_val & CXL_DVSEC_CTRL2_DISABLE_CACHING) + hw_val |= CXL_DVSEC_CTRL2_DISABLE_CACHING; + pci_write_config_word(pdev, abs_off, hw_val); + } + + /* + * CXL Reset: not yet supported - do not forward to HW. + * TODO: invoke CXL protocol reset via cxl subsystem + */ + if (new_val & CXL_DVSEC_CTRL2_INITIATE_CXL_RESET) + pci_warn(pdev, "vfio-cxl: CXL reset requested but not yet supported\n"); + + dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL2_OFFSET, + new_val & ~CXL_CTRL2_HW_BITS_MASK); +} + +static void cxl_dvsec_status2_write(struct vfio_pci_core_device *vdev, + u16 new_val) +{ + u16 cap3 = dvsec_virt_read16(vdev, CXL_DVSEC_CAPABILITY3_OFFSET); + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + u16 abs_off = dvsec + CXL_DVSEC_STATUS2_OFFSET; + + /* RW1CS: write 1 to clear, but only if the capability is supported */ + if ((cap3 & CXL_DVSEC_CAP3_VOLATILE_HDM_CONFIGURABILITY) && + (new_val & CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR)) + pci_write_config_word(vdev->pdev, abs_off, + CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR); + /* STATUS2 is not mirrored in vconfig - reads go to hardware */ +} + +static void cxl_dvsec_lock_write(struct vfio_pci_core_device *vdev, + u16 new_val) +{ + u16 cur_val = dvsec_virt_read16(vdev, CXL_DVSEC_LOCK_OFFSET); + + /* Once the LOCK bit is set it can only be cleared by conventional reset */ + if (cur_val & CXL_DVSEC_LOCK_CONFIG_LOCK) + return; + + new_val &= ~CXL_LOCK_RESERVED_MASK; + dvsec_virt_write16(vdev, CXL_DVSEC_LOCK_OFFSET, new_val); +} + +static void cxl_range_base_lo_write(struct vfio_pci_core_device *vdev, + u16 dvsec_off, u32 new_val) +{ + new_val &= ~CXL_BASE_LO_RESERVED_MASK; + dvsec_virt_write32(vdev, dvsec_off, new_val); +} + +/** + * vfio_cxl_dvsec_readfn - Per-device DVSEC read handler for CXL capable devices. + * @vdev: VFIO PCI core device + * @pos: Absolute byte position in PCI config space + * @count: Number of bytes to read + * @perm: Permission bits for this capability (passed through to fallback) + * @offset: Byte offset within the capability structure (passed through) + * @val: Output buffer for the read value (little-endian) + * + * Called via vfio_pci_dvsec_dispatch_read() for CXL devices. Returns shadow + * vconfig values for virtualized DVSEC registers (CONTROL, STATUS, CONTROL2, + * LOCK) so that userspace reads reflect emulated state rather than raw + * hardware. All other DVSEC bytes pass through to vfio_raw_config_read(). + * + * Return: @count on success, or negative error code from the fallback read. + */ +static int vfio_cxl_dvsec_readfn(struct vfio_pci_core_device *vdev, + int pos, int count, + struct perm_bits *perm, + int offset, __le32 *val) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + u16 dvsec_off; + + if (!cxl || (u16)pos < dvsec || + (u16)pos >= dvsec + cxl->dvsec_len) + return vfio_raw_config_read(vdev, pos, count, perm, offset, val); + + dvsec_off = (u16)pos - dvsec; + + switch (dvsec_off) { + case CXL_DVSEC_CONTROL_OFFSET: + case CXL_DVSEC_STATUS_OFFSET: + case CXL_DVSEC_CONTROL2_OFFSET: + case CXL_DVSEC_LOCK_OFFSET: + /* Return shadow vconfig value for virtualized registers */ + memcpy(val, vdev->vconfig + pos, count); + return count; + default: + return vfio_raw_config_read(vdev, pos, count, + perm, offset, val); + } +} + +/** + * vfio_cxl_dvsec_writefn - ecap_perms write handler for PCI_EXT_CAP_ID_DVSEC. + * + * Installed once into ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn by + * vfio_pci_init_perm_bits() when CONFIG_VFIO_CXL_CORE=y. Applies to every + * device opened under vfio-pci; the vdev->cxl NULL check distinguishes CXL + * devices from non-CXL devices that happen to expose a DVSEC capability. + * + * @vdev: VFIO PCI core device + * @pos: Absolute byte position in PCI config space + * @count: Number of bytes to write + * @perm: Permission bits for this capability (passed through to fallback) + * @offset: Byte offset within the capability structure (passed through) + * @val: Value to write (little-endian) + * + * Return: @count on success; non-CXL devices continue to + * vfio_raw_config_write() which also returns @count or negative error. + */ +static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev, + int pos, int count, + struct perm_bits *perm, + int offset, __le32 val) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u16 dvsec = _cxlds_get_dvsec(vdev->cxl); + u16 abs_off = (u16)pos; + u16 dvsec_off, dword_start, byte_in_dword; + u16 wval16; + u32 wval32; + + if (!cxl || (u16)pos < dvsec || + (u16)pos >= dvsec + cxl->dvsec_len) + return vfio_raw_config_write(vdev, pos, count, perm, + offset, val); + + pci_dbg(vdev->pdev, + "vfio_cxl: DVSEC write: abs=0x%04x dvsec_off=0x%04x count=%d raw_val=0x%08x\n", + abs_off, abs_off - dvsec, count, le32_to_cpu(val)); + + dvsec_off = abs_off - dvsec; + + dword_start = dvsec_off & ~3u; + byte_in_dword = dvsec_off - dword_start; + + switch (dword_start) { + case CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET: + case CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET: + wval32 = dvsec_virt_merge_write32(vdev, dword_start, byte_in_dword, count, val); + dvsec_virt_write32(vdev, dword_start, wval32); + return count; + case CXL_DVSEC_RANGE1_BASE_LOW_OFFSET: + case CXL_DVSEC_RANGE2_BASE_LOW_OFFSET: + wval32 = dvsec_virt_merge_write32(vdev, dword_start, byte_in_dword, count, val); + cxl_range_base_lo_write(vdev, dword_start, wval32); + return count; + } + + /* Route to the appropriate per-register handler */ + switch (dvsec_off) { + case CXL_DVSEC_CONTROL_OFFSET: + wval16 = (u16)le32_to_cpu(val); + cxl_dvsec_control_write(vdev, wval16); + break; + case CXL_DVSEC_STATUS_OFFSET: + wval16 = (u16)le32_to_cpu(val); + cxl_dvsec_status_write(vdev, wval16); + break; + case CXL_DVSEC_CONTROL2_OFFSET: + wval16 = (u16)le32_to_cpu(val); + cxl_dvsec_control2_write(vdev, wval16); + break; + case CXL_DVSEC_STATUS2_OFFSET: + wval16 = (u16)le32_to_cpu(val); + cxl_dvsec_status2_write(vdev, wval16); + break; + case CXL_DVSEC_LOCK_OFFSET: + wval16 = (u16)le32_to_cpu(val); + cxl_dvsec_lock_write(vdev, wval16); + break; + default: + /* RO registers: header, capability, range sizes - discard */ + break; + } + + return count; +} + +/** + * vfio_cxl_setup_dvsec_perms - Install per-device CXL DVSEC read/write hooks. + * @vdev: VFIO PCI core device + * + * Called once per device open after vfio_config_init() has seeded vdev->vconfig + * from hardware. Installs vfio_cxl_dvsec_readfn and vfio_cxl_dvsec_writefn + * as per-device DVSEC handlers so that the global ecap_perms[DVSEC] dispatcher + * routes reads and writes through CXL-aware emulation. + * + * Forces CXL.io IO_ENABLE in the CONTROL vconfig shadow at init time so the + * initial guest read returns the correct value before the first write. + */ +void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) +{ + u16 ctrl = dvsec_virt_read16(vdev, CXL_DVSEC_CONTROL_OFFSET); + + vdev->dvsec_readfn = vfio_cxl_dvsec_readfn; + vdev->dvsec_writefn = vfio_cxl_dvsec_writefn; + + /* Force IO_ENABLE; cxl_dvsec_control_write() maintains this invariant. */ + ctrl |= CXL_DVSEC_CTRL_IO_ENABLE; + dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL_OFFSET, ctrl); +} +EXPORT_SYMBOL_GPL(vfio_cxl_setup_dvsec_perms); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 04f6f5cb47f6d..4d35ab1370278 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -73,13 +73,13 @@ vfio_cxl_create_device_state(struct pci_dev *pdev, u16 dvsec) * CACHE_CAPABLE is forwarded to the VMM so it knows whether a WBI * sequence is needed before FLR. */ - if (!FIELD_GET(CXL_DVSEC_MEM_CAPABLE, cap_word) || + if (!FIELD_GET(CXL_DVSEC_CAP_MEM_CAPABLE, cap_word) || (pdev->class >> 8) == PCI_CLASS_MEMORY_CXL) { devm_kfree(&pdev->dev, cxl); return ERR_PTR(-ENODEV); } - cxl->cache_capable = FIELD_GET(CXL_DVSEC_CACHE_CAPABLE, cap_word); + cxl->cache_capable = FIELD_GET(CXL_DVSEC_CAP_CACHE_CAPABLE, cap_word); return cxl; } diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index 088ea882a6567..baf2d2dffa741 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -86,14 +86,43 @@ struct vfio_pci_cxl_state { #define CXL_HDM_DECODER_GLOBAL_CTRL_POISON_EN_BIT BIT(0) /* - * CXL DVSEC for CXL Devices - register offsets within the DVSEC - * (CXL 4.0 8.1.3). - * Offsets are relative to the DVSEC capability base (cxl->dvsec). + * DVSEC register offsets and per-bit hardware definitions are in + * as CXL_DVSEC_*. The masks below encode + * emulation policy: which bits to ignore, which to preserve separately + * from their raw hardware state. */ -#define CXL_DVSEC_CAPABILITY_OFFSET 0xa -#define CXL_DVSEC_MEM_CAPABLE BIT(2) -/* CXL DVSEC Capability register bit 0: device supports CXL.cache (HDM-DB) */ -#define CXL_DVSEC_CACHE_CAPABLE BIT(0) +/* DVSEC Control (0x0C): bits 13 (RsvdP) and 15 (RsvdP) are always discarded */ +#define CXL_CTRL_RESERVED_MASK (BIT(13) | BIT(15)) +/* bit 12 (P2P_Mem_Enable) treated as reserved if Cap3.P2P_Mem_Capable=0 */ +#define CXL_CTRL_P2P_REV_MASK CXL_DVSEC_CTRL_P2P_MEM_ENABLE + +/* DVSEC Status (0x0E): bits 13:0 and 15 are RsvdZ */ +#define CXL_STATUS_RESERVED_MASK (GENMASK(13, 0) | BIT(15)) + +/* + * DVSEC Control2 (0x10) emulation masks. + * + * CXL_CTRL2_HW_BITS_MASK: bits 1 (Initiate_Cache_WBI) and 2 + * (Initiate_CXL_Reset) always read 0 from hardware _ they are write-only + * action triggers per CXL 4.0 _8.1.3.8 Table 8-8. Forward these to the + * device to trigger the hardware action; clear them from vconfig shadow so + * that subsequent guest reads return 0 as hardware requires. + * + * NOTE: bit 0 (Disable_Caching) and bit 3 (CXL_Reset_Mem_Clr_Enable) are + * ordinary RW fields _ they must be preserved in vconfig, not forwarded. + */ +#define CXL_CTRL2_RESERVED_MASK GENMASK(15, 6) +#define CXL_CTRL2_HW_BITS_MASK (BIT(1) | BIT(2)) +/* bit 4 is RsvdP if Cap3.Volatile_HDM_Configurability=0 */ +#define CXL_CTRL2_VOLATILE_HDM_REV_MASK CXL_DVSEC_CTRL2_DESIRED_VOLATILE_HDM +/* bit 5 is RsvdP if Cap2.Mod_Completion_Capable=0 */ +#define CXL_CTRL2_MODIFIED_COMP_REV_MASK CXL_DVSEC_CTRL2_MOD_COMPLETION_ENABLE + +/* DVSEC Lock (0x14): bits 15:1 are RsvdP */ +#define CXL_LOCK_RESERVED_MASK GENMASK(15, 1) + +/* DVSEC Range Base Low: bits 27:0 are reserved per Tables 8-15/8-19 */ +#define CXL_BASE_LO_RESERVED_MASK CXL_DVSEC_RANGE_BASE_LOW_RSVD_MASK int vfio_cxl_setup_virt_regs(struct vfio_pci_core_device *vdev, struct vfio_pci_cxl_state *cxl, diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index 61e7cdf7bec86..bf4730601949e 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -245,9 +245,9 @@ static int vfio_default_config_write(struct vfio_pci_core_device *vdev, int pos, } /* Allow direct read from hardware, except for capability next pointer */ -static int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos, - int count, struct perm_bits *perm, - int offset, __le32 *val) +int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 *val) { int ret; @@ -1085,6 +1085,47 @@ static int __init init_pci_ext_cap_pwr_perm(struct perm_bits *perm) return 0; } +/* + * vfio_pci_dvsec_dispatch_read - per-device DVSEC read dispatcher. + * + * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn at module init. + * Calls vdev->dvsec_readfn when a shadow-read handler has been registered + * (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2 devices); otherwise + * fall back to vfio_direct_config_read so non-CXL DVSEC devices keep the + * extended-cap header mangling that the default ecap readfn applies. + */ +static int vfio_pci_dvsec_dispatch_read(struct vfio_pci_core_device *vdev, + int pos, int count, + struct perm_bits *perm, + int offset, __le32 *val) +{ + if (vdev->dvsec_readfn) + return vdev->dvsec_readfn(vdev, pos, count, perm, offset, val); + return vfio_direct_config_read(vdev, pos, count, perm, offset, val); +} + +/* + * vfio_pci_dvsec_dispatch_write - per-device DVSEC write dispatcher. + * + * Installed as ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn at module init. + * Calls vdev->dvsec_writefn when a handler has been registered for this + * device (e.g. by vfio_cxl_setup_dvsec_perms() for CXL Type-2 devices), + * otherwise proceed to vfio_raw_config_write so that non-CXL devices + * with a DVSEC capability continue to pass writes to hardware. + * + * This indirection allows per-device DVSEC handlers to be registered + * without touching the global ecap_perms[] table. + */ +static int vfio_pci_dvsec_dispatch_write(struct vfio_pci_core_device *vdev, + int pos, int count, + struct perm_bits *perm, + int offset, __le32 val) +{ + if (vdev->dvsec_writefn) + return vdev->dvsec_writefn(vdev, pos, count, perm, offset, val); + return vfio_raw_config_write(vdev, pos, count, perm, offset, val); +} + /* * Initialize the shared permission tables */ @@ -1121,7 +1162,8 @@ int __init vfio_pci_init_perm_bits(void) ret |= init_pci_ext_cap_err_perm(&ecap_perms[PCI_EXT_CAP_ID_ERR]); ret |= init_pci_ext_cap_pwr_perm(&ecap_perms[PCI_EXT_CAP_ID_PWR]); ecap_perms[PCI_EXT_CAP_ID_VNDR].writefn = vfio_raw_config_write; - ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_raw_config_write; + ecap_perms[PCI_EXT_CAP_ID_DVSEC].readfn = vfio_pci_dvsec_dispatch_read; + ecap_perms[PCI_EXT_CAP_ID_DVSEC].writefn = vfio_pci_dvsec_dispatch_write; if (ret) vfio_pci_uninit_perm_bits(); diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index 726063b6ff70e..bc86cb03ae820 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -45,6 +45,10 @@ int vfio_raw_config_read(struct vfio_pci_core_device *vdev, int pos, int count, struct perm_bits *perm, int offset, __le32 *val); +int vfio_direct_config_read(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 *val); + ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf, size_t count, loff_t *ppos, bool iswrite); @@ -147,6 +151,7 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev); void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev); void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev); void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev); +void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev); #else @@ -158,6 +163,8 @@ static inline void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { } static inline void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { } +static inline void +vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { } #endif /* CONFIG_VFIO_CXL_CORE */ diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 6e8885f79d264..6158a12b9b277 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -30,6 +30,7 @@ struct vfio_pci_region; struct p2pdma_provider; struct dma_buf_attachment; struct vfio_pci_cxl_state; +struct perm_bits; struct vfio_pci_eventfd { struct eventfd_ctx *ctx; @@ -139,6 +140,12 @@ struct vfio_pci_core_device { struct list_head ioeventfds_list; struct vfio_pci_vf_token *vf_token; struct vfio_pci_cxl_state *cxl; + int (*dvsec_readfn)(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 *val); + int (*dvsec_writefn)(struct vfio_pci_core_device *vdev, int pos, + int count, struct perm_bits *perm, + int offset, __le32 val); struct list_head sriov_pfs_item; struct vfio_pci_core_device *sriov_pf_core_dev; struct notifier_block nb; diff --git a/include/uapi/cxl/cxl_regs.h b/include/uapi/cxl/cxl_regs.h index aa9fea9b88f7f..6a6507ebf3191 100644 --- a/include/uapi/cxl/cxl_regs.h +++ b/include/uapi/cxl/cxl_regs.h @@ -60,4 +60,102 @@ #define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i) #define CXL_HDM_DECODER0_SKIP_HIGH(i) CXL_HDM_DECODER0_TL_HIGH(i) +/* + * CXL r4.0 8.1.3: DVSEC for CXL Devices + * + * Register offsets are relative to the DVSEC capability base address, + * as discovered via PCI_EXT_CAP_ID_DVSEC with DVSEC ID 0x0. + * All registers in this section are 16-bit wide. + */ + +/* DVSEC register offsets */ +#define CXL_DVSEC_CAPABILITY_OFFSET 0x0a +#define CXL_DVSEC_CONTROL_OFFSET 0x0c +#define CXL_DVSEC_STATUS_OFFSET 0x0e +#define CXL_DVSEC_CONTROL2_OFFSET 0x10 +#define CXL_DVSEC_STATUS2_OFFSET 0x12 +#define CXL_DVSEC_LOCK_OFFSET 0x14 +#define CXL_DVSEC_CAPABILITY2_OFFSET 0x16 +#define CXL_DVSEC_RANGE1_SIZE_HIGH_OFFSET 0x18 +#define CXL_DVSEC_RANGE1_SIZE_LOW_OFFSET 0x1c +#define CXL_DVSEC_RANGE1_BASE_HIGH_OFFSET 0x20 +#define CXL_DVSEC_RANGE1_BASE_LOW_OFFSET 0x24 +#define CXL_DVSEC_RANGE2_SIZE_HIGH_OFFSET 0x28 +#define CXL_DVSEC_RANGE2_SIZE_LOW_OFFSET 0x2c +#define CXL_DVSEC_RANGE2_BASE_HIGH_OFFSET 0x30 +#define CXL_DVSEC_RANGE2_BASE_LOW_OFFSET 0x34 +#define CXL_DVSEC_CAPABILITY3_OFFSET 0x38 + +/* DVSEC Range Base Low registers: bits [27:0] are reserved */ +#define CXL_DVSEC_RANGE_BASE_LOW_RSVD_MASK __GENMASK(27, 0) + +/* CXL r4.0 8.1.3.1 Table 8-5 DVSEC CXL Capability (offset 0x0A) */ +#define CXL_DVSEC_CAP_CACHE_CAPABLE _BITUL(0) +#define CXL_DVSEC_CAP_IO_CAPABLE _BITUL(1) +#define CXL_DVSEC_CAP_MEM_CAPABLE _BITUL(2) +#define CXL_DVSEC_CAP_MEM_HW_INIT_MODE _BITUL(3) +#define CXL_DVSEC_CAP_HDM_COUNT_MASK __GENMASK(5, 4) +#define CXL_DVSEC_CAP_CACHE_WBI_CAPABLE _BITUL(6) +#define CXL_DVSEC_CAP_CXL_RESET_CAPABLE _BITUL(7) +#define CXL_DVSEC_CAP_CXL_RESET_TIMEOUT_MASK __GENMASK(10, 8) +#define CXL_DVSEC_CAP_CXL_RESET_MEM_CLR_CAPABLE _BITUL(11) +#define CXL_DVSEC_CAP_TSP_CAPABLE _BITUL(12) +#define CXL_DVSEC_CAP_MLD_CAPABLE _BITUL(13) +#define CXL_DVSEC_CAP_VIRAL_CAPABLE _BITUL(14) +#define CXL_DVSEC_CAP_PM_INIT_REPORTING_CAPABLE _BITUL(15) + +/* CXL r4.0 8.1.3.2 Table 8-6 DVSEC CXL Control (offset 0x0C) */ +#define CXL_DVSEC_CTRL_CACHE_ENABLE _BITUL(0) +#define CXL_DVSEC_CTRL_IO_ENABLE _BITUL(1) +#define CXL_DVSEC_CTRL_MEM_ENABLE _BITUL(2) +#define CXL_DVSEC_CTRL_CACHE_SF_COVERAGE_MASK __GENMASK(7, 3) +#define CXL_DVSEC_CTRL_CACHE_SF_GRANULARITY_MASK __GENMASK(10, 8) +#define CXL_DVSEC_CTRL_CACHE_CLEAN_EVICTION _BITUL(11) +#define CXL_DVSEC_CTRL_P2P_MEM_ENABLE _BITUL(12) +/* bit 13: RsvdP */ +#define CXL_DVSEC_CTRL_VIRAL_ENABLE _BITUL(14) +/* bit 15: RsvdP */ + +/* CXL r4.0 8.1.3.3 Table 8-7 DVSEC CXL Status (offset 0x0E) */ +/* bits 13:0 = RsvdZ */ +#define CXL_DVSEC_STATUS_VIRAL_STATUS _BITUL(14) +/* bit 15 = RsvdZ */ + +/* CXL r4.0 8.1.3.4 Table 8-8 DVSEC CXL Control2 (offset 0x10) */ +#define CXL_DVSEC_CTRL2_DISABLE_CACHING _BITUL(0) +#define CXL_DVSEC_CTRL2_INITIATE_CACHE_WBI _BITUL(1) +#define CXL_DVSEC_CTRL2_INITIATE_CXL_RESET _BITUL(2) +#define CXL_DVSEC_CTRL2_CXL_RESET_MEM_CLR_ENABLE _BITUL(3) +#define CXL_DVSEC_CTRL2_DESIRED_VOLATILE_HDM _BITUL(4) +#define CXL_DVSEC_CTRL2_MOD_COMPLETION_ENABLE _BITUL(5) +/* bits 15:6 = RsvdP */ + +/* CXL r4.0 8.1.3.5 Table 8-9 DVSEC CXL Status2 (offset 0x12) */ +#define CXL_DVSEC_STATUS2_CACHE_INVALID _BITUL(0) +#define CXL_DVSEC_STATUS2_CXL_RESET_COMPLETE _BITUL(1) +#define CXL_DVSEC_STATUS2_CXL_RESET_ERROR _BITUL(2) +/* RW1CS; RsvdZ if Cap3.Volatile_HDM_Configurability=0 */ +#define CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR _BITUL(3) +/* bits 14:4 = RsvdZ */ +#define CXL_DVSEC_STATUS2_PM_INIT_COMPLETION _BITUL(15) + +/* CXL r4.0 _8.1.3.6 Table 8-10 _ DVSEC CXL Lock (offset 0x14) */ +#define CXL_DVSEC_LOCK_CONFIG_LOCK _BITUL(0) +/* bits 15:1 = RsvdP */ + +/* CXL r4.0 8.1.3.7 Table 8-11 DVSEC CXL Capability2 (offset 0x16) */ +#define CXL_DVSEC_CAP2_CACHE_SIZE_UNIT_MASK __GENMASK(3, 0) +#define CXL_DVSEC_CAP2_FALLBACK_CAPABILITY_MASK __GENMASK(5, 4) +#define CXL_DVSEC_CAP2_MOD_COMPLETION_CAPABLE _BITUL(6) +#define CXL_DVSEC_CAP2_NO_CLEAN_WRITEBACK _BITUL(7) +#define CXL_DVSEC_CAP2_CACHE_SIZE_MASK __GENMASK(15, 8) + +/* CXL r4.0 8.1.3.14 Table 8-20 DVSEC CXL Capability3 (offset 0x38) */ +#define CXL_DVSEC_CAP3_DEFAULT_VOLATILE_HDM_COLD_RESET _BITUL(0) +#define CXL_DVSEC_CAP3_DEFAULT_VOLATILE_HDM_WARM_RESET _BITUL(1) +#define CXL_DVSEC_CAP3_DEFAULT_VOLATILE_HDM_HOT_RESET _BITUL(2) +#define CXL_DVSEC_CAP3_VOLATILE_HDM_CONFIGURABILITY _BITUL(3) +#define CXL_DVSEC_CAP3_P2P_MEM_CAPABLE _BITUL(4) +/* bits 15:5 = RsvdP */ + #endif /* _UAPI_CXL_REGS_H_ */ From 6651fbdfb699b062661e22fa9a505cecbdeabdf7 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:13 +0530 Subject: [PATCH 402/464] NVIDIA: VR: SAUCE: vfio/cxl: Register regions with VFIO layer BugLink: https://bugs.launchpad.net/bugs/2152222 Register the DPA and component register region with VFIO layer. Region indices for both these regions are cached for quick lookup. vfio_cxl_register_cxl_region() - memremap(WB) the region HPA (treat CXL.mem as RAM, not MMIO) - Register VFIO_REGION_SUBTYPE_CXL - Records dpa_region_idx. vfio_cxl_register_comp_regs_region() - Registers VFIO_REGION_SUBTYPE_CXL_COMP_REGS with size hdm_reg_offset + hdm_reg_size - Records comp_reg_region_idx. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 6e2d9e5f273d9088c00fc3a766a224dd8abc7903 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Check HDM COMMITTED bit before activating DPA region on precommitted decoders, add pm_runtime/memory-enabled gate in fault and rw paths, split vfio_cxl_zap_dpa() from prepare_reset(), add DPA zap in vfio_pci_zap_and_down_write_memory_lock(), add hot-reset CXL prepare/finish passes] [kobak: Withheld DPA mmap advertisement on BOS until CPU-readable backing for CXL DPA PFNMAP can be proven; DPA fd read/write remains advertised.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 118 ++++++++++++++++++++++++++- drivers/vfio/pci/cxl/vfio_cxl_emu.c | 34 ++++++++ drivers/vfio/pci/cxl/vfio_cxl_priv.h | 2 + drivers/vfio/pci/vfio_pci.c | 23 ++++++ drivers/vfio/pci/vfio_pci_priv.h | 11 +++ 5 files changed, 187 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 4d35ab1370278..e3da65e2385a9 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -80,6 +80,8 @@ vfio_cxl_create_device_state(struct pci_dev *pdev, u16 dvsec) } cxl->cache_capable = FIELD_GET(CXL_DVSEC_CAP_CACHE_CAPABLE, cap_word); + cxl->dpa_region_idx = -1; + cxl->comp_reg_region_idx = -1; return cxl; } @@ -537,14 +539,19 @@ static int vfio_cxl_region_mmap(struct vfio_pci_core_device *vdev, */ void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { + struct vfio_device *core_vdev = &vdev->vdev; struct vfio_pci_cxl_state *cxl = vdev->cxl; lockdep_assert_held_write(&vdev->memory_lock); - if (!cxl) + if (!cxl || cxl->dpa_region_idx < 0) return; WRITE_ONCE(cxl->region_active, false); + unmap_mapping_range(core_vdev->inode->i_mapping, + VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_NUM_REGIONS + + cxl->dpa_region_idx), + cxl->region_size, true); } /* @@ -652,6 +659,7 @@ static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev, static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev, struct vfio_pci_region *region) { + struct vfio_device *core_vdev = &vdev->vdev; struct vfio_pci_cxl_state *cxl = region->data; /* @@ -661,6 +669,16 @@ static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev, */ WRITE_ONCE(cxl->region_active, false); + /* + * Remove all user mappings of the DPA region while the device is + * still alive. + */ + if (cxl->dpa_region_idx >= 0) + unmap_mapping_range(core_vdev->inode->i_mapping, + VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_NUM_REGIONS + + cxl->dpa_region_idx), + cxl->region_size, true); + if (cxl->region_vaddr) { memunmap(cxl->region_vaddr); cxl->region_vaddr = NULL; @@ -673,4 +691,102 @@ static const struct vfio_pci_regops vfio_cxl_regops = { .release = vfio_cxl_region_release, }; +int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u32 flags; + int ret; + + if (!cxl) + return -ENODEV; + + if (!cxl->region || cxl->region_vaddr) + return -ENODEV; + + /* + * CXL device memory is RAM, not MMIO. Use memremap() rather than + * ioremap_cache() so the correct memory-mapping API is used. + * The WB attribute matches the cache-coherent nature of CXL.mem. + */ + cxl->region_vaddr = memremap(cxl->region_hpa, cxl->region_size, + MEMREMAP_WB); + if (!cxl->region_vaddr) + return -ENOMEM; + + /* + * BOS/backport policy: do not advertise DPA mmap until the CXL DPA + * backing is proven safe for userspace CPU mappings. Keep fd + * read/write available via the memremap() kernel mapping. + */ + flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE; + + ret = vfio_pci_core_register_dev_region(vdev, + PCI_VENDOR_ID_CXL | + VFIO_REGION_TYPE_PCI_VENDOR_TYPE, + VFIO_REGION_SUBTYPE_CXL, + &vfio_cxl_regops, + cxl->region_size, flags, + cxl); + if (ret) { + memunmap(cxl->region_vaddr); + cxl->region_vaddr = NULL; + return ret; + } + + /* + * Cache the vdev->region[] index before activating the region. + * vfio_pci_core_register_dev_region() placed the new entry at + * vdev->region[num_regions - 1] and incremented num_regions. + * vfio_cxl_zap_region_locked() uses this to avoid scanning + * vdev->region[] on every FLR. + */ + cxl->dpa_region_idx = vdev->num_regions - 1; + + vfio_cxl_reinit_comp_regs(cxl); + + /* + * Only activate the DPA region when the HDM decoder is currently + * committed. vfio_pci_core_enable() runs pci_try_reset_function() + * before regions are registered; that FLR clears the decoder + * COMMITTED bit and firmware may not have re-committed it yet. + * Mirror vfio_cxl_finish_reset(): if COMMITTED is not set here, the + * region stays inactive and guest DPA access returns + * VM_FAULT_SIGBUS / -EIO until a subsequent reset re-runs + * finish_reset with the decoder committed. + */ + if (cxl->precommitted && cxl->comp_reg_virt) { + u32 ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_CTRL_OFFSET(0))); + + if (ctrl & CXL_HDM_DECODER0_CTRL_COMMITTED) + WRITE_ONCE(cxl->region_active, true); + } + + return 0; +} +EXPORT_SYMBOL_GPL(vfio_cxl_register_cxl_region); + +/** + * vfio_cxl_unregister_cxl_region - Undo vfio_cxl_register_cxl_region() + * @vdev: VFIO PCI device + * + * Marks the DPA region inactive and resets dpa_region_idx. + * Does NOT touch CXL subsystem state (cxl->region, cxl->cxled, cxl->cxlrd). + * The caller must call vfio_cxl_destroy_cxl_region() separately to release + * those objects. + */ +void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + if (!cxl || cxl->dpa_region_idx < 0) + return; + + WRITE_ONCE(cxl->region_active, false); + + cxl->dpa_region_idx = -1; +} +EXPORT_SYMBOL_GPL(vfio_cxl_unregister_cxl_region); + MODULE_IMPORT_NS("CXL"); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c index 1b65260c80ced..8f1eefec7c44b 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_emu.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -502,3 +502,37 @@ void vfio_cxl_clean_virt_regs(struct vfio_pci_cxl_state *cxl) kfree(cxl->comp_reg_virt); cxl->comp_reg_virt = NULL; } + +/* + * vfio_cxl_register_comp_regs_region - Register the COMP_REGS device region. + * + * Exposes the emulated HDM decoder register state as a VFIO device region + * with type VFIO_REGION_SUBTYPE_CXL_COMP_REGS. QEMU attaches a + * notify_change callback to this region to intercept HDM COMMIT writes + * and map the DPA MemoryRegion at the appropriate GPA. + * + * The region is read+write only (no mmap) to ensure all accesses pass + * through comp_regs_dispatch_write() for proper bit-field enforcement. + */ +int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + u32 flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE; + int ret; + + if (!cxl || !cxl->comp_reg_virt) + return -ENODEV; + + ret = vfio_pci_core_register_dev_region(vdev, + PCI_VENDOR_ID_CXL | + VFIO_REGION_TYPE_PCI_VENDOR_TYPE, + VFIO_REGION_SUBTYPE_CXL_COMP_REGS, + &vfio_cxl_comp_regs_ops, + cxl->hdm_reg_offset + + cxl->hdm_reg_size, flags, cxl); + if (!ret) + cxl->comp_reg_region_idx = vdev->num_regions - 1; + + return ret; +} +EXPORT_SYMBOL_GPL(vfio_cxl_register_comp_regs_region); diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index baf2d2dffa741..ac8ea3893c7af 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -38,6 +38,8 @@ struct vfio_pci_cxl_state { __le32 *comp_reg_virt; size_t dpa_size; void __iomem *hdm_iobase; + int dpa_region_idx; + int comp_reg_region_idx; u16 dvsec_len; u8 hdm_count; u8 comp_reg_bar; diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 0c771064c0b84..22cf9ea831f97 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -120,6 +120,29 @@ static int vfio_pci_open_device(struct vfio_device *core_vdev) } } + if (vdev->cxl) { + /* + * pci_config_map and vconfig are valid now (allocated by + * vfio_config_init() inside vfio_pci_core_enable() above). + */ + vfio_cxl_setup_dvsec_perms(vdev); + + ret = vfio_cxl_register_cxl_region(vdev); + if (ret) { + pci_warn(pdev, "Failed to setup CXL region\n"); + vfio_pci_core_disable(vdev); + return ret; + } + + ret = vfio_cxl_register_comp_regs_region(vdev); + if (ret) { + pci_warn(pdev, "Failed to register COMP_REGS region\n"); + vfio_cxl_unregister_cxl_region(vdev); + vfio_pci_core_disable(vdev); + return ret; + } + } + vfio_pci_core_finish_enable(vdev); return 0; diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index bc86cb03ae820..10ef78d68576d 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -152,6 +152,9 @@ void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev); void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev); void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev); void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev); +int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev); +void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev); +int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev); #else @@ -165,6 +168,14 @@ static inline void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { } static inline void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { } +static inline int +vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) +{ return 0; } +static inline void +vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev) { } +static inline int +vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev) +{ return 0; } #endif /* CONFIG_VFIO_CXL_CORE */ From 2f6af06b82341aa0a313b19f12bbcb9ec682161f Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:14 +0530 Subject: [PATCH 403/464] NVIDIA: VR: SAUCE: vfio/pci: Advertise CXL cap and sparse component BAR to userspace BugLink: https://bugs.launchpad.net/bugs/2152222 Expose CXL device capability through the VFIO device info ioctl and give userspace mmap access to the GPU/accelerator register windows in the component BAR while keeping the CXL component register block off-limits to user mappings. vfio_cxl_get_info() fills VFIO_DEVICE_INFO_CAP_CXL with the HDM register BAR index and byte offset, commit flags, and VFIO region indices for the DPA and COMP_REGS regions. HDM decoder count and the HDM block offset within COMP_REGS are not populated; both are derivable from the CXL Capability Array in the COMP_REGS region itself. vfio_cxl_get_region_info() handles VFIO_DEVICE_GET_REGION_INFO for the component register BAR. It builds a sparse-mmap capability that advertises only the GPU/accelerator register windows, carving out the CXL component register block. Three physical layouts are handled: Topology A comp block at BAR end: one area [0, comp_reg_offset) Topology B comp block at BAR start: one area [comp_end, bar_len) Topology C comp block in the middle: two areas, one on each side vfio_cxl_mmap_overlaps_comp_regs() checks whether an mmap request overlaps [comp_reg_offset, comp_reg_offset + comp_reg_size). vfio_pci_core_mmap() calls it to reject mmap of the component register block while allowing mmap of the GPU register windows in the sparse capability. This replaces the earlier blanket rejection of any mmap on the component BAR index. vfio_pci_bar_rw() applies the same overlap check, so fd pread()/pwrite() on the component BAR is also rejected when it would touch the component register subrange. All access to those registers goes through the dedicated COMP_REGS region, where the emulated HDM shadow lives. Hook both helpers into vfio_pci_ioctl_get_info() and vfio_pci_ioctl_get_region_info() in vfio_pci_core.c. The component BAR cannot be claimed exclusively since the CXL subsystem holds persistent sub-range iomem claims during HDM decoder setup. pci_request_selected_regions() returns EBUSY; pass bars=0 to skip the request and map directly via pci_iomap(). Physical ownership is assured by driver binding. Signed-off-by: Zhi Wang Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (cherry-picked from commit 9cd924807287d84bf9bd3b257a6f4b67342b203e from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Add BAR bounds check for component block, handle full-BAR component reg case, add bar_mmap_supported gate, block BAR fd read/write and ioeventfd in component reg subrange] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 191 +++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 31 ++++- drivers/vfio/pci/vfio_pci_priv.h | 24 ++++ drivers/vfio/pci/vfio_pci_rdwr.c | 37 +++++- 4 files changed, 278 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index e3da65e2385a9..81d195256d3e6 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -21,6 +21,197 @@ #include "../vfio_pci_priv.h" #include "vfio_cxl_priv.h" +u8 vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev) +{ + return vdev->cxl->comp_reg_bar; +} + +int vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev, + struct vfio_region_info *info, + struct vfio_info_cap *caps) +{ + unsigned long minsz = offsetofend(struct vfio_region_info, offset); + struct vfio_region_info_cap_sparse_mmap *sparse; + struct vfio_pci_cxl_state *cxl = vdev->cxl; + resource_size_t bar_len, comp_end; + u32 nr_areas, cap_size; + int ret; + + if (!cxl) + return -ENOTTY; + + if (!info) + return -ENOTTY; + + if (info->argsz < minsz) + return -EINVAL; + + if (info->index != cxl->comp_reg_bar) + return -ENOTTY; + + /* + * The device state is not fully initialised; + * fall through to the default BAR handler. + */ + if (!cxl->comp_reg_size) + return -ENOTTY; + + bar_len = pci_resource_len(vdev->pdev, info->index); + comp_end = cxl->comp_reg_offset + cxl->comp_reg_size; + + /* + * A component block past the end of the BAR would walk subsequent + * readl()s off the ioremap window. Reject that up front. + */ + if (comp_end > bar_len) + return -EINVAL; + + /* + * If the component block covers the entire BAR there is nothing to + * mmap; return the BAR with read/write access only and let userspace + * use the COMP_REGS device region for register access. + */ + if (cxl->comp_reg_size == bar_len) { + info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index); + info->size = bar_len; + info->flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE; + return 0; + } + + /* + * Preserve the existing vfio-pci bar_mmap_supported gate. When the + * BAR is non-mappable for any reason (non-page-aligned resource, the + * non_mappable_bars policy, etc.), advertising a sparse-mmap cap and + * VFIO_REGION_INFO_FLAG_MMAP would let userspace try to mmap and get + * a stale -EINVAL from the mmap path. Return the bare BAR descriptor + * instead and let userspace fall back to fd read/write. + */ + if (!vdev->bar_mmap_supported[info->index]) { + info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index); + info->size = bar_len; + info->flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE; + return 0; + } + + /* + * Advertise the GPU/accelerator register windows as mmappable by + * carving the CXL component register block out of the BAR. The + * number of sparse areas depends on where the block sits: + * + * [A] comp block at BAR end [gpu_regs | comp_regs]: + * comp_reg_offset > 0 && comp_end == bar_len + * = 1 area: [0, comp_reg_offset) + * + * [B] comp block at BAR start [comp_regs | gpu_regs]: + * comp_reg_offset == 0 && comp_end < bar_len + * = 1 area: [comp_end, bar_len) + * + * [C] comp block in middle [gpu_regs | comp_regs | gpu_regs]: + * comp_reg_offset > 0 && comp_end < bar_len + * = 2 areas: [0, comp_reg_offset) and [comp_end, bar_len) + */ + if (cxl->comp_reg_offset > 0 && comp_end < bar_len) + nr_areas = 2; + else + nr_areas = 1; + + cap_size = struct_size(sparse, areas, nr_areas); + sparse = kzalloc(cap_size, GFP_KERNEL); + if (!sparse) + return -ENOMEM; + + sparse->header.id = VFIO_REGION_INFO_CAP_SPARSE_MMAP; + sparse->header.version = 1; + sparse->nr_areas = nr_areas; + + if (nr_areas == 2) { + /* [C]: window before and after comp block */ + sparse->areas[0].offset = 0; + sparse->areas[0].size = cxl->comp_reg_offset; + sparse->areas[1].offset = comp_end; + sparse->areas[1].size = bar_len - comp_end; + } else if (cxl->comp_reg_offset == 0) { + /* [B]: comp block at BAR start, window follows */ + sparse->areas[0].offset = comp_end; + sparse->areas[0].size = bar_len - comp_end; + } else { + /* [A]: comp block at BAR end, window precedes */ + sparse->areas[0].offset = 0; + sparse->areas[0].size = cxl->comp_reg_offset; + } + + ret = vfio_info_add_capability(caps, &sparse->header, cap_size); + kfree(sparse); + if (ret) + return ret; + + info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index); + info->size = bar_len; + info->flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE | + VFIO_REGION_INFO_FLAG_MMAP; + + return 0; +} + +bool vfio_cxl_mmap_overlaps_comp_regs(struct vfio_pci_core_device *vdev, + u64 req_start, u64 req_len) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + if (!cxl->comp_reg_size) + return false; + + return req_start < cxl->comp_reg_offset + cxl->comp_reg_size && + req_start + req_len > cxl->comp_reg_offset; +} + +int vfio_cxl_get_info(struct vfio_pci_core_device *vdev, + struct vfio_info_cap *caps) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + struct vfio_device_info_cap_cxl cxl_cap = {0}; + + if (!cxl) + return 0; + + /* + * Device is not fully initialised? + */ + if (WARN_ON(cxl->dpa_region_idx < 0 || cxl->comp_reg_region_idx < 0)) + return -ENODEV; + + /* Fill in from CXL device structure */ + cxl_cap.header.id = VFIO_DEVICE_INFO_CAP_CXL; + cxl_cap.header.version = 1; + /* + * COMP_REGS region starts at comp_reg_offset + CXL_CM_OFFSET within + * the BAR. This is the byte offset of the CXL.mem register area (where + * the CXL Capability Array Header lives) within the component register + * block. Userspace derives hdm_decoder_offset and hdm_count from the + * COMP_REGS region itself (CXL Capability Array traversal + HDMC read). + */ + cxl_cap.hdm_regs_offset = cxl->comp_reg_offset + CXL_CM_OFFSET; + cxl_cap.hdm_regs_bar_index = cxl->comp_reg_bar; + + if (cxl->precommitted) + cxl_cap.flags |= VFIO_CXL_CAP_FIRMWARE_COMMITTED; + if (cxl->cache_capable) + cxl_cap.flags |= VFIO_CXL_CAP_CACHE_CAPABLE; + + /* + * Populate absolute VFIO region indices so userspace can query them + * directly with VFIO_DEVICE_GET_REGION_INFO. + */ + cxl_cap.dpa_region_index = VFIO_PCI_NUM_REGIONS + cxl->dpa_region_idx; + cxl_cap.comp_regs_region_index = + VFIO_PCI_NUM_REGIONS + cxl->comp_reg_region_idx; + + return vfio_info_add_capability(caps, &cxl_cap.header, sizeof(cxl_cap)); +} + /* * Scope-based cleanup wrappers for the CXL resource APIs */ diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b3f9ef7a1016f..d91e2d10a77a5 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -591,7 +591,7 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) struct pci_dev *pdev = vdev->pdev; struct vfio_pci_dummy_resource *dummy_res, *tmp; struct vfio_pci_ioeventfd *ioeventfd, *ioeventfd_tmp; - int i, bar; + int i, bar, bars; /* For needs_reset */ lockdep_assert_held(&vdev->vdev.dev_set->lock); @@ -650,8 +650,10 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) bar = i + PCI_STD_RESOURCES; if (!vdev->barmap[bar]) continue; + bars = (vdev->cxl && i == vfio_cxl_get_component_reg_bar(vdev)) ? + 0 : (1 << bar); pci_iounmap(pdev, vdev->barmap[bar]); - pci_release_selected_regions(pdev, 1 << bar); + pci_release_selected_regions(pdev, bars); vdev->barmap[bar] = NULL; } @@ -997,6 +999,13 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, if (vdev->reset_works) info.flags |= VFIO_DEVICE_FLAGS_RESET; + if (vdev->cxl) { + ret = vfio_cxl_get_info(vdev, &caps); + if (ret) + return ret; + info.flags |= VFIO_DEVICE_FLAGS_CXL; + } + info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions; info.num_irqs = VFIO_PCI_NUM_IRQS; @@ -1042,6 +1051,12 @@ int vfio_pci_ioctl_get_region_info(struct vfio_device *core_vdev, struct pci_dev *pdev = vdev->pdev; int i, ret; + if (vdev->cxl) { + ret = vfio_cxl_get_region_info(vdev, info, caps); + if (ret != -ENOTTY) + return ret; + } + switch (info->index) { case VFIO_PCI_CONFIG_REGION_INDEX: info->offset = VFIO_PCI_INDEX_TO_OFFSET(info->index); @@ -1771,6 +1786,18 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma if (req_start + req_len > phys_len) return -EINVAL; + /* + * CXL devices: mmap is permitted for the GPU/accelerator register + * windows listed in the sparse-mmap capability. Block any request + * that overlaps the CXL component register block + * [comp_reg_offset, comp_reg_offset + comp_reg_size); those registers + * must be accessed exclusively through the COMP_REGS device region so + * that the emulation layer (notify_change) intercepts every write. + */ + if (vdev->cxl && index == vfio_cxl_get_component_reg_bar(vdev) && + vfio_cxl_mmap_overlaps_comp_regs(vdev, req_start, req_len)) + return -EINVAL; + /* * Even though we don't make use of the barmap for the mmap, * we need to request the region and the barmap tracks that. diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index 10ef78d68576d..e925f66923593 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -155,6 +155,14 @@ void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev); int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev); void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev); int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev); +int vfio_cxl_get_info(struct vfio_pci_core_device *vdev, + struct vfio_info_cap *caps); +int vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev, + struct vfio_region_info *info, + struct vfio_info_cap *caps); +u8 vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev); +bool vfio_cxl_mmap_overlaps_comp_regs(struct vfio_pci_core_device *vdev, + u64 req_start, u64 req_len); #else @@ -176,6 +184,22 @@ vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev) { } static inline int vfio_cxl_register_comp_regs_region(struct vfio_pci_core_device *vdev) { return 0; } +static inline int +vfio_cxl_get_info(struct vfio_pci_core_device *vdev, + struct vfio_info_cap *caps) +{ return -ENOTTY; } +static inline int +vfio_cxl_get_region_info(struct vfio_pci_core_device *vdev, + struct vfio_region_info *info, + struct vfio_info_cap *caps) +{ return -ENOTTY; } +static inline u8 +vfio_cxl_get_component_reg_bar(struct vfio_pci_core_device *vdev) +{ return U8_MAX; } +static inline bool +vfio_cxl_mmap_overlaps_comp_regs(struct vfio_pci_core_device *vdev, + u64 req_start, u64 req_len) +{ return false; } #endif /* CONFIG_VFIO_CXL_CORE */ diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c index 4251ee03e1463..3e0ec0b082ff8 100644 --- a/drivers/vfio/pci/vfio_pci_rdwr.c +++ b/drivers/vfio/pci/vfio_pci_rdwr.c @@ -201,19 +201,29 @@ EXPORT_SYMBOL_GPL(vfio_pci_core_do_io_rw); int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar) { struct pci_dev *pdev = vdev->pdev; - int ret; + int ret, bars; void __iomem *io; if (vdev->barmap[bar]) return 0; - ret = pci_request_selected_regions(pdev, 1 << bar, "vfio"); + /* + * The CXL component register BAR cannot be claimed exclusively: the + * CXL subsystem holds persistent sub-range iomem claims during HDM + * decoder setup. pci_request_selected_regions() for the full BAR + * fails with EBUSY. Pass bars=0 to make the request a no-op and map + * directly via pci_iomap(). + */ + bars = (vdev->cxl && bar == vfio_cxl_get_component_reg_bar(vdev)) ? + 0 : (1 << bar); + + ret = pci_request_selected_regions(pdev, bars, "vfio"); if (ret) return ret; io = pci_iomap(pdev, bar, 0); if (!io) { - pci_release_selected_regions(pdev, 1 << bar); + pci_release_selected_regions(pdev, bars); return -ENOMEM; } @@ -248,6 +258,17 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_core_device *vdev, char __user *buf, count = min(count, (size_t)(end - pos)); + /* + * For CXL devices, the component register subrange is emulated through + * the dedicated COMP_REGS region (comp_regs_dispatch_write). Reject fd + * read/write that targets that subrange so userspace cannot bypass the + * emulation by issuing pread()/pwrite() on the BAR fd. This matches + * the mmap path, which rejects overlapping mmap requests. + */ + if (vdev->cxl && bar == vfio_cxl_get_component_reg_bar(vdev) && + vfio_cxl_mmap_overlaps_comp_regs(vdev, pos, count)) + return -EINVAL; + if (bar == PCI_ROM_RESOURCE) { /* * The ROM can fill less space than the BAR, so we start the @@ -449,6 +470,16 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset, pos >= vdev->msix_offset + vdev->msix_size)) return -EINVAL; + /* + * Disallow ioeventfds that would land inside the CXL component + * register subrange. Without this check, the eventfd handler would + * iowrite directly into the BAR mapping, bypassing the COMP_REGS + * emulation enforced on the mmap and pread/pwrite paths. + */ + if (vdev->cxl && bar == vfio_cxl_get_component_reg_bar(vdev) && + vfio_cxl_mmap_overlaps_comp_regs(vdev, pos, count)) + return -EINVAL; + if (count == 8) return -EINVAL; From 800941ef7d4a39b05f9d6d569384606d51d47011 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:15 +0530 Subject: [PATCH 404/464] NVIDIA: VR: SAUCE: vfio/cxl: Provide opt-out for CXL feature BugLink: https://bugs.launchpad.net/bugs/2152222 This commit provides an opt-out mechanism to disable the CXL support from vfio module. The opt-out is provided both build time and module load time. Build time option CONFIG_VFIO_CXL_CORE is used to enable/disable CXL support in vfio-pci module. For runtime disabling the CXL support, use the module parameter disable_cxl. The bare vfio-pci driver copies that parameter into the per-device core state before registration. Variant drivers own their probe policy and must set vdev->disable_cxl explicitly before registering the core device. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 595c1ad9c3c from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Resolve context mismatch in vfio_pci.c probe function due to missing upstream pci_ops assignment in NV-Kernels base, Wrap disable_cxl field in #if IS_ENABLED(CONFIG_VFIO_CXL_CORE), update MODULE_PARM_DESC wording] [kobak: Preserved existing vfio-pci pci_ops assignment while wiring the CXL opt-out parameter.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 4 ++++ drivers/vfio/pci/vfio_pci.c | 9 +++++++++ include/linux/vfio_pci_core.h | 3 +++ 3 files changed, 16 insertions(+) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 81d195256d3e6..622482064890b 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -523,6 +523,10 @@ void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) u16 dvsec; int ret; + /* Honor the user opt-out decision */ + if (vdev->disable_cxl) + return; + if (!pcie_is_cxl(pdev)) return; diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 22cf9ea831f97..9d9b1116e6417 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -60,6 +60,12 @@ static bool disable_denylist; module_param(disable_denylist, bool, 0444); MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users."); +#if IS_ENABLED(CONFIG_VFIO_CXL_CORE) +static bool disable_cxl; +module_param(disable_cxl, bool, 0444); +MODULE_PARM_DESC(disable_cxl, "Disable CXL extensions on devices probed by the bare vfio-pci driver. Variant drivers do not consult this parameter; they must set vdev->disable_cxl explicitly in their probe path."); +#endif + static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev) { switch (pdev->vendor) { @@ -190,6 +196,9 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) dev_set_drvdata(&pdev->dev, vdev); vdev->pci_ops = &vfio_pci_dev_ops; +#if IS_ENABLED(CONFIG_VFIO_CXL_CORE) + vdev->disable_cxl = disable_cxl; +#endif ret = vfio_pci_core_register_device(vdev); if (ret) goto out_put_vdev; diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 6158a12b9b277..d9190930d22f1 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -129,6 +129,9 @@ struct vfio_pci_core_device { bool needs_pm_restore:1; bool pm_intx_masked:1; bool pm_runtime_engaged:1; +#if IS_ENABLED(CONFIG_VFIO_CXL_CORE) + bool disable_cxl:1; +#endif struct pci_saved_state *pci_saved_state; struct pci_saved_state *pm_save; int ioeventfds_nr; From 5fc80f2eaf98286c77ef363b5a1d95859a7e1dbe Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:16 +0530 Subject: [PATCH 405/464] NVIDIA: VR: SAUCE: docs: vfio-pci: Document CXL Type-2 device passthrough BugLink: https://bugs.launchpad.net/bugs/2152222 Add Documentation/driver-api/vfio-pci-cxl.rst describing the architecture, VFIO interfaces, and operational constraints for CXL Type-2 (cache-coherent accelerator) passthrough via vfio-pci-core, and link it from the driver-api index. The document covers: - VFIO_DEVICE_FLAGS_CXL and VFIO_DEVICE_INFO_CAP_CXL: what the capability struct contains and what the FIRMWARE_COMMITTED and CACHE_CAPABLE flags mean - How to derive hdm_decoder_offset and hdm_count from the COMP_REGS region by traversing the CXL Capability Array to find cap ID 0x5 and reading the HDM Decoder Capability register - Topology-aware sparse mmap on the component BAR (topologies A, B, C covering comp block at end, start, or middle of the BAR) - Two extra VFIO device regions: COMP_REGS for the emulated HDM register state and the DPA memory window - DVSEC config write virtualization: what the guest sees vs. hardware - FLR coordination: DPA PTEs zapped before reset, restored after Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from https://lore.kernel.org/linux-cxl/20260401143917.108413-1-mhonap@nvidia.com/) (backported from commit 696f0b100f0379bd64e352cfec40d4804aa9c20c from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [jan: Rename vfio_cxl_zap_region_locked to vfio_cxl_prepare_reset and vfio_cxl_reactivate_region to vfio_cxl_finish_reset in docs] [kobak: Document BOS DPA policy as READ|WRITE without MMAP while preserving fd read/write support.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- Documentation/driver-api/index.rst | 1 + Documentation/driver-api/vfio-pci-cxl.rst | 386 ++++++++++++++++++++++ 2 files changed, 387 insertions(+) create mode 100644 Documentation/driver-api/vfio-pci-cxl.rst diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index eaf7161ff9578..52f0c06a376a8 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -47,6 +47,7 @@ of interest to most developers working on device drivers. vfio-mediated-device vfio vfio-pci-device-specific-driver-acceptance + vfio-pci-cxl Bus-level documentation ======================= diff --git a/Documentation/driver-api/vfio-pci-cxl.rst b/Documentation/driver-api/vfio-pci-cxl.rst new file mode 100644 index 0000000000000..a65653a0aa237 --- /dev/null +++ b/Documentation/driver-api/vfio-pci-cxl.rst @@ -0,0 +1,386 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================= +VFIO PCI CXL Type-2 device passthrough +======================================= + +Overview +-------- + +Type-2 CXL devices are PCIe accelerators (GPUs, compute ASICs, and similar) +with coherent device memory on CXL.mem. DPA is mapped into host physical +address space through HDM decoders that the kernel's CXL subsystem owns. A +guest cannot program that hardware directly. + +This ``vfio-pci`` mode hands a VMM: + +- A read/write VFIO device region (COMP_REGS) that emulates the HDM decoder + register block with CXL register rules enforced in kernel code. +- A read/write VFIO device region (DPA) backed by the kernel-chosen host + physical range for device memory. BOS does not advertise userspace mmap for + this region. +- DVSEC config-space emulation so the guest cannot change host-owned CXL.io / + CXL.mem enable bits. + +Build with ``CONFIG_VFIO_CXL_CORE=y``. At runtime you can turn it off with:: + + modprobe vfio-pci disable_cxl=1 + +or, in a variant driver, set ``vdev->disable_cxl = true`` before registration. + + +Device detection +---------------- + +At ``vfio_pci_core_register_device()`` the driver checks for a Type-2 style +setup. All of the following must hold: + +1. CXL Device DVSEC present (PCIe DVSEC Vendor ID ``0x1E98``, DVSEC ID + ``0x0000``). +2. ``Mem_Capable`` (bit 2) set in the CXL Capability register inside that DVSEC. +3. PCI class code is **not** ``0x050210`` (CXL Type-3 memory expander). +4. An HDM Decoder capability block reachable through the Register Locator DVSEC. +5. At least one HDM decoder committed by firmware with non-zero size. + +The CXL spec labels "Type-2" as devices with both ``Mem_Capable`` and +``Cache_Capable``. This driver also takes ``Mem_Capable``-only devices +(``Cache_Capable=0``), which behave like Type-3 style accelerators without the +usual class code. ``VFIO_CXL_CAP_CACHE_CAPABLE`` exposes the cache bit to +userspace so a VMM can treat FLR differently when needed. + +When detection succeeds, ``VFIO_DEVICE_FLAGS_CXL`` is ORed into +``vfio_device_info.flags`` together with ``VFIO_DEVICE_FLAGS_PCI``. + +.. note:: + + **Firmware must commit an HDM decoder before open.** The driver only + discovers DPA range and size from a decoder that firmware already committed. + Devices without that, or hot-plugged setups that never get it, are out of + scope for now. + + Follow-up options under discussion include CXL range registers in the + Device DVSEC (often enough on single-decoder parts), CDAT over DOE, mailbox + Get Partition Info, or a future DVSEC field from the consortium for + base/size/NUMA without extra side channels. There is also talk of a sysfs + path, modeled on resizable BAR, where an orchestrator fixes the DPA window + before vfio-pci binds so the driver still sees a committed range. + + +UAPI: VFIO_DEVICE_INFO_CAP_CXL +------------------------------ + +When ``VFIO_DEVICE_FLAGS_CXL`` is set, the device info capability chain +includes a ``vfio_device_info_cap_cxl`` structure (cap ID 6, version 1):: + + struct vfio_device_info_cap_cxl { + struct vfio_info_cap_header header; /* id=6, version=1 */ + __u8 hdm_regs_bar_index; /* BAR index containing component regs */ + __u8 reserved[3]; + __u32 flags; /* VFIO_CXL_CAP_* flags */ + __u64 hdm_regs_offset; /* byte offset within the BAR to the + * CXL.mem register area start. This + * equals comp_reg_offset + CXL_CM_OFFSET + * where CXL_CM_OFFSET = 0x1000. */ + __u32 dpa_region_index; /* VFIO region index for DPA memory */ + __u32 comp_regs_region_index; /* VFIO region index for COMP_REGS */ + }; + /* + * hdm_count and hdm_decoder_offset are intentionally absent from this + * struct. Both are derivable from the COMP_REGS region. See the + * "Deriving HDM info from COMP_REGS" section below. + */ + + #define VFIO_CXL_CAP_FIRMWARE_COMMITTED (1 << 0) + #define VFIO_CXL_CAP_CACHE_CAPABLE (1 << 1) + +``VFIO_CXL_CAP_FIRMWARE_COMMITTED`` + At least one HDM decoder was pre-committed by firmware. The DPA region + is live at device open; the VMM can use fd read/write without waiting for + a guest COMMIT cycle. + +``VFIO_CXL_CAP_CACHE_CAPABLE`` + The device has an HDM-DB decoder (CXL.mem + CXL.cache). This mirrors the + ``Cache_Capable`` bit from the CXL DVSEC Capability register. The kernel + does not run Write-Back Invalidation (WBI) before FLR; with this flag set + that stays the VMM's job. + +DPA region size comes from ``VFIO_DEVICE_GET_REGION_INFO`` on +``dpa_region_index``, not from this struct. + + +VFIO regions +------------ + +A CXL device adds two device regions on top of the usual BARs. Their indices +are in ``dpa_region_index`` and ``comp_regs_region_index``. + +DPA region (``VFIO_REGION_SUBTYPE_CXL``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Flags: ``READ | WRITE``. + +The backing store is the host physical range the kernel assigned for DPA. The +kernel maps it with ``memremap(MEMREMAP_WB)`` because CXL device memory on a +coherent link sits in the CPU cache hierarchy. That mapping is normal cached +memory, so ``copy_to/from_user`` works without extra barriers. + +BOS intentionally withholds ``VFIO_REGION_INFO_FLAG_MMAP`` for this region +until CPU-readable backing for CXL DPA PFNMAP can be proven safe. VMMs must +not assume the DPA region is mmap-capable; check the region flags and use fd +read/write when ``MMAP`` is absent. + +Region read/write through the fd uses the same ``MEMREMAP_WB`` mapping with +``copy_to/from_user``. ``ioread``/``iowrite`` MMIO helpers are not used on +this path. + +During FLR, ``unmap_mapping_range()`` drops user PTEs and ``region_active`` +clears before the reset runs. Ongoing faults or region I/O then error instead +of touching a dead mapping. IOMMU ATC invalidation from the zap has to finish +before the device resets; doing it the other way around can leave an SMMU +waiting on a device that no longer responds. + +After reset, the region comes back once ``COMMITTED`` shows up again in fresh +HDM hardware state. The VMM can resume fd read/write without reopening the +device. + +COMP_REGS region (``VFIO_REGION_SUBTYPE_CXL_COMP_REGS``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Flags: ``READ | WRITE`` (no mmap). + +Emulated registers for the CXL.mem slice of the component register block: the +CXL Capability Array header at offset 0, then the HDM Decoder capability +starting at ``hdm_decoder_offset`` (the byte offset derived by traversing the +CXL Capability Array — see "Deriving HDM info from COMP_REGS" below). +Region size from ``VFIO_DEVICE_GET_REGION_INFO`` covers the full capability +array prefix plus all HDM decoder blocks. + +Only 32-bit, 32-bit-aligned accesses are allowed. 8- and 16-bit attempts get +``-EINVAL``. + +Offsets below ``hdm_decoder_offset`` return the snapshot from device open. +Writes there are dropped (with a WARN); the capability array stays read-only. + +From ``hdm_decoder_offset`` upward the kernel keeps a shadow +(``comp_reg_virt[]``) and applies field rules: + +- At open, hardware HDM state is snapshotted. For firmware-committed decoders + the LOCK bit is cleared and BASE_HI/BASE_LO are zeroed in the shadow so the + VMM can program guest GPA; the host HPA is not carried in the shadow after + that. +- ``COMMIT`` (bit 9 of CTRL): writing 1 sets ``COMMITTED`` (bit 10) in the + shadow immediately. Real hardware stays committed; the shadow tracks what + the guest should see. +- When LOCK is set, writes to BASE_HI and SIZE_HI are ignored so + firmware-committed values survive. + +Region type identifiers:: + + /* type = PCI_VENDOR_ID_CXL | VFIO_REGION_TYPE_PCI_VENDOR_TYPE */ + #define VFIO_REGION_SUBTYPE_CXL 1 /* DPA memory region */ + #define VFIO_REGION_SUBTYPE_CXL_COMP_REGS 2 /* HDM register shadow */ + + +BAR access +---------- + +``VFIO_DEVICE_GET_REGION_INFO`` for ``hdm_regs_bar_index`` reports the full +BAR size with ``READ | WRITE | MMAP`` flags and a +``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` capability listing the GPU or +accelerator register windows — the mmappable parts of the BAR that do **not** +contain CXL component registers. + +The number of sparse areas depends on where the CXL component register block +``[comp_reg_offset, comp_reg_offset + comp_reg_size)`` sits within the BAR: + +* **Topology A** - component block at BAR end: + ``[gpu_regs | comp_regs]`` → 1 area: ``[0, comp_reg_offset)`` + +* **Topology B** - component block at BAR start: + ``[comp_regs | gpu_regs]`` → 1 area: ``[comp_reg_size, bar_len)`` + +* **Topology C** - component block in middle: + ``[gpu_regs | comp_regs | gpu_regs]`` → 2 areas: + ``[0, comp_reg_offset)`` and ``[comp_reg_offset + comp_reg_size, bar_len)`` + +VMMs **must** iterate all ``nr_areas`` entries; do not assume a single area or +that the first area starts at offset zero. + +The GPU/accelerator register windows listed in the sparse capability **are** +physically mmappable: ``mmap()`` on the VFIO device fd at the corresponding +BAR offset succeeds and yields a host-physical-backed mapping suitable for +KVM stage-2 installation. + +The CXL component register block itself **is not** mmappable. Any ``mmap()`` +request whose range overlaps ``[comp_reg_offset, comp_reg_offset + +comp_reg_size)`` returns ``-EINVAL``; those registers must be accessed through +the ``COMP_REGS`` device region. + + +DVSEC configuration space emulation +----------------------------------- + +With ``CONFIG_VFIO_CXL_CORE=y``, vfio-pci installs a handler for +``PCI_EXT_CAP_ID_DVSEC`` (``0x23``) in the config access table. Non-CXL +devices fall through as before. + +On CXL devices, writes to these DVSEC registers are caught and reflected in +``vdev->vconfig`` (shadow config space): + ++--------------------+--------+--------------------------------------------------+ +| Register | Offset | Emulation | ++====================+========+==================================================+ +| CXL Control | +0x0c | RWL; IO_Enable held at 1; locked when Lock | +| | | bit 0 is set. | ++--------------------+--------+--------------------------------------------------+ +| CXL Status | +0x0e | Bit 14 (Viral_Status) is RW1CS. | ++--------------------+--------+--------------------------------------------------+ +| CXL Control2 | +0x10 | Bits 1 and 2 forwarded to hardware. | ++--------------------+--------+--------------------------------------------------+ +| CXL Status2 | +0x12 | Bit 3 forwarded when Capability3 bit 3 is set. | ++--------------------+--------+--------------------------------------------------+ +| CXL Lock | +0x14 | RWO; once set, Control becomes read-only until | +| | | conventional reset. | ++--------------------+--------+--------------------------------------------------+ +| Range Base Hi/Lo | varies | Stored in vconfig; Base Low [27:0] reserved bits | +| | | cleared on write. | ++--------------------+--------+--------------------------------------------------+ + +Reads return the shadow. Read-only registers (Capability, Size High/Low) are +filled from hardware at open. + + +FLR and reset +------------- + +FLR goes through ``vfio_pci_ioctl_reset()``. The CXL-specific part is: + +1. ``vfio_cxl_prepare_reset()`` runs under the write side of + ``memory_lock``. It clears ``region_active`` and calls + ``unmap_mapping_range()`` on the DPA inode mapping so user PTEs go away. + Concurrent faults or fd I/O hit the inactive flag and error. IOMMU ATC must + drain before reset (see the DPA region notes above). + +2. After FLR, ``vfio_cxl_finish_reset()`` reads HDM hardware again into + ``comp_reg_virt[]``. If ``COMMITTED`` is set (common when firmware left the + decoder committed), ``region_active`` turns back on and the VMM can refault + without remapping. + + +Known limitations +----------------- + +**Pre-committed HDM decoder required** + See `Device detection`_ and the note there. + +**CXL hot-plug not supported** + Slots need to be present and programmed by firmware at boot. + +**CXL.cache Write-Back Invalidation not implemented** + For HDM-DB devices (``VFIO_CXL_CAP_CACHE_CAPABLE``), the kernel does not + run WBI before FLR. The VMM must do it and expose Back-Invalidation in the + guest topology where required. + + +VMM integration notes +--------------------- + +For a ``VFIO_CXL_CAP_FIRMWARE_COMMITTED`` device (what works today):: + + /* 1. Get device info and locate the CXL cap */ + vfio_device_get_info(fd, &dinfo); + assert(dinfo.flags & VFIO_DEVICE_FLAGS_CXL); + cxl = find_cap(&dinfo, VFIO_DEVICE_INFO_CAP_CXL); + + /* 2. Get DPA and COMP_REGS region sizes */ + get_region_info(fd, cxl->dpa_region_index, &dpa_ri); + get_region_info(fd, cxl->comp_regs_region_index, &comp_ri); + + /* 3. Map DPA region at a guest physical address */ + gpa_base = allocate_guest_phys(dpa_ri.size); + mmap(gpa_base, dpa_ri.size, PROT_READ|PROT_WRITE, + MAP_SHARED|MAP_FIXED, vfio_fd, + (off_t)cxl->dpa_region_index << VFIO_PCI_OFFSET_SHIFT); + + /* 4. Derive hdm_decoder_offset from COMP_REGS (see section below) */ + uint64_t hdm_decoder_offset = derive_hdm_offset(vfio_fd, comp_ri); + + /* 5. Write guest GPA into HDM Decoder 0 BASE via COMP_REGS pwrite */ + u32 base_hi = gpa_base >> 32; + comp_off = (off_t)cxl->comp_regs_region_index << VFIO_PCI_OFFSET_SHIFT; + pwrite(vfio_fd, &base_hi, 4, + comp_off + hdm_decoder_offset + CXL_HDM_DECODER0_BASE_HIGH_OFFSET); + + /* 6. Build guest CXL topology using gpa_base and dpa_ri.size */ + build_cfmws(gpa_base, dpa_ri.size); + + /* 7. If CACHE_CAPABLE: issue WBI before any guest FLR */ + +Extra detail: + +- DPA size is ``dpa_ri.size`` from region info. +- ``CXL_HDM_DECODER0_BASE_HIGH_OFFSET`` lives in ``include/uapi/cxl/cxl_regs.h``. +- On the BAR, ``mmaps[0].size`` from the sparse-mmap cap on + ``hdm_regs_bar_index`` splits GPU MMIO (BAR fd) from the CXL block (COMP_REGS + region). +- If ``VFIO_CXL_CAP_CACHE_CAPABLE`` is set, the guest CXL topology should + advertise Back-Invalidation and the VMM should run WBI before FLR. + + +Deriving HDM info from COMP_REGS +--------------------------------- + +``hdm_decoder_offset`` and ``hdm_count`` are not in ``vfio_device_info_cap_cxl`` +because both are directly readable from the ``COMP_REGS`` region. + +**Finding hdm_decoder_offset:** + +Read dwords from the COMP_REGS region starting at offset 0 (the CXL Capability +Array). ``comp_off`` is the VFIO file offset for the COMP_REGS region: +``(off_t)cxl->comp_regs_region_index << VFIO_PCI_OFFSET_SHIFT``:: + + /* Dword 0: CXL Capability Array Header */ + pread(fd, &hdr, 4, comp_off + 0); + /* bits[15:0] must be 1 (CM_CAP_HDR_CAP_ID) */ + /* bits[31:24] = number of capability entries */ + num_caps = (hdr >> 24) & 0xff; /* CXL_CM_CAP_HDR_ARRAY_SIZE_MASK */ + + /* Walk entries at dword 1..num_caps */ + for (i = 1; i <= num_caps; i++) { + pread(fd, &entry, 4, comp_off + i * 4); + cap_id = entry & 0xffff; /* CXL_CM_CAP_HDR_ID_MASK */ + if (cap_id == 0x5) { /* CXL_CM_CAP_CAP_ID_HDM */ + hdm_decoder_offset = (entry >> 20) & 0xfff; /* CXL_CM_CAP_PTR_MASK */ + break; + } + } + +**Finding hdm_count:** + +Read the HDM Decoder Capability register (HDMC) at ``hdm_decoder_offset + 0``:: + + pread(fd, &hdmc, 4, comp_off + hdm_decoder_offset); + field = hdmc & 0xf; /* CXL_HDM_DECODER_COUNT_MASK bits[3:0] */ + hdm_count = field ? field * 2 : 1; /* 0→1, N→N*2 decoders */ + +All constants are in ``include/uapi/cxl/cxl_regs.h``. + + +Kernel configuration +-------------------- + +``CONFIG_VFIO_CXL_CORE`` (bool) + CXL Type-2 passthrough in ``vfio-pci-core``. Needs ``CONFIG_VFIO_PCI_CORE``, + ``CONFIG_CXL_BUS``, and ``CONFIG_CXL_MEM``. + +References +---------- + +* CXL Specification 4.0, 8.1.3 - PCIe DVSEC for CXL Devices +* CXL Specification 4.0, 8.2.4.20 - CXL HDM Decoder Capability Structure +* ``include/uapi/linux/vfio.h`` - ``VFIO_DEVICE_INFO_CAP_CXL``, + ``VFIO_REGION_SUBTYPE_CXL``, ``VFIO_REGION_SUBTYPE_CXL_COMP_REGS`` +* ``include/uapi/cxl/cxl_regs.h`` - ``CXL_CM_OFFSET``, + ``CXL_CM_CAP_HDR_ARRAY_SIZE_MASK``, ``CXL_CM_CAP_HDR_ID_MASK``, + ``CXL_CM_CAP_PTR_MASK``, ``CXL_HDM_DECODER_COUNT_MASK``, + ``CXL_HDM_DECODER0_BASE_HIGH_OFFSET`` From c40dfd764bda0a7a2ebf515b0561080062f024e4 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 1 Apr 2026 20:09:17 +0530 Subject: [PATCH 406/464] selftests/vfio: Add CXL Type-2 VFIO assignment test BugLink: https://bugs.launchpad.net/bugs/2152222 Add vfio_cxl_type2_test and build it from the vfio selftest Makefile. The binary expects a PCI BDF (argv or VFIO_SELFTESTS_BDF) with the device already on vfio-pci and CONFIG_VFIO_CXL_CORE enabled. It exercises: - VFIO_DEVICE_GET_INFO, - GET_REGION_INFO, - VFIO_DEVICE_INFO_CAP_CXL capability list, - sparse component-BAR vs DPA/COMP_REG regions, - HDM decoder emulation (masks, commit, lock), - DVSEC-backed config where the driver exposes it. Large region read/write loops and FLR-heavy test cases are still pending; Need to revisit these in next version of patches. vfio_pci_device_setup() skips auto-mmap for BARs that carry sparse-mmap capabilities; those require the caller to mmap only the windows advertised by the capability. Signed-off-by: Manish Honap (backported from https://lore.kernel.org/r/20260401143917.108413-21-mhonap@nvidia.com) [kobak: Treat DPA mmap as optional because BOS intentionally withholds VFIO_REGION_INFO_FLAG_MMAP for the DPA region.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- tools/testing/selftests/vfio/Makefile | 1 + .../selftests/vfio/lib/vfio_pci_device.c | 3 +- .../selftests/vfio/vfio_cxl_type2_test.c | 923 ++++++++++++++++++ 3 files changed, 926 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/vfio/vfio_cxl_type2_test.c diff --git a/tools/testing/selftests/vfio/Makefile b/tools/testing/selftests/vfio/Makefile index 0684932d91bfc..25f2a9420ef6d 100644 --- a/tools/testing/selftests/vfio/Makefile +++ b/tools/testing/selftests/vfio/Makefile @@ -12,6 +12,7 @@ TEST_GEN_PROGS += vfio_iommufd_setup_test TEST_GEN_PROGS += vfio_pci_device_test TEST_GEN_PROGS += vfio_pci_device_init_perf_test TEST_GEN_PROGS += vfio_pci_driver_test +TEST_GEN_PROGS += vfio_cxl_type2_test TEST_FILES += scripts/cleanup.sh TEST_FILES += scripts/lib.sh diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/testing/selftests/vfio/lib/vfio_pci_device.c index fc75e04ef010c..ccefc5adc64c9 100644 --- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c +++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c @@ -281,7 +281,8 @@ static void vfio_pci_device_setup(struct vfio_pci_device *device) struct vfio_pci_bar *bar = device->bars + i; vfio_pci_region_get(device, i, &bar->info); - if (bar->info.flags & VFIO_REGION_INFO_FLAG_MMAP) + if ((bar->info.flags & VFIO_REGION_INFO_FLAG_MMAP) && + !(bar->info.flags & VFIO_REGION_INFO_FLAG_CAPS)) vfio_pci_bar_map(device, i); } diff --git a/tools/testing/selftests/vfio/vfio_cxl_type2_test.c b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c new file mode 100644 index 0000000000000..8d932250864f4 --- /dev/null +++ b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c @@ -0,0 +1,923 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * vfio_cxl_type2_test - selftests for CXL Type-2 device passthrough via vfio-pci + * + * Tests the UAPI and emulation layer introduced by CONFIG_VFIO_CXL_CORE + * + * Usage: + * ./vfio_cxl_type2_test + * or set the environment variable VFIO_SELFTESTS_BDF before running. + * + * The device must be a CXL Type-2 device (e.g. a GPU with coherent memory). + * Tests adapt automatically to firmware-committed (COMMITTED/COMMIT_LOCK set) + * and CONFIG_LOCK-set hardware states instead of skipping. + * + * Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +#include + +#include "kselftest_harness.h" + +/* Userspace equivalents of kernel helpers not available in user headers */ +#ifndef BIT +#define BIT(n) (1u << (n)) +#endif +#ifndef GENMASK +#define GENMASK(h, l) (((~0u) >> (31 - (h))) & ((~0u) << (l))) +#endif +#define VFIO_PCI_INDEX_TO_OFFSET(idx) ((uint64_t)(idx) << 40) + +static const char *device_bdf; + +/* ------------------------------------------------------------------ */ +/* CXL UAPI constants (mirrors include/uapi/linux/vfio.h) */ +/* ------------------------------------------------------------------ */ + +#define VFIO_DEVICE_INFO_CAP_CXL 6 + +#define PCI_VENDOR_ID_CXL 0x1e98 + +#ifndef VFIO_REGION_SUBTYPE_CXL +#define VFIO_REGION_SUBTYPE_CXL 1 +#endif +#ifndef VFIO_REGION_SUBTYPE_CXL_COMP_REGS +#define VFIO_REGION_SUBTYPE_CXL_COMP_REGS 2 +#endif + +/* + * HDM Decoder register layout within the component register block. + * Offsets relative to the start of the HDM decoder capability block. + * The HDM decoder block begins at hdm_decoder_offset within the COMP_REGS + * region; add hdm_decoder_offset before indexing into the region. + */ +#define HDM_GLOBAL_CTRL_OFFSET 0x04 +#define HDM_DECODER_FIRST_OFFSET 0x10 +#define HDM_DECODER_STRIDE 0x20 +#define HDM_DECODER_BASE_LO 0x00 +#define HDM_DECODER_BASE_HI 0x04 +#define HDM_DECODER_SIZE_LO 0x08 +#define HDM_DECODER_SIZE_HI 0x0c +#define HDM_DECODER_CTRL 0x10 + +#define HDM_CTRL_COMMIT BIT(9) +#define HDM_CTRL_COMMITTED BIT(10) +#define HDM_CTRL_RESERVED_MASK (BIT(15) | GENMASK(31, 28)) + +#define CXL_LOCK_RESERVED_MASK GENMASK(15, 1) + +/* ------------------------------------------------------------------ */ +/* Helpers */ +/* ------------------------------------------------------------------ */ + +/* + * Walk the vfio_device_info capability chain embedded in @buf. + * Returns a pointer to the capability with the given @id, or NULL. + */ +static const struct vfio_info_cap_header * +find_device_cap(const void *buf, size_t bufsz, uint16_t id) +{ + const struct vfio_device_info *info = buf; + const struct vfio_info_cap_header *cap; + + if (!(info->flags & VFIO_DEVICE_FLAGS_CAPS) || !info->cap_offset) + return NULL; + + cap = (const struct vfio_info_cap_header *) + ((const char *)buf + info->cap_offset); + + while (true) { + if (cap->id == id) + return cap; + if (!cap->next) + return NULL; + cap = (const struct vfio_info_cap_header *) + ((const char *)buf + cap->next); + if ((const char *)cap + sizeof(*cap) > (const char *)buf + bufsz) + return NULL; + } +} + +/* + * Walk the vfio_region_info capability chain embedded in @buf. + * Returns a pointer to the capability with the given @id, or NULL. + * @buf must have been obtained from VFIO_DEVICE_GET_REGION_INFO with + * argsz large enough to hold the full capability chain. + */ +static const struct vfio_info_cap_header * +find_region_cap(const void *buf, size_t bufsz, uint16_t id) +{ + const struct vfio_region_info *info = buf; + const struct vfio_info_cap_header *cap; + + if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS) || !info->cap_offset) + return NULL; + + cap = (const struct vfio_info_cap_header *) + ((const char *)buf + info->cap_offset); + + while (true) { + if (cap->id == id) + return cap; + if (!cap->next) + return NULL; + cap = (const struct vfio_info_cap_header *) + ((const char *)buf + cap->next); + if ((const char *)cap + sizeof(*cap) > (const char *)buf + bufsz) + return NULL; + } +} + +/* + * Read a 32-bit value from the COMP_REGS region at @offset (HDM-relative). + */ +static uint32_t comp_regs_read32(struct vfio_pci_device *dev, + uint32_t region_idx, uint64_t offset) +{ + uint32_t val; + loff_t pos = (loff_t)VFIO_PCI_INDEX_TO_OFFSET(region_idx) + offset; + ssize_t r; + + r = pread(dev->fd, &val, sizeof(val), pos); + if (r != sizeof(val)) + return ~0u; + return val; +} + +/* + * Write a 32-bit value to the COMP_REGS region at @offset. + * Mirrors the error-propagation contract of comp_regs_read32() which returns + * ~0u on a short or failed pread. + */ +static ssize_t comp_regs_write32(struct vfio_pci_device *dev, + uint32_t region_idx, uint64_t offset, + uint32_t val) +{ + loff_t pos = (loff_t)VFIO_PCI_INDEX_TO_OFFSET(region_idx) + offset; + + return pwrite(dev->fd, &val, sizeof(val), pos); +} + +/* + * HDM register accessors. + * + * The COMP_REGS region starts at the CXL component register block + * start (comp_reg_offset). The HDM decoder capability block begins at + * hdm_decoder_offset within this region. These helpers add + * hdm_decoder_offset so that callers can continue to use the HDM-relative + * offsets defined by the macros above. + */ +static uint32_t hdm_regs_read32(struct vfio_pci_device *dev, + uint32_t region_idx, + uint64_t hdm_decoder_offset, + uint64_t hdm_off) +{ + return comp_regs_read32(dev, region_idx, hdm_decoder_offset + hdm_off); +} + +static ssize_t hdm_regs_write32(struct vfio_pci_device *dev, + uint32_t region_idx, + uint64_t hdm_decoder_offset, + uint64_t hdm_off, + uint32_t val) +{ + return comp_regs_write32(dev, region_idx, hdm_decoder_offset + hdm_off, val); +} + +/* + * Traverse the CXL Capability Array at COMP_REGS region offset 0 to find the + * HDM Decoder capability block offset and decoder count. + * + * COMP_REGS region layout at offset 0 (CXL Capability Array): + * Dword 0 bits[31:24] (CXL_CM_CAP_HDR_ARRAY_SIZE_MASK): entry count N. + * Dwords 1..N at offset (cap*4): bits[15:0] = cap ID (CXL_CM_CAP_HDR_ID_MASK), + * bits[31:20] = byte offset from COMP_REGS start (CXL_CM_CAP_PTR_MASK). + * + * HDM Decoder cap ID = 0x5 (CXL_CM_CAP_CAP_ID_HDM). + * HDMC at hdm_decoder_offset+0 bits[3:0]: count = (field==0) ? 1 : field*2. + * + * Returns true on success; sets *hdm_off and *hdm_cnt. + */ +static bool find_hdm_decoder_info(struct vfio_pci_device *dev, + uint32_t comp_regs_idx, + uint64_t *hdm_off, uint8_t *hdm_cnt) +{ + uint32_t hdr, num_caps, i; + + /* Read CXL Capability Array Header (dword 0) */ + hdr = comp_regs_read32(dev, comp_regs_idx, 0); + if (hdr == ~0u) + return false; + + /* Validate: bits[15:0] must be CM_CAP_HDR_CAP_ID (1) */ + if ((hdr & 0xffff) != 1) + return false; + + /* bits[31:24] = number of capability entries */ + num_caps = (hdr >> 24) & 0xff; + + for (i = 1; i <= num_caps; i++) { + uint32_t entry = comp_regs_read32(dev, comp_regs_idx, i * 4); + uint32_t cap_id = entry & 0xffff; /* CXL_CM_CAP_HDR_ID_MASK */ + + if (cap_id == 0x5) { /* CXL_CM_CAP_CAP_ID_HDM */ + uint32_t hdmc; + uint32_t field; + + /* bits[31:20]: byte offset from COMP_REGS start */ + *hdm_off = (entry >> 20) & 0xfff; + + /* Read HDMC register at hdm_decoder_offset + 0 */ + hdmc = comp_regs_read32(dev, comp_regs_idx, *hdm_off); + if (hdmc == ~0u) + return false; + + /* bits[3:0]: 0 = 1 decoder, N = N*2 decoders */ + field = hdmc & 0xf; + *hdm_cnt = field ? (uint8_t)(field * 2) : 1; + return true; + } + } + return false; +} + +/* + * Find the CXL DVSEC capability base in config space. + */ +#define PCI_DVSEC_VENDOR_ID_CXL 0x1e98 +#define PCI_DVSEC_ID_CXL_DEVICE 0x0000 +#define PCI_EXT_CAP_ID_DVSEC 0x23 + +static uint16_t find_cxl_dvsec(struct vfio_pci_device *dev) +{ + uint16_t pos = PCI_CFG_SPACE_SIZE; /* 0x100 */ + int iter = 0; + + while (pos && iter++ < 64) { + uint32_t hdr = vfio_pci_config_readl(dev, pos); + uint32_t hdr1, hdr2; + uint16_t cap_id = hdr & 0xffff; + uint16_t next = (hdr >> 20) & 0xffc; + + if (cap_id == PCI_EXT_CAP_ID_DVSEC) { + hdr1 = vfio_pci_config_readl(dev, pos + 4); + hdr2 = vfio_pci_config_readl(dev, pos + 8); + /* + * PCIe DVSEC Header 1 layout (Table 9-16): + * Bits [15: 0] = DVSEC Vendor ID + * Bits [19:16] = DVSEC Revision + * Bits [31:20] = DVSEC Length + * DVSEC Header 2 layout: + * Bits [15: 0] = DVSEC ID + */ + if ((hdr1 & 0xffff) == PCI_DVSEC_VENDOR_ID_CXL && + (hdr2 & 0xffff) == PCI_DVSEC_ID_CXL_DEVICE) + return pos; + } + pos = next; + } + return 0; +} + +/* ------------------------------------------------------------------ */ +/* Fixture */ +/* ------------------------------------------------------------------ */ + +FIXTURE(cxl_type2) { + struct iommu *iommu; + struct vfio_pci_device *dev; + + /* Filled in during FIXTURE_SETUP from the CXL cap */ + struct vfio_device_info_cap_cxl cxl_cap; + uint16_t dvsec_base; + + /* + * Sizes derived from VFIO_DEVICE_GET_REGION_INFO at setup time. + * These are not in the CXL cap struct; query the region directly. + */ + uint64_t dpa_size; /* size of the DPA region */ + uint64_t hdm_regs_size; /* size of the COMP_REGS region */ + + /* + * HDM decoder info derived from the COMP_REGS region at setup time. + * hdm_count and hdm_decoder_offset are no longer in the UAPI cap struct; + * they are derived by traversing the CXL Capability Array and reading + * the HDM Decoder Capability register (HDMC). + */ + uint64_t hdm_decoder_offset; /* byte offset in COMP_REGS to HDM block */ + uint8_t hdm_count; /* number of HDM decoders */ + + /* DPA mmap pointer (may be NULL if test skips mmap sub-tests) */ + void *dpa_mmap; + size_t dpa_mmap_size; +}; + +FIXTURE_SETUP(cxl_type2) +{ + uint8_t infobuf[512] = {}; + struct vfio_device_info *info = (void *)infobuf; + const struct vfio_device_info_cap_cxl *cap; + + self->iommu = iommu_init(default_iommu_mode); + self->dev = vfio_pci_device_init(device_bdf, self->iommu); + + /* Query device info with space for capability chain */ + info->argsz = sizeof(infobuf); + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_INFO, info)); + + if (!(info->flags & VFIO_DEVICE_FLAGS_CXL)) { + printf("Device %s is not a CXL Type-2 device — skipping\n", + device_bdf); + SKIP(return, "not a CXL Type-2 device"); + } + + cap = (const struct vfio_device_info_cap_cxl *) + find_device_cap(infobuf, sizeof(infobuf), + VFIO_DEVICE_INFO_CAP_CXL); + ASSERT_NE(NULL, cap); + memcpy(&self->cxl_cap, cap, sizeof(*cap)); + + /* + * Populate dpa_size and hdm_regs_size from region queries. + */ + { + struct vfio_region_info ri = { .argsz = sizeof(ri) }; + + ri.index = cap->dpa_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, + VFIO_DEVICE_GET_REGION_INFO, &ri)); + self->dpa_size = ri.size; + + ri.index = cap->comp_regs_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, + VFIO_DEVICE_GET_REGION_INFO, &ri)); + self->hdm_regs_size = ri.size; + } + + /* + * Derive hdm_decoder_offset and hdm_count from the COMP_REGS region. + * These fields were removed from vfio_device_info_cap_cxl to keep the + * UAPI minimal; userspace derives them via the CXL Capability Array. + */ + ASSERT_TRUE(find_hdm_decoder_info(self->dev, + cap->comp_regs_region_index, + &self->hdm_decoder_offset, + &self->hdm_count)); + + self->dvsec_base = find_cxl_dvsec(self->dev); + self->dpa_mmap = MAP_FAILED; + self->dpa_mmap_size = 0; +} + +FIXTURE_TEARDOWN(cxl_type2) +{ + if (self->dpa_mmap != MAP_FAILED && self->dpa_mmap_size) + munmap(self->dpa_mmap, self->dpa_mmap_size); + vfio_pci_device_cleanup(self->dev); + iommu_cleanup(self->iommu); +} + +/* ------------------------------------------------------------------ */ +/* Tests: VFIO_DEVICE_GET_INFO */ +/* ------------------------------------------------------------------ */ + +/* + * CXL and PCI flags must both be set; CAPS must be set since we have a cap. + */ +TEST_F(cxl_type2, device_flags) +{ + uint8_t infobuf[512] = {}; + struct vfio_device_info *info = (void *)infobuf; + + info->argsz = sizeof(infobuf); + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_INFO, info)); + + ASSERT_TRUE(info->flags & VFIO_DEVICE_FLAGS_CXL); + ASSERT_TRUE(info->flags & VFIO_DEVICE_FLAGS_PCI); + ASSERT_TRUE(info->flags & VFIO_DEVICE_FLAGS_CAPS); + + printf("device flags: 0x%x num_regions: %u\n", + info->flags, info->num_regions); +} + +/* + * The CXL capability must report sane HDM and DPA values. + * hdm_count and hdm_decoder_offset are no longer in the cap struct; they + * are derived from the COMP_REGS region in FIXTURE_SETUP and stored in + * self->hdm_count and self->hdm_decoder_offset. + */ +TEST_F(cxl_type2, cxl_cap_fields) +{ + const struct vfio_device_info_cap_cxl *c = &self->cxl_cap; + + ASSERT_EQ(VFIO_DEVICE_INFO_CAP_CXL, c->header.id); + ASSERT_EQ(1, c->header.version); + + /* Must have at least one HDM decoder (derived from HDMC bits[3:0]) */ + ASSERT_GT(self->hdm_count, 0); + + /* COMP_REGS region size must be non-zero and 4-byte aligned */ + ASSERT_GT(self->hdm_regs_size, 0ULL); + ASSERT_EQ(0ULL, self->hdm_regs_size % 4); + + /* + * hdm_decoder_offset is derived from the CXL Capability Array. + * It must be: + * - non-zero (the CXL Capability Array Header precedes the HDM block) + * - dword-aligned + * - strictly less than hdm_regs_size (HDM block fits in the region) + */ + ASSERT_GT(self->hdm_decoder_offset, 0ULL); + ASSERT_EQ(0ULL, self->hdm_decoder_offset % 4); + ASSERT_LT(self->hdm_decoder_offset, self->hdm_regs_size); + + /* Region indices must not be ~0U (sentinel for "not found") */ + ASSERT_NE(~0U, c->dpa_region_index); + ASSERT_NE(~0U, c->comp_regs_region_index); + + /* The two regions must be distinct */ + ASSERT_NE(c->dpa_region_index, c->comp_regs_region_index); + + /* + * FIRMWARE_COMMITTED: decoder was pre-programmed by firmware; DPA + * region is immediately live. dpa_size must be non-zero in this case. + */ + if (c->flags & VFIO_CXL_CAP_FIRMWARE_COMMITTED) + ASSERT_GT(self->dpa_size, 0ULL); + + printf("hdm_count=%u dpa_size=0x%llx hdm_regs_size=0x%llx " + "hdm_decoder_offset=0x%llx " + "dpa_idx=%u comp_regs_idx=%u flags=0x%x " + "(firmware_committed=%d cache_capable=%d)\n", + self->hdm_count, (unsigned long long)self->dpa_size, + (unsigned long long)self->hdm_regs_size, + (unsigned long long)self->hdm_decoder_offset, + c->dpa_region_index, c->comp_regs_region_index, c->flags, + !!(c->flags & VFIO_CXL_CAP_FIRMWARE_COMMITTED), + !!(c->flags & VFIO_CXL_CAP_CACHE_CAPABLE)); +} + +/* ------------------------------------------------------------------ */ +/* Tests: VFIO_DEVICE_GET_REGION_INFO */ +/* ------------------------------------------------------------------ */ + +/* + * The component register BAR must report its real (non-zero) size with + * READ/WRITE/MMAP flags and a VFIO_REGION_INFO_CAP_SPARSE_MMAP capability. + * The sparse areas advertise the GPU/accelerator register windows — the + * mmappable parts of the BAR that do NOT contain CXL component registers. + * + * Three topologies are possible depending on where comp_regs sits in the BAR: + * Topology A [gpu_regs | comp_regs] → 1 area: [0, comp_reg_offset) + * Topology B [comp_regs | gpu_regs] → 1 area: [comp_end, bar_len) + * Topology C [gpu_regs | comp_regs | gpu_regs] → 2 areas + * + * In all cases each sparse area is a GPU register window; no area may overlap + * the CXL component register block at [comp_reg_offset, comp_reg_offset + + * comp_reg_size). + */ +TEST_F(cxl_type2, component_bar_sparse_mmap) +{ + struct vfio_region_info probe = { .argsz = sizeof(probe) }; + struct vfio_region_info *reg; + const struct vfio_region_info_cap_sparse_mmap *sparse; + uint32_t bar_idx = self->cxl_cap.hdm_regs_bar_index; + uint64_t comp_reg_offset; + uint64_t total_gpu_size; + uint8_t *buf; + uint32_t needed; + uint32_t i; + + /* First probe: learn required buffer size and basic flags */ + probe.index = bar_idx; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, &probe)); + + ASSERT_GT(probe.size, 0ULL); + ASSERT_TRUE(probe.flags & VFIO_REGION_INFO_FLAG_READ); + ASSERT_TRUE(probe.flags & VFIO_REGION_INFO_FLAG_WRITE); + ASSERT_TRUE(probe.flags & VFIO_REGION_INFO_FLAG_MMAP); + + /* Kernel must signal caps are present by expanding argsz */ + ASSERT_GT(probe.argsz, (uint32_t)sizeof(probe)); + needed = probe.argsz; + + buf = calloc(1, needed); + ASSERT_NE(NULL, buf); + reg = (struct vfio_region_info *)buf; + reg->argsz = needed; + reg->index = bar_idx; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, reg)); + + /* Must carry a sparse-mmap cap */ + sparse = (const struct vfio_region_info_cap_sparse_mmap *) + find_region_cap(buf, needed, VFIO_REGION_INFO_CAP_SPARSE_MMAP); + ASSERT_NE(NULL, sparse); + + /* 1 area (topology A or B) or 2 areas (topology C); never more */ + ASSERT_GE(sparse->nr_areas, 1U); + ASSERT_LE(sparse->nr_areas, 2U); + + /* + * comp_reg_offset = hdm_regs_offset - CXL_CM_OFFSET. + * hdm_regs_offset is the BAR-relative address of the CXL.mem area + * start, which sits CXL_CM_OFFSET (0x1000) bytes into the component + * register block. + */ + ASSERT_GE(self->cxl_cap.hdm_regs_offset, (uint64_t)CXL_CM_OFFSET); + comp_reg_offset = self->cxl_cap.hdm_regs_offset - CXL_CM_OFFSET; + + total_gpu_size = 0; + for (i = 0; i < sparse->nr_areas; i++) { + uint64_t area_start = sparse->areas[i].offset; + uint64_t area_end = area_start + sparse->areas[i].size; + + /* Each area must be non-empty and fit within the BAR */ + ASSERT_GT(sparse->areas[i].size, 0ULL); + ASSERT_LE(area_end, reg->size); + + /* + * No sparse area may overlap the CXL component register block. + * Use hdm_regs_offset as a witness point: it is comp_reg_offset + * + CXL_CM_OFFSET, guaranteed inside the block. + */ + ASSERT_FALSE(area_start <= self->cxl_cap.hdm_regs_offset && + self->cxl_cap.hdm_regs_offset < area_end); + + total_gpu_size += sparse->areas[i].size; + + printf(" sparse area[%u]: offset=0x%llx size=0x%llx\n", i, + (unsigned long long)area_start, + (unsigned long long)sparse->areas[i].size); + } + + /* GPU windows together must be strictly smaller than the full BAR */ + ASSERT_LT(total_gpu_size, reg->size); + + printf("component BAR %u: bar_size=0x%llx comp_reg_offset=0x%llx " + "nr_areas=%u total_gpu=0x%llx flags=0x%x\n", + bar_idx, (unsigned long long)reg->size, + (unsigned long long)comp_reg_offset, + sparse->nr_areas, (unsigned long long)total_gpu_size, + reg->flags); + + free(buf); +} + +/* + * DPA region must be readable and writable. MMAP is optional because kernels + * may withhold it until the DPA backing is known to be CPU-mappable safely. + * Its size must be non-zero (verified in fixture setup via self->dpa_size). + */ +TEST_F(cxl_type2, dpa_region_info) +{ + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + + reg.index = self->cxl_cap.dpa_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ®)); + + ASSERT_EQ(self->dpa_size, reg.size); + ASSERT_TRUE(reg.flags & VFIO_REGION_INFO_FLAG_READ); + ASSERT_TRUE(reg.flags & VFIO_REGION_INFO_FLAG_WRITE); + + printf("DPA region: size=0x%llx offset=0x%llx flags=0x%x\n", + (unsigned long long)reg.size, + (unsigned long long)reg.offset, reg.flags); +} + +/* + * COMP_REGS region must be readable and writable but not mmappable. + * Its size covers [comp_reg_offset, comp_reg_offset + hdm_regs_size), which + * includes both the CXL Capability Array prefix (hdm_decoder_offset bytes) + * and the HDM decoder block. Size is available in self->hdm_regs_size + * (populated from this same region query at fixture setup time). + */ +TEST_F(cxl_type2, comp_regs_region_info) +{ + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + + reg.index = self->cxl_cap.comp_regs_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ®)); + + ASSERT_EQ(self->hdm_regs_size, reg.size); + ASSERT_TRUE(reg.flags & VFIO_REGION_INFO_FLAG_READ); + ASSERT_TRUE(reg.flags & VFIO_REGION_INFO_FLAG_WRITE); + ASSERT_FALSE(reg.flags & VFIO_REGION_INFO_FLAG_MMAP); + + printf("COMP_REGS region: size=0x%llx offset=0x%llx flags=0x%x\n", + (unsigned long long)reg.size, + (unsigned long long)reg.offset, reg.flags); +} + +/* ------------------------------------------------------------------ */ +/* Tests: DPA region mmap */ +/* ------------------------------------------------------------------ */ + +/* + * mmap() the DPA region and verify the first page can be read. + * The region uses lazy fault insertion so the first access triggers the + * vfio_cxl_region_page_fault path. + */ +TEST_F(cxl_type2, dpa_mmap_fault) +{ + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + size_t map_size; + void *ptr; + uint8_t *p; + uint8_t val; + + reg.index = self->cxl_cap.dpa_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ®)); + + if (!(reg.flags & VFIO_REGION_INFO_FLAG_MMAP)) + SKIP(return, "DPA region does not advertise mmap"); + + /* Map just the first 2MB or the full region, whichever is smaller */ + map_size = (size_t)reg.size < (size_t)(2 * SZ_1M) + ? (size_t)reg.size : (size_t)(2 * SZ_1M); + + ptr = mmap(NULL, map_size, PROT_READ | PROT_WRITE, + MAP_SHARED, self->dev->fd, (off_t)reg.offset); + ASSERT_NE(MAP_FAILED, ptr); + + self->dpa_mmap = ptr; + self->dpa_mmap_size = map_size; + + /* First access - triggers vmf_insert_pfn */ + p = (uint8_t *)ptr; + val = *p; + (void)val; + + printf("DPA mmap: ptr=%p size=0x%zx first byte=0x%02x\n", + ptr, map_size, (uint8_t)val); + + /* Write a pattern and read it back */ + *p = 0xab; + ASSERT_EQ(0xab, *p); +} + +/* + * mmap() of the COMP_REGS region (no MMAP flag) must fail. + */ +TEST_F(cxl_type2, comp_regs_no_mmap) +{ + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + void *ptr; + + reg.index = self->cxl_cap.comp_regs_region_index; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ®)); + + ptr = mmap(NULL, (size_t)reg.size, PROT_READ, + MAP_SHARED, self->dev->fd, (off_t)reg.offset); + ASSERT_EQ(MAP_FAILED, ptr); + + printf("mmap of COMP_REGS correctly failed (errno=%d)\n", errno); +} + +/* + * mmap() of the CXL component register block within the component BAR must + * fail with EINVAL. The kernel blocks any mmap request whose range overlaps + * [comp_reg_offset, comp_reg_offset + comp_reg_size) even though the BAR as + * a whole carries the MMAP flag (GPU windows are mmappable). + * + * hdm_regs_offset (= comp_reg_offset + CXL_CM_OFFSET) is a page-aligned + * address guaranteed to lie inside the component register block. + */ +TEST_F(cxl_type2, comp_reg_mmap_blocked) +{ + struct vfio_region_info bar_reg = { .argsz = sizeof(bar_reg) }; + void *ptr; + + bar_reg.index = self->cxl_cap.hdm_regs_bar_index; + ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, &bar_reg)); + ASSERT_TRUE(bar_reg.flags & VFIO_REGION_INFO_FLAG_MMAP); + + /* + * hdm_regs_offset is page-aligned and is comp_reg_offset + CXL_CM_OFFSET + * (0x1000), so it is always within the component register block. + */ + ASSERT_EQ(0ULL, self->cxl_cap.hdm_regs_offset % SZ_4K); + + ptr = mmap(NULL, (size_t)SZ_4K, PROT_READ, + MAP_SHARED, self->dev->fd, + (off_t)(bar_reg.offset + self->cxl_cap.hdm_regs_offset)); + ASSERT_EQ(MAP_FAILED, ptr); + ASSERT_EQ(EINVAL, errno); + + printf("comp_reg_mmap_blocked: hdm_regs_offset=0x%llx correctly blocked " + "(errno=%d)\n", + (unsigned long long)self->cxl_cap.hdm_regs_offset, errno); +} + +/* ------------------------------------------------------------------ */ +/* Tests: COMP_REGS region (HDM decoder emulation) */ +/* ------------------------------------------------------------------ */ + +/* + * Reading HDM Capability (offset 0x00) must return a non-zero value + * consistent with at least one decoder being present. + * Bits [3:0] encode the HDM decoder count. + */ +TEST_F(cxl_type2, hdm_cap_read) +{ + uint32_t cap; + uint32_t idx = self->cxl_cap.comp_regs_region_index; + uint64_t hdm_off = self->hdm_decoder_offset; + + cap = hdm_regs_read32(self->dev, idx, hdm_off, CXL_HDM_DECODER_CAP_OFFSET); + ASSERT_NE(~0u, cap); + + /* + * Verify the live HDMC register matches the count we derived in setup. + * Encoding: bits[3:0] = 0 → 1 decoder; N → N*2 decoders. + */ + { + uint32_t field = cap & 0xf; + uint8_t expected = field ? (uint8_t)(field * 2) : 1; + + ASSERT_EQ(self->hdm_count, expected); + } + + printf("HDM Capability register: 0x%08x decoder_count_field=%u hdm_count=%u\n", + cap, cap & 0xf, self->hdm_count); +} + +/* + * HDM decoder COMMIT -> COMMITTED transition. + * + * On firmware-committed hardware (COMMITTED already set) the COMMIT path + * is not exercisable. Instead verify the committed state is self-consistent: + * COMMITTED set, BASE/SIZE non-zero and large enough to cover dpa_size, and + * reserved bits cleared by the emulation layer. + * + * On hardware where the decoder is not yet committed, exercise the full + * COMMIT=1 -> COMMITTED=1 path followed by COMMIT=0 -> COMMITTED=0. + */ +TEST_F(cxl_type2, hdm_ctrl_commit_to_committed) +{ + uint32_t idx = self->cxl_cap.comp_regs_region_index; + uint64_t hdm_off = self->hdm_decoder_offset; + uint64_t base_lo_off = HDM_DECODER_FIRST_OFFSET + HDM_DECODER_BASE_LO; + uint64_t base_hi_off = HDM_DECODER_FIRST_OFFSET + HDM_DECODER_BASE_HI; + uint64_t size_lo_off = HDM_DECODER_FIRST_OFFSET + HDM_DECODER_SIZE_LO; + uint64_t size_hi_off = HDM_DECODER_FIRST_OFFSET + HDM_DECODER_SIZE_HI; + uint64_t ctrl_off = HDM_DECODER_FIRST_OFFSET + HDM_DECODER_CTRL; + uint32_t ctrl_readback; + uint32_t base_lo, base_hi, size_lo, size_hi; + uint64_t dec_base, dec_size; + + ctrl_readback = hdm_regs_read32(self->dev, idx, hdm_off, ctrl_off); + + if (ctrl_readback & HDM_CTRL_COMMITTED) { + /* + * Firmware-committed decoder: verify the committed state is + * self-consistent. + * + * BASE is expected to be zero: the kernel clears BASE_LO/HI in + * the shadow for firmware-committed decoders so that the host + * HPA does not leak to the guest. The VMM will write the guest + * GPA into BASE before booting the VM. + * + * SIZE must cover at least dpa_size, and reserved bits must be + * clear (the emulation scrubs them on every write). + */ + base_lo = hdm_regs_read32(self->dev, idx, hdm_off, base_lo_off); + base_hi = hdm_regs_read32(self->dev, idx, hdm_off, base_hi_off); + size_lo = hdm_regs_read32(self->dev, idx, hdm_off, size_lo_off); + size_hi = hdm_regs_read32(self->dev, idx, hdm_off, size_hi_off); + dec_base = ((uint64_t)base_hi << 32) | (base_lo & ~GENMASK(27, 0)); + dec_size = ((uint64_t)size_hi << 32) | (size_lo & ~GENMASK(27, 0)); + + ASSERT_EQ(0ULL, dec_base); + ASSERT_GE(dec_size, self->dpa_size); + ASSERT_EQ(0u, ctrl_readback & HDM_CTRL_RESERVED_MASK); + + printf("Decoder 0 firmware-committed: ctrl=0x%08x " + "base=0x%llx (zeroed by kernel) size=0x%llx dpa_size=0x%llx\n", + ctrl_readback, + (unsigned long long)dec_base, + (unsigned long long)dec_size, + (unsigned long long)self->dpa_size); + return; + } + + /* Decoder not committed: exercise COMMIT=1 -> COMMITTED=1 path */ + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, base_lo_off, 0x10000000)); + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, base_hi_off, 0)); + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, size_lo_off, 0x10000000)); + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, size_hi_off, 0)); + + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, ctrl_off, HDM_CTRL_COMMIT)); + ctrl_readback = hdm_regs_read32(self->dev, idx, hdm_off, ctrl_off); + ASSERT_TRUE(ctrl_readback & HDM_CTRL_COMMITTED); + + printf("HDM decoder 0 CTRL after COMMIT=1: 0x%08x (COMMITTED set)\n", + ctrl_readback); + + ASSERT_EQ(4, hdm_regs_write32(self->dev, idx, hdm_off, ctrl_off, 0)); + ctrl_readback = hdm_regs_read32(self->dev, idx, hdm_off, ctrl_off); + ASSERT_FALSE(ctrl_readback & HDM_CTRL_COMMITTED); + + printf("HDM decoder 0 CTRL after COMMIT=0: 0x%08x (COMMITTED cleared)\n", + ctrl_readback); +} + +/* + * CXL Lock (DVSEC offset 0x14): + * - Reserved bits GENMASK(15,1) must be cleared. + * - Once locked, CXL Control writes must be discarded. + * + * On firmware-committed hardware CONFIG_LOCK is set before OS load by the + * BIOS. + * In this case verify: + * (a) Lock reserved bits are zero, + * (b) a write to CXL Control is silently discarded by the emulation. + * Both are directly testable without needing to transition from unlocked + * to locked. + * + * On hardware where CONFIG_LOCK is not yet set, exercise the full sequence: + * write reserved bits (must be cleared), set CONFIG_LOCK, verify Control + * writes are then discarded. + */ +TEST_F(cxl_type2, dvsec_lock_semantics) +{ + uint16_t dvsec = self->dvsec_base; + uint16_t lock_val, ctrl_before, ctrl_after; + + if (!dvsec) + SKIP(return, "CXL DVSEC not found in config space"); + + lock_val = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_LOCK_OFFSET); + + if (lock_val & CXL_DVSEC_LOCK_CONFIG_LOCK) { + /* + * Lock is already set: verify reserved bits are zero in the + * current shadow, then verify a Control write is discarded. + */ + ASSERT_EQ(0u, lock_val & CXL_LOCK_RESERVED_MASK); + + ctrl_before = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_CONTROL_OFFSET); + /* Attempt to flip CXL_Mem_Enable (bit 2) */ + vfio_pci_config_writew(self->dev, dvsec + CXL_DVSEC_CONTROL_OFFSET, + ctrl_before ^ BIT(2)); + ctrl_after = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_CONTROL_OFFSET); + ASSERT_EQ(ctrl_before, ctrl_after); + + printf("CONFIG_LOCK set: lock=0x%04x, " + "Control write discarded (ctrl=0x%04x unchanged)\n", + lock_val, ctrl_after); + return; + } + + /* Lock is not set: exercise reserved-bit masking and lock-set sequence */ + vfio_pci_config_writew(self->dev, dvsec + CXL_DVSEC_LOCK_OFFSET, + CXL_LOCK_RESERVED_MASK); + lock_val = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_LOCK_OFFSET); + ASSERT_EQ(0u, lock_val & CXL_LOCK_RESERVED_MASK); + ASSERT_FALSE(lock_val & CXL_DVSEC_LOCK_CONFIG_LOCK); + + ctrl_before = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_CONTROL_OFFSET); + vfio_pci_config_writew(self->dev, dvsec + CXL_DVSEC_LOCK_OFFSET, + CXL_DVSEC_LOCK_CONFIG_LOCK); + lock_val = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_LOCK_OFFSET); + ASSERT_TRUE(lock_val & CXL_DVSEC_LOCK_CONFIG_LOCK); + + vfio_pci_config_writew(self->dev, dvsec + CXL_DVSEC_CONTROL_OFFSET, + ctrl_before ^ BIT(0)); + ctrl_after = vfio_pci_config_readw(self->dev, + dvsec + CXL_DVSEC_CONTROL_OFFSET); + ASSERT_EQ(ctrl_before, ctrl_after); + + printf("Lock set, Control write discarded: " + "lock=0x%04x ctrl_before=0x%04x ctrl_after=0x%04x\n", + lock_val, ctrl_before, ctrl_after); +} + +/* ------------------------------------------------------------------ */ +/* main */ +/* ------------------------------------------------------------------ */ + +int main(int argc, char *argv[]) +{ + device_bdf = vfio_selftests_get_bdf(&argc, argv); + return test_harness_run(argc, argv); +} From cffd8ff5fc16916e889cf6ddc770ee6a5b6c5940 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Thu, 30 Apr 2026 08:44:02 +0530 Subject: [PATCH 407/464] NVIDIA: VR: SAUCE: cxl: Export the CXL reset helpers for VFIO users BugLink: https://bugs.launchpad.net/bugs/2152222 Export CXL reset helper entry points for VFIO CXL users so vfio-pci can coordinate CXL reset and memory/cache state safely. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from commit 2d40efbb4f42 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [kobak: Kept the BOS CXL core tail and placed the exported reset helpers after cxl_port_get_possible_dports().] [kobak: Adapted to the BOS cxl_pci_functions_reset_prepare() error-return flow and added the target-local CXL reset helper prototypes required by public and private CXL headers.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/cxl/core/pci.c | 57 ++++++++++++++++++++++++++++++++--- drivers/cxl/cxl.h | 5 +++ include/cxl/pci.h | 3 ++ include/linux/pci.h | 1 - include/uapi/linux/pci_regs.h | 2 +- 5 files changed, 62 insertions(+), 6 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 3b6c878f822c0..15fc16b29a03e 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -1261,7 +1261,7 @@ static void cxl_pci_functions_reset_done(struct cxl_reset_context *ctx) /* * CXL device reset execution */ -static int cxl_dev_reset(struct pci_dev *pdev, int dvsec) +int cxl_dev_reset(struct pci_dev *pdev, int dvsec, bool mem_clr_en) { static const u32 reset_timeout_ms[] = { 10, 100, 1000, 10000, 100000 }; u16 cap, ctrl2, status2; @@ -1331,7 +1331,17 @@ static int cxl_dev_reset(struct pci_dev *pdev, int dvsec) if (rc) return rc; - ctrl2 |= PCI_DVSEC_CXL_RST_MEM_CLR_EN; + /* + * Explicitly set or clear RST_MEM_CLR_EN rather than only + * setting it. A previous reset may have left the bit set in + * hardware; if mem_clr_en is false we must clear it so that a + * guest-triggered reset does not unexpectedly scrub DPA. + */ + if (mem_clr_en) + ctrl2 |= PCI_DVSEC_CXL_RST_MEM_CLR_EN; + else + ctrl2 &= ~PCI_DVSEC_CXL_RST_MEM_CLR_EN; + rc = pci_write_config_word(pdev, dvsec + PCI_DVSEC_CXL_CTRL2, ctrl2); if (rc) @@ -1380,6 +1390,44 @@ static int cxl_dev_reset(struct pci_dev *pdev, int dvsec) return 0; } +EXPORT_SYMBOL_NS_GPL(cxl_dev_reset, "CXL"); + +/** + * cxl_dev_reset_locked() - cxl_dev_reset() under cxl_reset_mutex with sibling + * CXL.cachemem function save/restore. + * @pdev: Target CXL function + * @dvsec: CXL DVSEC capability offset (pci_find_dvsec_capability()) + * @mem_clr_en: Pass-through to cxl_dev_reset() (Mem_Clr_Enable in CTRL2) + * + * Return: 0 on success, negative errno from cxl_dev_reset() or sibling + * coordination failure. + */ +int cxl_dev_reset_locked(struct pci_dev *pdev, int dvsec, bool mem_clr_en) +{ + struct cxl_reset_context ctx = { .target = pdev }; + bool siblings_prepared = false; + int rc; + + mutex_lock(&cxl_reset_mutex); + pci_dev_lock(pdev); + + pci_dev_save_and_disable(pdev); + rc = cxl_pci_functions_reset_prepare(&ctx); + if (!rc) { + siblings_prepared = true; + rc = cxl_dev_reset(pdev, dvsec, mem_clr_en); + } + + if (siblings_prepared) + cxl_pci_functions_reset_done(&ctx); + + pci_dev_restore(pdev); + pci_dev_unlock(pdev); + mutex_unlock(&cxl_reset_mutex); + + return rc; +} +EXPORT_SYMBOL_NS_GPL(cxl_dev_reset_locked, "CXL"); static int match_memdev_by_parent(struct device *dev, const void *parent) { @@ -1411,7 +1459,7 @@ static int __cxl_do_reset(struct pci_dev *pdev, struct cxl_memdev *cxlmd, rc = cxl_pci_functions_reset_prepare(&ctx); if (!rc) { siblings_prepared = true; - rc = cxl_dev_reset(pdev, dvsec); + rc = cxl_dev_reset(pdev, dvsec, true); } if (siblings_prepared) @@ -1459,7 +1507,7 @@ static int cxl_do_reset(struct pci_dev *pdev) * devices under bus core serialization. */ -static bool pci_cxl_reset_capable(struct pci_dev *pdev) +bool pci_cxl_reset_capable(struct pci_dev *pdev) { int dvsec; u16 cap; @@ -1478,6 +1526,7 @@ static bool pci_cxl_reset_capable(struct pci_dev *pdev) return !!(cap & PCI_DVSEC_CXL_RST_CAPABLE); } +EXPORT_SYMBOL_NS_GPL(pci_cxl_reset_capable, "CXL"); static ssize_t cxl_reset_store(struct device *dev, struct device_attribute *attr, diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index c52676d604976..caee223c0e611 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -163,6 +163,11 @@ int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map, unsigned int index); int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index, resource_size_t *bar_offset); +int cxl_dev_reset(struct pci_dev *pdev, int dvsec, bool mem_clr_en); +int cxl_dev_reset_locked(struct pci_dev *pdev, int dvsec, bool mem_clr_en); +bool pci_cxl_reset_capable(struct pci_dev *pdev); +void cxl_reset_sysfs_init(void); +void cxl_reset_sysfs_exit(void); struct cxl_dport; int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev, struct cxl_dport *dport); diff --git a/include/cxl/pci.h b/include/cxl/pci.h index 38bd7c73575a8..83d43e00c9245 100644 --- a/include/cxl/pci.h +++ b/include/cxl/pci.h @@ -28,5 +28,8 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base, int cxl_await_range_active(struct cxl_dev_state *cxlds); int cxl_regblock_get_bar_info(const struct cxl_register_map *map, u8 *bar_index, resource_size_t *bar_offset); +int cxl_dev_reset(struct pci_dev *pdev, int dvsec, bool mem_clr_en); +int cxl_dev_reset_locked(struct pci_dev *pdev, int dvsec, bool mem_clr_en); +bool pci_cxl_reset_capable(struct pci_dev *pdev); int cxl_setup_regs(struct cxl_register_map *map); #endif diff --git a/include/linux/pci.h b/include/linux/pci.h index 6137e0b4b964a..5f226bdac994d 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2001,7 +2001,6 @@ void pci_dev_lock(struct pci_dev *dev); int pci_dev_trylock(struct pci_dev *dev); void pci_dev_unlock(struct pci_dev *dev); DEFINE_GUARD(pci_dev, struct pci_dev *, pci_dev_lock(_T), pci_dev_unlock(_T)) - void pci_dev_save_and_disable(struct pci_dev *dev); void pci_dev_restore(struct pci_dev *dev); diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index c4a6c0a916dc7..806a6237ec1c0 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -1360,7 +1360,7 @@ #define PCI_DVSEC_CXL_MEM_ENABLE _BITUL(2) #define PCI_DVSEC_CXL_CTRL_RWL 0x5FED #define PCI_DVSEC_CXL_CTRL2 0x10 -#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) +#define PCI_DVSEC_CXL_DISABLE_CACHING _BITUL(0) #define PCI_DVSEC_CXL_INIT_CACHE_WBI _BITUL(1) #define PCI_DVSEC_CXL_INIT_CXL_RST _BITUL(2) #define PCI_DVSEC_CXL_RST_MEM_CLR_EN _BITUL(3) From 6b011bbdbbd2dac95f7af0b80530fc628c2016ef Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Wed, 20 May 2026 12:57:07 -0500 Subject: [PATCH 408/464] NVIDIA: VR: SAUCE: vfio/pci: Wire CXL DPA reset handling BugLink: https://bugs.launchpad.net/bugs/2152222 Wire the VFIO CXL reset prepare/finish paths into VFIO PCI reset flows so DPA mappings are zapped before reset and restored after successful reset. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from commit 0bd9c4c7ab7 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [kobak: Preserved existing VFIO PCI DMABUF reset movement while adding CXL reset prepare/finish handling.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 83 +++++++++++++++++++++++----- drivers/vfio/pci/cxl/vfio_cxl_emu.c | 2 +- drivers/vfio/pci/vfio_pci_config.c | 4 ++ drivers/vfio/pci/vfio_pci_core.c | 36 ++++++++++-- drivers/vfio/pci/vfio_pci_priv.h | 15 +++-- 5 files changed, 116 insertions(+), 24 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 622482064890b..e6bcf486d4a6b 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -660,6 +660,18 @@ static vm_fault_t vfio_cxl_region_vm_fault(struct vm_fault *vmf) */ down_read(&vdev->memory_lock); + /* + * Mirror vfio_pci_vmf_insert_pfn(): reject faults while runtime PM is + * engaged or PCI Memory Space / power state would make the underlying + * memory inaccessible. vfio_pci_zap_and_down_write_memory_lock() has + * already unmapped existing PTEs in those paths; this gate stops the + * fault path from faulting them back in. + */ + if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) { + ret = VM_FAULT_SIGBUS; + goto out; + } + if (!cxl->region_active) { ret = VM_FAULT_SIGBUS; goto out; @@ -724,15 +736,27 @@ static int vfio_cxl_region_mmap(struct vfio_pci_core_device *vdev, return 0; } +bool vfio_cxl_reset_capable(struct vfio_pci_core_device *vdev) +{ + return vdev->cxl && pci_cxl_reset_capable(vdev->pdev); +} + /* - * vfio_cxl_zap_region_locked - Invalidate all DPA region PTEs. + * vfio_cxl_zap_dpa - Invalidate DPA region PTEs without touching region_active. * - * Must be called with vdev->memory_lock held for writing. Sets - * region_active=false before zapping so any subsequent I/O to the region - * sees the inactive state and returns an error rather than accessing - * stale mappings. + * Used by paths that revoke user access transiently (runtime PM entry, D3 + * power transitions, Memory Space disable) but do not perform a reset. + * The DPA region offset range is separate from the BAR range zapped by + * vfio_pci_zap_bars(), so existing DPA mmaps and fd I/O would otherwise + * continue to touch CXL.mem while the device is suspended. + * + * The fault handler and fd I/O path additionally check pm_runtime_engaged + * and __vfio_pci_memory_enabled() to refuse re-faulting while the device + * is in the revoked state. + * + * Must be called with vdev->memory_lock held for writing. */ -void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) +void vfio_cxl_zap_dpa(struct vfio_pci_core_device *vdev) { struct vfio_device *core_vdev = &vdev->vdev; struct vfio_pci_cxl_state *cxl = vdev->cxl; @@ -742,7 +766,6 @@ void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) if (!cxl || cxl->dpa_region_idx < 0) return; - WRITE_ONCE(cxl->region_active, false); unmap_mapping_range(core_vdev->inode->i_mapping, VFIO_PCI_INDEX_TO_OFFSET(VFIO_PCI_NUM_REGIONS + cxl->dpa_region_idx), @@ -750,13 +773,34 @@ void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) } /* - * vfio_cxl_reactivate_region - Re-enable DPA region after successful reset. + * vfio_cxl_prepare_reset - Invalidate all DPA region PTEs. + * + * Must be called with vdev->memory_lock held for writing. Sets + * region_active=false before zapping so any subsequent I/O to the region + * sees the inactive state and returns an error rather than accessing + * stale mappings. + */ +void vfio_cxl_prepare_reset(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + + lockdep_assert_held_write(&vdev->memory_lock); + + if (!cxl || cxl->dpa_region_idx < 0) + return; + + WRITE_ONCE(cxl->region_active, false); + vfio_cxl_zap_dpa(vdev); +} + +/* + * vfio_cxl_finish_reset - Re-enable DPA region after reset. * * Must be called with vdev->memory_lock held for writing. Re-reads the - * HDM decoder state from hardware (FLR cleared it) and sets region_active - * so that subsequent I/O to the region is permitted again. + * HDM decoder state from hardware and sets region_active so that + * subsequent I/O to the region is permitted again. */ -void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) +void vfio_cxl_finish_reset(struct vfio_pci_core_device *vdev) { struct vfio_pci_cxl_state *cxl = vdev->cxl; @@ -766,8 +810,8 @@ void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) return; /* * Re-initialise the emulated HDM comp_reg_virt[] from hardware. - * After FLR the decoder registers read as zero; mirror that in - * the emulated state so QEMU sees a clean slate. + * A reset clears decoder registers; mirror that in the emulated + * state so the guest device manager sees the post-reset hardware. */ vfio_cxl_reinit_comp_regs(cxl); @@ -818,6 +862,17 @@ static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev, */ down_read(&core_dev->memory_lock); + /* + * Mirror the BAR-rw / fault gates: refuse fd I/O while the device is + * runtime suspended or has Memory Space / power state that makes the + * memremap'd window inaccessible. + */ + if (core_dev->pm_runtime_engaged || + !__vfio_pci_memory_enabled(core_dev)) { + ret = -EIO; + goto out; + } + if (!cxl->region_active || !cxl->region_vaddr) { ret = -EIO; goto out; @@ -933,7 +988,7 @@ int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) * Cache the vdev->region[] index before activating the region. * vfio_pci_core_register_dev_region() placed the new entry at * vdev->region[num_regions - 1] and incremented num_regions. - * vfio_cxl_zap_region_locked() uses this to avoid scanning + * vfio_cxl_prepare_reset() uses this to avoid scanning * vdev->region[] on every FLR. */ cxl->dpa_region_idx = vdev->num_regions - 1; diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c index 8f1eefec7c44b..43d79e2fe5477 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_emu.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -437,7 +437,7 @@ vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev, } /* - * Called with memory_lock write side held (from vfio_cxl_reactivate_region). + * Called with memory_lock write side held (from vfio_cxl_finish_reset). * Uses the pre-established hdm_iobase, no ioremap() under the lock, * which would deadlock on PREEMPT_RT where ioremap() can sleep. */ diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index bf4730601949e..75fe0fc19b435 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -910,7 +910,9 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos, if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) { vfio_pci_zap_and_down_write_memory_lock(vdev); vfio_pci_dma_buf_move(vdev, true); + vfio_cxl_prepare_reset(vdev); pci_try_reset_function(vdev->pdev); + vfio_cxl_finish_reset(vdev); if (__vfio_pci_memory_enabled(vdev)) vfio_pci_dma_buf_move(vdev, false); up_write(&vdev->memory_lock); @@ -995,7 +997,9 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos, if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) { vfio_pci_zap_and_down_write_memory_lock(vdev); vfio_pci_dma_buf_move(vdev, true); + vfio_cxl_prepare_reset(vdev); pci_try_reset_function(vdev->pdev); + vfio_cxl_finish_reset(vdev); if (__vfio_pci_memory_enabled(vdev)) vfio_pci_dma_buf_move(vdev, false); up_write(&vdev->memory_lock); diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index d91e2d10a77a5..c58971d286054 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1247,7 +1247,7 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, vfio_pci_zap_and_down_write_memory_lock(vdev); /* Zap CXL DPA region PTEs before hardware reset clears HDM state */ - vfio_cxl_zap_region_locked(vdev); + vfio_cxl_prepare_reset(vdev); /* * This function can be invoked while the power state is non-D0. If @@ -1266,11 +1266,10 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, vfio_pci_dma_buf_move(vdev, false); /* - * Re-enable DPA region if reset succeeded; fault handler will - * re-insert PFNs on next access without requiring a new mmap. + * finish_reset checks the COMMITTED bit from hardware + * and only brings the region back if it is actually set. */ - if (!ret) - vfio_cxl_reactivate_region(vdev); + vfio_cxl_finish_reset(vdev); up_write(&vdev->memory_lock); @@ -1653,6 +1652,13 @@ void vfio_pci_zap_and_down_write_memory_lock(struct vfio_pci_core_device *vdev) { down_write(&vdev->memory_lock); vfio_pci_zap_bars(vdev); + /* + * Zap the CXL DPA region PTEs too: zap_bars only covers the BAR offset + * range, while the DPA region lives in the device-region offset range + * and would otherwise survive a runtime-PM entry or D3 transition. + * No-op on non-CXL devices. + */ + vfio_cxl_zap_dpa(vdev); } u16 vfio_pci_memory_lock_and_enable(struct vfio_pci_core_device *vdev) @@ -2529,6 +2535,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, goto err_undo; } + /* + * All devices in the set are now locked. Commit the CXL prepare + * step in its own pass: it clears region_active and zaps DPA PTEs, + * which must be paired with a finish_reset call for every device it + * touches. Doing this only after all trylocks have succeeded keeps + * a mid-loop failure from leaving earlier devices with + * region_active=false and no matching reset. + */ + list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) + vfio_cxl_prepare_reset(vdev); + /* * The pci_reset_bus() will reset all the devices in the bus. * The power state can be non-D0 for some of the devices in the bus. @@ -2543,6 +2560,15 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, ret = pci_reset_bus(pdev); + /* + * Mirror vfio_pci_ioctl_reset(): re-read the post-reset HDM state and + * reactivate the DPA region for CXL devices that hardware committed. + * Runs under each device's memory_lock write side acquired earlier and + * pairs with the prepare_reset pass above. + */ + list_for_each_entry(vdev, &dev_set->device_list, vdev.dev_set_list) + vfio_cxl_finish_reset(vdev); + vdev = list_last_entry(&dev_set->device_list, struct vfio_pci_core_device, vdev.dev_set_list); diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_priv.h index e925f66923593..f3d7f66552393 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -149,8 +149,10 @@ static inline void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev); void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev); -void vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev); -void vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev); +bool vfio_cxl_reset_capable(struct vfio_pci_core_device *vdev); +void vfio_cxl_prepare_reset(struct vfio_pci_core_device *vdev); +void vfio_cxl_finish_reset(struct vfio_pci_core_device *vdev); +void vfio_cxl_zap_dpa(struct vfio_pci_core_device *vdev); void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev); int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev); void vfio_cxl_unregister_cxl_region(struct vfio_pci_core_device *vdev); @@ -170,10 +172,15 @@ static inline void vfio_pci_cxl_detect_and_init(struct vfio_pci_core_device *vdev) { } static inline void vfio_pci_cxl_cleanup(struct vfio_pci_core_device *vdev) { } +static inline bool +vfio_cxl_reset_capable(struct vfio_pci_core_device *vdev) +{ return false; } +static inline void +vfio_cxl_prepare_reset(struct vfio_pci_core_device *vdev) { } static inline void -vfio_cxl_zap_region_locked(struct vfio_pci_core_device *vdev) { } +vfio_cxl_finish_reset(struct vfio_pci_core_device *vdev) { } static inline void -vfio_cxl_reactivate_region(struct vfio_pci_core_device *vdev) { } +vfio_cxl_zap_dpa(struct vfio_pci_core_device *vdev) { } static inline void vfio_cxl_setup_dvsec_perms(struct vfio_pci_core_device *vdev) { } static inline int From 78c73fd7c88503df20bb8c7f8cd64b249e94fa34 Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Wed, 20 May 2026 13:37:57 -0500 Subject: [PATCH 409/464] NVIDIA: VR: SAUCE: vfio/cxl: Ensure PCI Memory Space is enabled before post-reset BAR access BugLink: https://bugs.launchpad.net/bugs/2152222 A reset caller may disable Memory Space to quiesce device DMA before issuing the reset. The reset path saves and restores PCI_COMMAND via pci_dev_save_and_disable() and pci_dev_restore(). If Memory Space was disabled before FLR, it will be restored in the disabled state. vfio_cxl_finish_reset() reads HDM decoder registers through the component register BAR immediately after reset. Accessing a BAR with Memory Space disabled produces an Unsupported Request completion; on platforms that promote UR to a fatal error this triggers DPC. Add vfio_cxl_enable_memory_space() and call it at the start of vfio_cxl_finish_reset() before touching any BAR. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from commit 5071d3b07627ab1fc42f0149352a2704907f8364 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index e6bcf486d4a6b..154e44b5c2558 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -793,6 +793,27 @@ void vfio_cxl_prepare_reset(struct vfio_pci_core_device *vdev) vfio_cxl_zap_dpa(vdev); } +/* + * vfio_cxl_enable_memory_space - ensure PCI Memory Space is on before BAR reads. + * + * A reset caller may disable Memory Space to quiesce device DMA before + * issuing the reset. If a guest request cleared PCI_COMMAND Memory Space + * before FLR, pci_dev_save_and_disable() captures it disabled and + * pci_dev_restore() restores it that way. This can leave Memory Space + * disabled on return. Accessing a BAR with Memory Space disabled produces + * an Unsupported Request completion; on platforms that promote UR to a + * fatal error this fires DPC. + */ +static void vfio_cxl_enable_memory_space(struct vfio_pci_core_device *vdev) +{ + u16 cmd; + + pci_read_config_word(vdev->pdev, PCI_COMMAND, &cmd); + if (!(cmd & PCI_COMMAND_MEMORY)) + pci_write_config_word(vdev->pdev, PCI_COMMAND, + cmd | PCI_COMMAND_MEMORY); +} + /* * vfio_cxl_finish_reset - Re-enable DPA region after reset. * @@ -808,6 +829,9 @@ void vfio_cxl_finish_reset(struct vfio_pci_core_device *vdev) if (!cxl) return; + + vfio_cxl_enable_memory_space(vdev); + /* * Re-initialise the emulated HDM comp_reg_virt[] from hardware. * A reset clears decoder registers; mirror that in the emulated From 320198909baa2b65e0cd8a8fe088d2ba8099ab16 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Wed, 29 Apr 2026 22:50:38 +0530 Subject: [PATCH 410/464] NVIDIA: VR: SAUCE: vfio/cxl: preserve HDM decoder base addresses across reset BugLink: https://bugs.launchpad.net/bugs/2152222 After FLR, reinit_comp_regs() re-reads HDM decoder registers from hardware into comp_reg_virt[]. Hardware is not all-zeros at this point: pci_dev_restore() ran first and re-committed the pre-reset host-physical decoder bases into the registers. reinit_comp_regs() therefore overwrites the emulated guest-physical bases that the device manager programmed with the host-physical bases used by the host CXL core. The kernel provides no notification that BASE was overwritten, so the emulated GPA bases are silently lost. The same issue affects the CTRL LOCK bit: FLR clears it in hardware and pci_dev_restore() does not re-apply it, so a decoder that the guest had locked re-emerges from reset with LOCK clear in shadow. Add vfio_cxl_reinit_hdm_shadow() which snapshots BASE_LOW, BASE_HIGH, and the CTRL LOCK bit from the shadow before calling reinit_comp_regs(), then writes them back after, keeping the emulated decoder consistent with what the guest programmed. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (cherry-picked from commit 9e0e291bfc29f40e5db40256353b89d34fd93ddf from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 88 ++++++++++++++++++++++++++-- drivers/vfio/pci/cxl/vfio_cxl_emu.c | 38 +++++++++--- 2 files changed, 113 insertions(+), 13 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 154e44b5c2558..2d6b804d8537b 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -814,6 +814,87 @@ static void vfio_cxl_enable_memory_space(struct vfio_pci_core_device *vdev) cmd | PCI_COMMAND_MEMORY); } +/* + * vfio_cxl_reinit_hdm_shadow - reinitialise comp_reg_virt, preserving the + * guest-visible BASE registers and CTRL LOCK across reset. + * + * reinit_comp_regs() re-reads hardware into comp_reg_virt[] after FLR. + * pci_dev_restore() re-commits the host-physical BASE values it saved + * before the reset, so reinit_comp_regs() sees those host bases and not + * the guest-physical bases the device manager programmed in shadow. The + * decoder CTRL LOCK bit is also cleared by FLR on hardware and is not + * re-applied by pci_dev_restore(). Snapshot BASE_LOW/BASE_HIGH and the + * LOCK bit from shadow before reinit, then write them back so the + * emulated decoder stays consistent with what the guest configured. + * + * Called with memory_lock write side held (from vfio_cxl_finish_reset). + */ +static void vfio_cxl_reinit_hdm_shadow(struct vfio_pci_cxl_state *cxl) +{ + __le32 *saved_lo = NULL, *saved_hi = NULL, *saved_ctrl = NULL; + u8 n, count = cxl->hdm_count; + + if (cxl->comp_reg_virt && count) { + saved_lo = kcalloc(count, sizeof(*saved_lo), GFP_KERNEL); + saved_hi = kcalloc(count, sizeof(*saved_hi), GFP_KERNEL); + saved_ctrl = kcalloc(count, sizeof(*saved_ctrl), GFP_KERNEL); + if (!saved_lo || !saved_hi || !saved_ctrl) { + /* + * Allocation failure: skip the snapshot and let reinit + * resync from hardware. The guest-visible BASE/LOCK + * state will diverge but the device is otherwise + * functional. This path is unlikely under normal load. + */ + pci_warn(cxl->vdev->pdev, + "vfio_cxl: HDM shadow snapshot allocation failed; resetting without GPA preservation\n"); + kfree(saved_lo); + kfree(saved_hi); + kfree(saved_ctrl); + saved_lo = saved_hi = saved_ctrl = NULL; + } else { + for (n = 0; n < count; n++) { + saved_lo[n] = *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_BASE_LOW_OFFSET(n)); + saved_hi[n] = *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(n)); + saved_ctrl[n] = *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_CTRL_OFFSET(n)); + } + } + } + + vfio_cxl_reinit_comp_regs(cxl); + + if (cxl->comp_reg_virt && saved_lo) { + for (n = 0; n < count; n++) { + u32 ctrl; + + *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_BASE_LOW_OFFSET(n)) = saved_lo[n]; + *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_BASE_HIGH_OFFSET(n)) = saved_hi[n]; + + /* + * Restore the LOCK bit from shadow. Other CTRL bits + * (COMMITTED, error indicators) should reflect the + * post-FLR hardware state that reinit_comp_regs() just + * snapshotted, so leave those alone. + */ + ctrl = le32_to_cpu(*hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_CTRL_OFFSET(n))); + ctrl |= le32_to_cpu(saved_ctrl[n]) & + CXL_HDM_DECODER0_CTRL_LOCK; + *hdm_reg_ptr(cxl, + CXL_HDM_DECODER0_CTRL_OFFSET(n)) = + cpu_to_le32(ctrl); + } + } + + kfree(saved_lo); + kfree(saved_hi); + kfree(saved_ctrl); +} + /* * vfio_cxl_finish_reset - Re-enable DPA region after reset. * @@ -833,11 +914,10 @@ void vfio_cxl_finish_reset(struct vfio_pci_core_device *vdev) vfio_cxl_enable_memory_space(vdev); /* - * Re-initialise the emulated HDM comp_reg_virt[] from hardware. - * A reset clears decoder registers; mirror that in the emulated - * state so the guest device manager sees the post-reset hardware. + * Re-initialise the emulated HDM comp_reg_virt[] from hardware, + * preserving the GPA decoder bases set by the device manager. */ - vfio_cxl_reinit_comp_regs(cxl); + vfio_cxl_reinit_hdm_shadow(cxl); /* * Only re-enable the DPA mmap if the hardware has actually diff --git a/drivers/vfio/pci/cxl/vfio_cxl_emu.c b/drivers/vfio/pci/cxl/vfio_cxl_emu.c index 43d79e2fe5477..bdd363a819a7c 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_emu.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_emu.c @@ -272,6 +272,7 @@ static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev, { struct vfio_pci_cxl_state *cxl = vdev->cxl; loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + ssize_t ret = 0; size_t done = 0; if (!count) @@ -283,14 +284,26 @@ static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev, count = min(count, (size_t)(cxl->hdm_reg_offset + cxl->hdm_reg_size - pos)); + /* + * Serialise against vfio_cxl_reinit_hdm_shadow(), which holds + * memory_lock write-side while it saves, zeroes, and restores + * comp_reg_virt[] during reset. Without this read lock a concurrent + * COMP_REGS write can land between the save snapshot and the restore, + * causing the restore to silently overwrite it. A concurrent read + * can observe the array mid-rebuild. + */ + down_read(&vdev->memory_lock); + while (done < count) { u32 sz = count - done; u32 off = pos + done; __le32 v; /* Enforce exactly 4-byte, 4-byte-aligned accesses */ - if (sz != CXL_REG_SIZE_DWORD || (off & 0x3)) - return done ? (ssize_t)done : -EINVAL; + if (sz != CXL_REG_SIZE_DWORD || (off & 0x3)) { + ret = done ? (ssize_t)done : -EINVAL; + goto out_unlock; + } if (iswrite) { if (off < cxl->hdm_reg_offset) { @@ -298,22 +311,29 @@ static ssize_t vfio_cxl_comp_regs_rw(struct vfio_pci_core_device *vdev, done += sizeof(v); continue; } - if (copy_from_user(&v, buf + done, sizeof(v))) - return done ? (ssize_t)done : -EFAULT; + if (copy_from_user(&v, buf + done, sizeof(v))) { + ret = done ? (ssize_t)done : -EFAULT; + goto out_unlock; + } comp_regs_dispatch_write(vdev, off - cxl->hdm_reg_offset, &v, sizeof(v)); } else { - /* Read from extended buffer _ covers cap array and HDM */ + /* Read from extended buffer - covers cap array and HDM */ v = cxl->comp_reg_virt[off / sizeof(__le32)]; - if (copy_to_user(buf + done, &v, sizeof(v))) - return done ? (ssize_t)done : -EFAULT; + if (copy_to_user(buf + done, &v, sizeof(v))) { + ret = done ? (ssize_t)done : -EFAULT; + goto out_unlock; + } } done += sizeof(v); } + ret = done; *ppos += done; - return done; +out_unlock: + up_read(&vdev->memory_lock); + return ret; } static void vfio_cxl_comp_regs_release(struct vfio_pci_core_device *vdev, @@ -437,7 +457,7 @@ vfio_cxl_read_committed_decoder_size(struct vfio_pci_core_device *vdev, } /* - * Called with memory_lock write side held (from vfio_cxl_finish_reset). + * Called with memory_lock write side held (from vfio_cxl_reinit_hdm_shadow). * Uses the pre-established hdm_iobase, no ioremap() under the lock, * which would deadlock on PREEMPT_RT where ioremap() can sleep. */ From 75f386be6ca8e6dd8e74571b552f54581ec66f70 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Thu, 30 Apr 2026 00:28:10 +0530 Subject: [PATCH 411/464] NVIDIA: VR: SAUCE: vfio/cxl: virtualize DVSEC STATUS2 register in vconfig shadow BugLink: https://bugs.launchpad.net/bugs/2152222 STATUS2 was read directly from hardware while all other DVSEC registers were served from the vconfig shadow. This created two problems: 1. VOLATILE_HDM_PRES_ERROR (RW1CS, bit 3): guest writes cleared the hardware bit but the shadow was not updated, so subsequent reads still returned the set bit from hardware (which the hardware had cleared). 2. CXL_RESET_COMPLETE and CXL_RESET_ERROR (bits 1-2): these outcome bits will be written by vfio_cxl_reset() into the shadow after a protocol reset. Hardware does not update them on its own; serving reads from hardware would hide the outcome from the guest. Add STATUS2 to the read switch so reads come from the shadow, and update cxl_dvsec_status2_write() to mirror VOLATILE_HDM_PRES_ERROR clears into the shadow after forwarding to hardware. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (backported from commit 14fbdcb4d592891c269d052317cc10b640aa4096 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_config.c | 191 +++++++++++++++++++++---- 1 file changed, 167 insertions(+), 24 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_config.c b/drivers/vfio/pci/cxl/vfio_cxl_config.c index 35d35e2ded9bf..4dc86c1de80be 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_config.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_config.c @@ -158,12 +158,34 @@ static void cxl_dvsec_status2_write(struct vfio_pci_core_device *vdev, u16 dvsec = _cxlds_get_dvsec(vdev->cxl); u16 abs_off = dvsec + CXL_DVSEC_STATUS2_OFFSET; - /* RW1CS: write 1 to clear, but only if the capability is supported */ + /* + * VOLATILE_HDM_PRES_ERROR (bit 3) and PM_INIT_COMPLETION (bit 15) are + * RW1CS. Forward each to hardware on a 1-bit write, then mirror the + * clear into the shadow so guest reads (which now come from the + * shadow) do not see the bit stuck after a successful clear. + * + * All other STATUS2 bits are RO hardware outputs; ignore guest writes. + */ if ((cap3 & CXL_DVSEC_CAP3_VOLATILE_HDM_CONFIGURABILITY) && - (new_val & CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR)) + (new_val & CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR)) { + u16 v; + pci_write_config_word(vdev->pdev, abs_off, CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR); - /* STATUS2 is not mirrored in vconfig - reads go to hardware */ + v = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS2_OFFSET); + v &= ~CXL_DVSEC_STATUS2_VOLATILE_HDM_PRES_ERROR; + dvsec_virt_write16(vdev, CXL_DVSEC_STATUS2_OFFSET, v); + } + + if (new_val & CXL_DVSEC_STATUS2_PM_INIT_COMPLETION) { + u16 v; + + pci_write_config_word(vdev->pdev, abs_off, + CXL_DVSEC_STATUS2_PM_INIT_COMPLETION); + v = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS2_OFFSET); + v &= ~CXL_DVSEC_STATUS2_PM_INIT_COMPLETION; + dvsec_virt_write16(vdev, CXL_DVSEC_STATUS2_OFFSET, v); + } } static void cxl_dvsec_lock_write(struct vfio_pci_core_device *vdev, @@ -186,6 +208,30 @@ static void cxl_range_base_lo_write(struct vfio_pci_core_device *vdev, dvsec_virt_write32(vdev, dvsec_off, new_val); } +/* + * status2_hw_shadow_merge - read STATUS2, merging hardware and vconfig shadow. + * + * RESET_COMPLETE and RESET_ERROR are written into vconfig by vfio_cxl_reset() + * after a protocol reset; pci_dev_restore() clears them from hardware, so they + * must survive in the shadow for a polling guest to see the reset outcome. + * + * All other STATUS2 bits are live hardware outputs and must come from hardware. + * In particular, CACHE_INVALID (bit 0) is polled by guests during a standalone + * write-back invalidation. + * + * @abs_pos: absolute PCI config space byte offset of the STATUS2 register. + */ +static u16 status2_hw_shadow_merge(struct vfio_pci_core_device *vdev, int abs_pos) +{ + const u16 shadow_mask = CXL_DVSEC_STATUS2_CXL_RESET_COMPLETE | + CXL_DVSEC_STATUS2_CXL_RESET_ERROR; + u16 hw = 0, virt; + + pci_read_config_word(vdev->pdev, abs_pos, &hw); + virt = get_unaligned_le16(vdev->vconfig + abs_pos); + return (hw & ~shadow_mask) | (virt & shadow_mask); +} + /** * vfio_cxl_dvsec_readfn - Per-device DVSEC read handler for CXL capable devices. * @vdev: VFIO PCI core device @@ -200,6 +246,10 @@ static void cxl_range_base_lo_write(struct vfio_pci_core_device *vdev, * LOCK) so that userspace reads reflect emulated state rather than raw * hardware. All other DVSEC bytes pass through to vfio_raw_config_read(). * + * A 4-byte (DWORD) access at the CONTROL2 offset spans both CONTROL2 and + * STATUS2 since CONTROL2 is DWORD-aligned and the two registers are adjacent. + * In that case STATUS2 is returned via the hardware-merge path. + * * Return: @count on success, or negative error code from the fallback read. */ static int vfio_cxl_dvsec_readfn(struct vfio_pci_core_device *vdev, @@ -208,26 +258,61 @@ static int vfio_cxl_dvsec_readfn(struct vfio_pci_core_device *vdev, int offset, __le32 *val) { struct vfio_pci_cxl_state *cxl = vdev->cxl; - u16 dvsec = _cxlds_get_dvsec(vdev->cxl); - u16 dvsec_off; + u16 dvsec, dvsec_off, reg_start, byte_in_reg; + + if (!cxl) + return vfio_direct_config_read(vdev, pos, count, perm, offset, + val); - if (!cxl || (u16)pos < dvsec || - (u16)pos >= dvsec + cxl->dvsec_len) - return vfio_raw_config_read(vdev, pos, count, perm, offset, val); + dvsec = _cxlds_get_dvsec(cxl); + if ((u16)pos < dvsec || (u16)pos >= dvsec + cxl->dvsec_len) + return vfio_direct_config_read(vdev, pos, count, perm, offset, + val); dvsec_off = (u16)pos - dvsec; - switch (dvsec_off) { + /* + * Route by the 2-byte-aligned start of the register so that a guest + * read at the high byte (dvsec_off | 1) hits the shadow path instead + * of falling through to the direct read and diverging from a prior + * shadow write. + */ + reg_start = dvsec_off & ~1u; + byte_in_reg = dvsec_off - reg_start; + + switch (reg_start) { case CXL_DVSEC_CONTROL_OFFSET: case CXL_DVSEC_STATUS_OFFSET: - case CXL_DVSEC_CONTROL2_OFFSET: case CXL_DVSEC_LOCK_OFFSET: - /* Return shadow vconfig value for virtualized registers */ + /* Fully virtualised; return shadow. Byte/word reads work too. */ memcpy(val, vdev->vconfig + pos, count); return count; + case CXL_DVSEC_CONTROL2_OFFSET: + if (count == 4 && byte_in_reg == 0) { + /* + * 4-byte access at the DWORD-aligned CONTROL2 offset + * spans both CONTROL2 (low 16 bits) and STATUS2 (high + * 16 bits). Return CONTROL2 from vconfig and STATUS2 + * via the hardware-merge path so CACHE_INVALID is fresh. + */ + __le32 combined = cpu_to_le32( + (u32)get_unaligned_le16(vdev->vconfig + pos) | + ((u32)status2_hw_shadow_merge(vdev, + dvsec + CXL_DVSEC_STATUS2_OFFSET) << 16)); + memcpy(val, &combined, 4); + } else { + memcpy(val, vdev->vconfig + pos, count); + } + return count; + case CXL_DVSEC_STATUS2_OFFSET: { + __le16 merged = cpu_to_le16(status2_hw_shadow_merge(vdev, + dvsec + CXL_DVSEC_STATUS2_OFFSET)); + memcpy(val, ((u8 *)&merged) + byte_in_reg, count); + return count; + } default: - return vfio_raw_config_read(vdev, pos, count, - perm, offset, val); + return vfio_direct_config_read(vdev, pos, count, + perm, offset, val); } } @@ -255,14 +340,18 @@ static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev, int offset, __le32 val) { struct vfio_pci_cxl_state *cxl = vdev->cxl; - u16 dvsec = _cxlds_get_dvsec(vdev->cxl); - u16 abs_off = (u16)pos; - u16 dvsec_off, dword_start, byte_in_dword; + u16 dvsec, abs_off, dvsec_off, reg_start, byte_in_reg; + u16 dword_start, byte_in_dword; u16 wval16; u32 wval32; - if (!cxl || (u16)pos < dvsec || - (u16)pos >= dvsec + cxl->dvsec_len) + if (!cxl) + return vfio_raw_config_write(vdev, pos, count, perm, + offset, val); + + dvsec = _cxlds_get_dvsec(cxl); + abs_off = (u16)pos; + if (abs_off < dvsec || abs_off >= dvsec + cxl->dvsec_len) return vfio_raw_config_write(vdev, pos, count, perm, offset, val); @@ -272,6 +361,16 @@ static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev, dvsec_off = abs_off - dvsec; + /* + * The 2-byte virtualised registers (CONTROL, STATUS, CONTROL2, + * STATUS2, LOCK) all live at 2-byte-aligned offsets. Compute the + * register-aligned offset so writes at the high byte still hit the + * right handler, and merge partial-byte writes against the shadow so + * the high byte of the matched register is not zeroed. + */ + reg_start = dvsec_off & ~1u; + byte_in_reg = dvsec_off - reg_start; + dword_start = dvsec_off & ~3u; byte_in_dword = dvsec_off - dword_start; @@ -288,26 +387,70 @@ static int vfio_cxl_dvsec_writefn(struct vfio_pci_core_device *vdev, return count; } + if (count == 1) { + u16 cur = dvsec_virt_read16(vdev, reg_start); + u8 byte = (u8)le32_to_cpu(val); + + wval16 = byte_in_reg ? (cur & 0x00ff) | ((u16)byte << 8) + : (cur & 0xff00) | byte; + } else { + wval16 = (u16)le32_to_cpu(val); + } + /* Route to the appropriate per-register handler */ - switch (dvsec_off) { + switch (reg_start) { case CXL_DVSEC_CONTROL_OFFSET: - wval16 = (u16)le32_to_cpu(val); cxl_dvsec_control_write(vdev, wval16); + if (count == 4 && byte_in_reg == 0) { + /* + * High half of a 32-bit write at CONTROL is STATUS; + * forward so RW1C VIRAL_STATUS is not silently dropped. + */ + cxl_dvsec_status_write(vdev, + (u16)(le32_to_cpu(val) >> 16)); + } break; case CXL_DVSEC_STATUS_OFFSET: - wval16 = (u16)le32_to_cpu(val); + /* + * STATUS is RO/W1C. A one-byte write must only act on bits in + * the byte the guest wrote: re-derive the value without merging + * the other byte from shadow, otherwise W1C bits set in shadow + * (e.g. VIRAL_STATUS) would be passed as fresh 1-writes and + * unintentionally cleared. + */ + if (count == 1) { + u8 byte = (u8)le32_to_cpu(val); + + wval16 = byte_in_reg ? ((u16)byte << 8) : byte; + } cxl_dvsec_status_write(vdev, wval16); break; case CXL_DVSEC_CONTROL2_OFFSET: - wval16 = (u16)le32_to_cpu(val); cxl_dvsec_control2_write(vdev, wval16); + if (count == 4 && byte_in_reg == 0) { + /* + * High half of a 32-bit write at CONTROL2 is STATUS2; + * forward so RW1CS VOLATILE_HDM_PRES_ERROR is not + * silently dropped. + */ + cxl_dvsec_status2_write(vdev, + (u16)(le32_to_cpu(val) >> 16)); + } break; case CXL_DVSEC_STATUS2_OFFSET: - wval16 = (u16)le32_to_cpu(val); + /* + * STATUS2 is RO/RW1CS. Same rule as STATUS: a one-byte write + * must not let W1CS bits set in shadow leak in as fresh + * 1-writes via the merge. + */ + if (count == 1) { + u8 byte = (u8)le32_to_cpu(val); + + wval16 = byte_in_reg ? ((u16)byte << 8) : byte; + } cxl_dvsec_status2_write(vdev, wval16); break; case CXL_DVSEC_LOCK_OFFSET: - wval16 = (u16)le32_to_cpu(val); cxl_dvsec_lock_write(vdev, wval16); break; default: From b3ecf046a9dda88e4837c1f613bba5b2c4d42ef0 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Thu, 30 Apr 2026 11:56:01 +0530 Subject: [PATCH 412/464] NVIDIA: VR: SAUCE: vfio/cxl: Implement vfio_cxl_reset() BugLink: https://bugs.launchpad.net/bugs/2152222 Add vfio_cxl_reset() to drive a CXL protocol reset on behalf of a guest. Unlike cxl_do_reset(), this path skips host memory offlining since the DPA region is guest memory. The function takes memory_lock for the full sequence, calls vfio_cxl_prepare_reset() to zap DPA region PTEs, drives the hardware via cxl_dev_reset_locked(), which performs pci_dev_save_and_disable(), cxl_dev_reset(), sibling CXL.cachemem coordination, and pci_dev_restore() under the CXL reset mutex, then calls vfio_cxl_finish_reset() to reinitialise emulated state. STATUS2 outcome bits (CXL_RESET_COMPLETE / CXL_RESET_ERROR) are written back to vconfig after the reset so the guest can poll for the result without reading hardware. cxl_save_dvsec() / cxl_restore_dvsec() cover CTRL, CTRL2, range_base_*, and LOCK; STATUS2 is not saved or restored across the reset, so the hardware value is re-read after restore (it will have both outcome bits clear) and the outcome is stamped on top. When the guest writes INIT_CXL_RST into DVSEC CONTROL2, invoke vfio_cxl_reset() to perform a CXL protocol reset. The bit is not forwarded to hardware; cxl_dev_reset() drives the reset sequence directly. Silently drop writes on devices that do not advertise RST_CAPABLE to avoid log noise for the reserved-bit case. Signed-off-by: Manish Honap Signed-off-by: Jiandi An (cherry-picked from commit 67c66e735df5762cfeb7b80e4fdf0815451899f9 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_config.c | 106 +++++++++++++++++++++++-- 1 file changed, 101 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_config.c b/drivers/vfio/pci/cxl/vfio_cxl_config.c index 4dc86c1de80be..aeecce1a3d50c 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_config.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_config.c @@ -107,6 +107,85 @@ static void cxl_dvsec_status_write(struct vfio_pci_core_device *vdev, dvsec_virt_write16(vdev, CXL_DVSEC_STATUS_OFFSET, new_val); } +/** + * vfio_cxl_reset - Service a guest CXL protocol reset. + * @vdev: VFIO PCI core device + * + * Unlike cxl_do_reset(), no host memory offlining is performed: the DPA + * region is guest memory, not host RAM. + * + * memory_lock is held for the entire sequence so neither BAR nor DPA + * mappings can fault back in. INIT_CXL_RST is not forwarded to hardware; + * cxl_dev_reset() drives the state machine directly. + * + * STATUS2 outcome bits are written back to vconfig on return so that the + * guest can poll for completion without going to hardware. + * + * Return: 0 on success, negative error code on failure. + */ +static int vfio_cxl_reset(struct vfio_pci_core_device *vdev) +{ + struct vfio_pci_cxl_state *cxl = vdev->cxl; + struct pci_dev *pdev = vdev->pdev; + u16 dvsec = _cxlds_get_dvsec(cxl); + u16 hw_status2 = 0; + int ret; + + vfio_pci_zap_and_down_write_memory_lock(vdev); + + /* + * CXL r4.0 Table 8-9: device must clear CXL_Reset_Complete before + * starting the reset flow, on the 0->1 transition of Initiate_CXL_Reset. + * Clear both reset outcome bits so a polling guest sees an unambiguous + * in-progress state rather than a stale result from a prior attempt. + */ + { + u16 s = dvsec_virt_read16(vdev, CXL_DVSEC_STATUS2_OFFSET); + + s &= ~(CXL_DVSEC_STATUS2_CXL_RESET_COMPLETE | + CXL_DVSEC_STATUS2_CXL_RESET_ERROR); + dvsec_virt_write16(vdev, CXL_DVSEC_STATUS2_OFFSET, s); + } + + vfio_cxl_prepare_reset(vdev); + + /* + * Hand the actual reset off to cxl_dev_reset_locked() so the CXL core + * applies its global reset mutex and saves/disables any CXL.cachemem + * sibling functions on the bus. A bare cxl_dev_reset() under just + * pci_dev_lock() leaves those siblings vulnerable to half-reset states + * and lets a guest-triggered CXL reset race a concurrent host sysfs + * reset. + */ + ret = cxl_dev_reset_locked(pdev, cxl->cxlds.cxl_dvsec, + !!(dvsec_virt_read16(vdev, + CXL_DVSEC_CONTROL2_OFFSET) & + CXL_DVSEC_CTRL2_CXL_RESET_MEM_CLR_ENABLE)); + + vfio_cxl_finish_reset(vdev); + + /* + * Re-read STATUS2 from hardware after restore. cxl_save_dvsec() / + * cxl_restore_dvsec() cover CTRL, CTRL2, range_base_*, and LOCK; + * STATUS2 is intentionally not saved or restored across the reset, so + * the hardware value here is fresh post-reset (both outcome bits clear) + * and reflects genuine hardware changes such as VOLATILE_HDM_PRES_ERROR + * clearing. Stamp the new outcome on top of that value below. + */ + pci_read_config_word(pdev, dvsec + CXL_DVSEC_STATUS2_OFFSET, + &hw_status2); + hw_status2 &= ~(CXL_DVSEC_STATUS2_CXL_RESET_COMPLETE | + CXL_DVSEC_STATUS2_CXL_RESET_ERROR); + if (ret) + hw_status2 |= CXL_DVSEC_STATUS2_CXL_RESET_ERROR; + else + hw_status2 |= CXL_DVSEC_STATUS2_CXL_RESET_COMPLETE; + dvsec_virt_write16(vdev, CXL_DVSEC_STATUS2_OFFSET, hw_status2); + + up_write(&vdev->memory_lock); + return ret; +} + static void cxl_dvsec_control2_write(struct vfio_pci_core_device *vdev, u16 new_val) { @@ -141,14 +220,31 @@ static void cxl_dvsec_control2_write(struct vfio_pci_core_device *vdev, } /* - * CXL Reset: not yet supported - do not forward to HW. - * TODO: invoke CXL protocol reset via cxl subsystem + * Commit the new CONTROL2 value to the shadow before triggering a + * reset. vfio_cxl_reset() reads Mem_Clr_Enable (bit 3) from the + * shadow; if the shadow is written after the reset call, a guest write + * that changes bit 3 in the same access as INITIATE_CXL_RESET would + * reset with the stale bit 3 value instead of the one just written. */ - if (new_val & CXL_DVSEC_CTRL2_INITIATE_CXL_RESET) - pci_warn(pdev, "vfio-cxl: CXL reset requested but not yet supported\n"); - dvsec_virt_write16(vdev, CXL_DVSEC_CONTROL2_OFFSET, new_val & ~CXL_CTRL2_HW_BITS_MASK); + + /* + * INIT_CXL_RST: not forwarded to hardware. cxl_dev_reset() drives + * the state machine; forwarding it after the reset would fire a + * second one. Drop writes on non-RST_CAPABLE devices silently; the + * spec reserves the bit there and logging every write is just noise. + */ + if (new_val & CXL_DVSEC_CTRL2_INITIATE_CXL_RESET) { + if (vfio_cxl_reset_capable(vdev)) { + int rc = vfio_cxl_reset(vdev); + + if (rc) + pci_warn(pdev, + "vfio-cxl: CXL reset failed (%d)\n", + rc); + } + } } static void cxl_dvsec_status2_write(struct vfio_pci_core_device *vdev, From d57b4c153d31729c1c36f5cba037405f5e3b3589 Mon Sep 17 00:00:00 2001 From: Jiandi An Date: Tue, 5 May 2026 16:20:03 -0500 Subject: [PATCH 413/464] NVIDIA: VR: SAUCE: config: Enable CONFIG_VFIO_CXL_CORE for CXL Type-2 passthrough BugLink: https://bugs.launchpad.net/bugs/2152222 Enable VFIO CXL core support on amd64 and arm64 to allow CXL Type-2 device passthrough via vfio-pci. Signed-off-by: Jiandi An (backported from commit 74b6b99bcd80 from https://github.com/JiandiAnNVIDIA/NV-Kernels.git cxl-vfio_2026-04-23) [kobak: Applied the equivalent annotation to debian.master/config/annotations because the 7.0 HWE target does not carry debian.nvidia-6.17/config/annotations.] Signed-off-by: Koba Ko Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Acked-by: Carol L Soto Signed-off-by: Brad Figg --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 3787b2b11b2c2..cf33cb565adb8 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -279,6 +279,9 @@ CONFIG_ULTRASOC_SMB note<'Required for Grace enablem CONFIG_VFIO_CONTAINER policy<{'amd64': 'y', 'arm64': 'n'}> CONFIG_VFIO_CONTAINER note<'LP: #2095028'> +CONFIG_VFIO_CXL_CORE policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_VFIO_CXL_CORE note<'Enable VFIO CXL core for CXL Type-2 device passthrough support'> + CONFIG_VFIO_IOMMU_TYPE1 policy<{'amd64': 'm', 'arm64': '-'}> CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> From 02e988be6a603adcdd363f862abf98f669e79edc Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 4 Jun 2026 09:14:24 -0500 Subject: [PATCH 414/464] UBUNTU: [Packaging] update Ubuntu.md BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- Ubuntu.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Ubuntu.md b/Ubuntu.md index e6dab098bdf6a..5693799265f98 100644 --- a/Ubuntu.md +++ b/Ubuntu.md @@ -2,7 +2,7 @@ Name: linux-nvidia-bos Version: 7.0.0 Series: 26.04 (resolute) Description: - This is the source code for the NVIDIA BOS linux kernel for the 26.04 series. This - source tree is used to produce the flavours: nvidia, nvidia-64k. + This is the source code for the Ubuntu linux kernel for the 26.04 series. This + source tree is used to produce the flavours: nvidia-bos, nvidia-bos-64k. This kernel is configured to support the widest range of desktop, laptop and server configurations. From cfd65a60e1f653956e38706b1d629c3e38878d39 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 4 Jun 2026 09:31:46 -0500 Subject: [PATCH 415/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 5de21acf22d47..7bf81f4588115 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2009.9) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Thu, 04 Jun 2026 09:31:46 -0500 + linux-nvidia-bos (7.0.0-2008.8) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2007.7 -proposed tracker (LP: #2153497) From 3b65374ec9dcaa03d7656dab0bd73b76c9da5485 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 4 Jun 2026 09:36:30 -0500 Subject: [PATCH 416/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2155210 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index 572c4a12d5dcf..a53d8c75304b6 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2153497 d2026.05.20-1 +2155210 d2026.06.03-1 From 1116defe9d101780b2e30ec9fde6357a7f9e51c7 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 4 Jun 2026 09:39:29 -0500 Subject: [PATCH 417/464] UBUNTU: [Config] nvidia: update annotations Minor reordering and expected auto-selected kernel configs. Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/config/annotations | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index cf33cb565adb8..152d6d69fc387 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -249,6 +249,9 @@ CONFIG_PINCTRL_MT8901 note<'LP: #2117784'> CONFIG_R8127 policy<{'amd64': 'n', 'arm64': 'm'}> CONFIG_R8127 note<'LP: #2109730'> +CONFIG_RESCTRL_FS policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_RESCTRL_FS note<'LP: #2122432'> + CONFIG_SAMPLE_CORESIGHT_SYSCFG policy<{'arm64': 'n'}> CONFIG_SAMPLE_CORESIGHT_SYSCFG note<'Required for Grace enablement'> @@ -285,11 +288,14 @@ CONFIG_VFIO_CXL_CORE note<'Enable VFIO CXL core for C CONFIG_VFIO_IOMMU_TYPE1 policy<{'amd64': 'm', 'arm64': '-'}> CONFIG_VFIO_IOMMU_TYPE1 note<'LP: #2095028'> -CONFIG_RESCTRL_FS policy<{'amd64': 'y', 'arm64': 'y'}> -CONFIG_RESCTRL_FS note<'LP: #2122432'> # ---- Annotations without notes ---- +CONFIG_ARCH_HAS_CPU_RESCTRL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_ARM64_MPAM_RESCTRL_FS policy<{'arm64': 'y'}> CONFIG_BCH policy<{'amd64': 'm', 'arm64': 'y'}> CONFIG_HISI_SOC_HHA policy<{'arm64': '-'}> CONFIG_MTD_NAND_CORE policy<{'amd64': 'm', 'arm64': 'y'}> +CONFIG_PROC_CPU_RESCTRL policy<{'amd64': 'y', 'arm64': 'y'}> +CONFIG_RESCTRL_IOMMU policy<{'arm64': 'y'}> +CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID policy<{'arm64': 'y'}> From bd9f668b762b433ec2a08091cc79100a74e948a2 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 4 Jun 2026 09:40:45 -0500 Subject: [PATCH 418/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2009.9 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 221 +++++++++++++++++++++++++++++++++++- 1 file changed, 216 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 7bf81f4588115..fe8d960379aac 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,221 @@ -linux-nvidia-bos (7.0.0-2009.9) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2009.9) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2009.9 -proposed tracker (LP: #2155210) - -- Jacob Martin Thu, 04 Jun 2026 09:31:46 -0500 + * Packaging resync (LP: #1786013) + - [Packaging] update Ubuntu.md + + * CXL VFIO: Add CXL Type-2 device passthrough support (LP: #2152222) + - NVIDIA: VR: SAUCE: cxl: Add cxl_get_hdm_info() for HDM decoder metadata + - NVIDIA: VR: SAUCE: cxl: Declare cxl_probe_component_regs in public + header + - NVIDIA: VR: SAUCE: cxl: Move component/HDM register defines to + uapi/cxl/cxl_regs.h + - NVIDIA: VR: SAUCE: cxl: Split cxl_await_range_active() from media-ready + wait + - NVIDIA: VR: SAUCE: cxl: Record BIR and BAR offset in cxl_register_map + - NVIDIA: VR: SAUCE: vfio: UAPI for CXL-capable PCI device assignment + - NVIDIA: VR: SAUCE: vfio/pci: Add CXL state to vfio_pci_core_device + - NVIDIA: VR: SAUCE: vfio/pci: Add CONFIG_VFIO_CXL_CORE and stub CXL hooks + - NVIDIA: VR: SAUCE: vfio/cxl: Detect CXL DVSEC and probe HDM block + - NVIDIA: VR: SAUCE: vfio/pci: Export config access helpers + - NVIDIA: VR: SAUCE: vfio/cxl: Introduce HDM decoder register emulation + framework + - NVIDIA: VR: SAUCE: vfio/cxl: Wait for HDM ranges and create memdev + - NVIDIA: VR: SAUCE: vfio/cxl: CXL region management support + - NVIDIA: VR: SAUCE: vfio/cxl: DPA VFIO region with demand fault mmap and + reset zap + - NVIDIA: VR: SAUCE: vfio/cxl: Virtualize CXL DVSEC config writes + - NVIDIA: VR: SAUCE: vfio/cxl: Register regions with VFIO layer + - NVIDIA: VR: SAUCE: vfio/pci: Advertise CXL cap and sparse component BAR + to userspace + - NVIDIA: VR: SAUCE: vfio/cxl: Provide opt-out for CXL feature + - NVIDIA: VR: SAUCE: docs: vfio-pci: Document CXL Type-2 device + passthrough + - selftests/vfio: Add CXL Type-2 VFIO assignment test + - NVIDIA: VR: SAUCE: cxl: Export the CXL reset helpers for VFIO users + - NVIDIA: VR: SAUCE: vfio/pci: Wire CXL DPA reset handling + - NVIDIA: VR: SAUCE: vfio/cxl: Ensure PCI Memory Space is enabled before + post-reset BAR access + - NVIDIA: VR: SAUCE: vfio/cxl: preserve HDM decoder base addresses across + reset + - NVIDIA: VR: SAUCE: vfio/cxl: virtualize DVSEC STATUS2 register in + vconfig shadow + - NVIDIA: VR: SAUCE: vfio/cxl: Implement vfio_cxl_reset() + - NVIDIA: VR: SAUCE: config: Enable CONFIG_VFIO_CXL_CORE for CXL Type-2 + passthrough + + * linux-nvidia: Port MPAM Functionality into Kernel (LP: #2154527) + - arm_mpam: Ensure in_reset_state is false after applying configuration + - arm_mpam: Reset when feature configuration bit unset + - arm64/sysreg: Add MPAMSM_EL1 register + - KVM: arm64: Preserve host MPAM configuration when changing traps + - KVM: arm64: Make MPAMSM_EL1 accesses UNDEF + - arm64: mpam: Context switch the MPAM registers + - arm64: mpam: Re-initialise MPAM regs when CPU comes online + - arm64: mpam: Drop the CONFIG_EXPERT restriction + - arm64: mpam: Advertise the CPUs MPAM limits to the driver + - arm64: mpam: Add cpu_pm notifier to restore MPAM sysregs + - arm64: mpam: Initialise and context switch the MPAMSM_EL1 register + - arm64: mpam: Add helpers to change a task or cpu's MPAM PARTID/PMG + values + - arm_mpam: resctrl: Add boilerplate cpuhp and domain allocation + - arm_mpam: resctrl: Pick the caches we will use as resctrl resources + - arm_mpam: resctrl: Implement resctrl_arch_reset_all_ctrls() + - arm_mpam: resctrl: Add resctrl_arch_get_config() + - arm_mpam: resctrl: Implement helpers to update configuration + - arm_mpam: resctrl: Add plumbing against arm64 task and cpu hooks + - arm_mpam: resctrl: Add CDP emulation + - arm_mpam: resctrl: Hide CDP emulation behind CONFIG_EXPERT + - arm_mpam: resctrl: Convert to/from MPAMs fixed-point formats + - arm_mpam: resctrl: Add rmid index helpers + - arm_mpam: resctrl: Wait for cacheinfo to be ready + - arm_mpam: resctrl: Add support for 'MB' resource + - arm_mpam: resctrl: Add kunit test for control format conversions + - arm_mpam: resctrl: Add monitor initialisation and domain boilerplate + - arm_mpam: resctrl: Add support for csu counters + - arm_mpam: resctrl: Allow resctrl to allocate monitors + - arm_mpam: resctrl: Add resctrl_arch_rmid_read() + - arm_mpam: resctrl: Update the rmid reallocation limit + - arm_mpam: resctrl: Add empty definitions for assorted resctrl functions + - ALSA: usb-audio: Replace hard-coded number with MAX_CHANNELS + - arm64: mpam: Select ARCH_HAS_CPU_RESCTRL + - arm_mpam: resctrl: Call resctrl_init() on platforms that can support + resctrl + - arm_mpam: Add quirk framework + - arm_mpam: Add workaround for T241-MPAM-1 + - arm_mpam: Add workaround for T241-MPAM-4 + - arm_mpam: Add workaround for T241-MPAM-6 + - arm_mpam: Quirk CMN-650's CSU NRDY behaviour + - arm64: mpam: Add initial MPAM documentation + - fs/resctrl: Report invalid domain ID when parsing io_alloc_cbm + - fs/resctrl: Add "*" shorthand to set io_alloc CBM for all domains + - MAINTAINERS: Update resctrl entry + - fs/resctrl: Add missing return value descriptions + - arm_mpam: resctrl: Fix MBA CDP alloc_capable handling on unmount + - arm_mpam: resctrl: Fix the check for no monitor components found + - arm_mpam: resctrl: Make resctrl_mon_ctx_waiters static + - NVIDIA: SAUCE: Update annotations to set CONFIG_RESCTRL_FS + - NVIDIA: SAUCE: untested: arm_mpam: resctrl: pick classes for use as mbm + counters + - NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate free running monitors + - NVIDIA: SAUCE: arm_mpam: resctrl: Pre-allocate assignable monitors + - NVIDIA: VR: SAUCE: arm_mpam: resctrl: Add kunit test for ABMC/CDP + interactions + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_config_cntr() for + ABMC use + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_rmid_read() and + resctrl_arch_reset_rmid() + - NVIDIA: SAUCE: arm_mpam: resctrl: Add resctrl_arch_cntr_read() & + resctrl_arch_reset_cntr() + - NVIDIA: VR: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and + closid_num_dirty_rmid[] + - NVIDIA: VR: SAUCE: fs/resctrl: Avoid a race with dom_data_exit() and + rmid_ptrs[] + - NVIDIA: VR: SAUCE: debugfs: Add helpers for creating cpumask entries in + debugfs + - NVIDIA: VR: SAUCE: arm_mpam: Add debugfs entries to show the MSC/RIS the + driver discovered + - NVIDIA: VR: SAUCE: arm_mpam: Add force-disable debugfs trigger + - NVIDIA: VR: SAUCE: arm_mpam: Expose the number of NRDY retries in + debugfs + - NVIDIA: SAUCE: arm_mpam: Add resctrl_arch_round_bw() + - NVIDIA: SAUCE: fs/resctrl,x86/resctrl: Factor mba rounding to be per- + arch + - NVIDIA: VR: SAUCE: arm_mpam: Split the locking around the mon_sel + registers + - NVIDIA: VR: SAUCE: arm_mpam: Allow the maximum partid to be overridden + from the command line + - NVIDIA: VR: SAUCE: arm_mpam: Allow MSC to be forced to have an unknown + location + - NVIDIA: VR: SAUCE: fs/resctrl: Add this_is_not_abi mount option + - NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Register SMMU capabilities with + MPAM + - NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Add mpam helpers to query and set + state + - NVIDIA: VR: SAUCE: iommu: Add helpers to get and set the QoS state + - NVIDIA: VR: SAUCE: iommu: Add helpers to retrieve iommu_groups by id or + kobject + - NVIDIA: VR: SAUCE: iommu: Add helper to retrieve iommu kset + - NVIDIA: VR: SAUCE: kobject: Add kset_get_next_obj() to allow a kset to + be walked + - NVIDIA: VR: SAUCE: arm_mpam: resctrl: Add iommu helpers to get/set the + partid and pmg + - NVIDIA: VR: SAUCE: fs/resctrl: Add support for assigning iommu_groups to + resctrl groups + - NVIDIA: SAUCE: x86/resctrl: Add stub to allow other architecture to + disable monitor overflow + - NVIDIA: SAUCE: arm_mpam: resctrl: Determine if any exposed counter can + overflow + - NVIDIA: SAUCE: fs/restrl: Allow the overflow handler to be disabled + - NVIDIA: VR: SAUCE: fs/resctrl: Uniform data type of + component_id/domid/id/cache_id + - NVIDIA: SAUCE: arm_mpam: Allow cmax/cmin to be configured + - NVIDIA: SAUCE: arm_mpam: Rename mbw conversion to 'fract16' for code re- + use + - NVIDIA: SAUCE: fs/resctrl: Group all the MBA specific properties in a + separate struct + - NVIDIA: SAUCE: fs/resctrl: Abstract duplicate domain test to a helper + - NVIDIA: SAUCE: fs/resctrl: Move MBA supported check to parse_line() + instead of parse_bw() + - NVIDIA: SAUCE: fs/resctrl: Rename resctrl_get_default_ctrl() to include + resource + - NVIDIA: SAUCE: fs/resctrl: Add a schema format to the schema, allowing + it to be different + - NVIDIA: VR: SAUCE: fs/resctrl: Use schema format to check the resource + is a bitmap + - NVIDIA: SAUCE: fs/resctrl: Add specific schema types for 'range' + - NVIDIA: SAUCE: x86/resctrl: Move over to specifying MBA control formats + - NVIDIA: VR: SAUCE: arm_mpam: resctrl: Convert MB resource to use + percentage + - NVIDIA: VR: SAUCE: fs/resctrl: Remove 'range' schema format + - NVIDIA: SAUCE: fs/resctrl: Add additional files for percentage and + bitmap controls + - NVIDIA: SAUCE: fs/resctrl: Add fflags_from_schema() for files based on + schema format + - NVIDIA: SAUCE: fs/resctrl: Expose the schema format to user-space + - NVIDIA: SAUCE: fs/resctrl: Add L2 and L3 'MAX' resource schema + - NVIDIA: SAUCE: arm_mpam: resctrl: Add the glue code to convert to/from + cmax + - NVIDIA: VR: SAUCE: arm_mpam: Avoid MSC teardown for the SW programming + errors + - NVIDIA: VR: SAUCE: arm_mpam: Handle CPU-less numa nodes + - NVIDIA: SAUCE: arm_mpam: Include all associated MSC components during + domain setup + - NVIDIA: SAUCE: resctrl/mpam: reset RIS by applying explicit default + config + - NVIDIA: VR: SAUCE: iommu/arm-smmu-v3: Fix MPAM for indentity-mappings + - NVIDIA: VR: SAUCE: resctrl: add MB_HLIM resource ID and schema type + - NVIDIA: VR: SAUCE: resctrl: wire MB_HLIM schemata parsing and group + setup + - NVIDIA: VR: SAUCE: resctrl/mpam: probe MPAMF_MBW_IDR MAX_LIM and + hardlim_rw + - NVIDIA: VR: SAUCE: resctrl/mpam: track MBW max hard-limit in config + - NVIDIA: VR: SAUCE: resctrl/mpam: bind MB_HLIM resource to MBA MPAM class + - NVIDIA: VR: SAUCE: resctrl: add membw.mb_max_lim and arch_has_mb_max_lim + - NVIDIA: VR: SAUCE: resctrl/mpam: populate MBA mb_max_lim from MPAM probe + - NVIDIA: VR: SAUCE: resctrl: add MBA max_lim sysfs and visibility from + init + - NVIDIA: VR: SAUCE: Documentation: resctrl: document max_lim and MB_HLIM + for MPAM MBA + - NVIDIA: SAUCE: Fix mbm_L3_assign and mon_local_bytes + + * Backport PCI/CXL: Hide SBR from reset_methods if masked by CXL + (LP: #2154302) + - PCI/CXL: Hide SBR from reset_methods if masked by CXL + + * linux-nvidia: backport FF-A partition info descriptor size fix + (LP: #2154045) + - firmware: arm_ffa: Bound PARTITION_INFO_GET_REGS copies + - firmware: arm_ffa: Honor partition info descriptor size + + * Backport BPMP ACPI + SoC Hub MBWT sysfs driver for T410/NVL72 + (LP: #2147212) + - NVIDIA: VR: SAUCE: tegra: bpmp: Move channel, resource init to helper + - NVIDIA: VR: SAUCE: tegra: bpmp: Add ACPI support + - NVIDIA: VR: SAUCE: tegra: bpmp: Add sysfs for memory bandwidth QoS + + -- Jacob Martin Thu, 04 Jun 2026 09:40:45 -0500 linux-nvidia-bos (7.0.0-2008.8) resolute; urgency=medium From 1f9a33862a0afff0bbe0afe81972b2a0e2778401 Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Tue, 19 May 2026 12:42:20 -0700 Subject: [PATCH 419/464] fs/ntfs3: fix mount failure on 64K page-size kernels BugLink: https://bugs.launchpad.net/bugs/2155467 On 64K page-size kernels, mounting NTFS volumes smaller than ~650 MB fails with EINVAL. The issue is in log_replay(): the initial log page size probe uses PAGE_SIZE (65536) instead of DefaultLogPageSize (4096) when PAGE_SIZE exceeds DefaultLogPageSize * 2. This makes norm_file_page() require the $LogFile to be at least 50 * 65536 = 3.2 MB, but mkfs.ntfs creates a $LogFile of only ~1.5 MB for a typical 300 MB volume. norm_file_page() returns 0 and the mount is rejected with EINVAL. On 4K kernels the #if guard evaluates to true, so use_default=true is passed and DefaultLogPageSize (4096) is used, requiring only ~200 KB. This path works fine. Fix this by always passing use_default=true, which forces the initial probe to use DefaultLogPageSize regardless of the kernel's PAGE_SIZE. This is safe because, after reading the on-disk restart area, log_replay() already re-adjusts log->page_size to match the volume's actual sys_page_size. Also fix read_log_page() to pass log->page_size instead of PAGE_SIZE to ntfs_fix_post_read(), matching the actual buffer size. Fixes: b46acd6a6a62 ("fs/ntfs3: Add NTFS journal") Tested-by: Matthew R. Ochs Signed-off-by: Jamie Nguyen Signed-off-by: Konstantin Komarov (cherry picked from commit b7a9125cac8645245d2473c6c0a50e338280ad23 linux-next) Signed-off-by: Jamie Nguyen Acked-by: Seth Forshee Acked-by: Carol L Soto Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- fs/ntfs3/fslog.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/fs/ntfs3/fslog.c b/fs/ntfs3/fslog.c index 037df47fa9f3a..272ea39e3c999 100644 --- a/fs/ntfs3/fslog.c +++ b/fs/ntfs3/fslog.c @@ -1172,7 +1172,7 @@ static int read_log_page(struct ntfs_log *log, u32 vbo, goto out; if (page_buf->rhdr.sign != NTFS_FFFF_SIGNATURE) - ntfs_fix_post_read(&page_buf->rhdr, PAGE_SIZE, false); + ntfs_fix_post_read(&page_buf->rhdr, log->page_size, false); if (page_buf != *buffer) memcpy(*buffer, Add2Ptr(page_buf, page_off), bytes); @@ -3796,11 +3796,7 @@ int log_replay(struct ntfs_inode *ni, bool *initialized) log->l_size = log->orig_file_size = ni->vfs_inode.i_size; /* Get the size of page. NOTE: To replay we can use default page. */ -#if PAGE_SIZE >= DefaultLogPageSize && PAGE_SIZE <= DefaultLogPageSize * 2 log->page_size = norm_file_page(PAGE_SIZE, &log->l_size, true); -#else - log->page_size = norm_file_page(PAGE_SIZE, &log->l_size, false); -#endif if (!log->page_size) { err = -EINVAL; goto out; From 0521fed1a4a1fae08479e808715bcff4901131f8 Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Fri, 22 May 2026 06:33:01 -0700 Subject: [PATCH 420/464] NVIDIA: SAUCE: PCI: quirks: mirror PI7C9X3G606GPC Port 4 BAR0 BugLink: https://bugs.launchpad.net/bugs/2154457 Some Pericom/Diodes PI7C9X3G606GPC switches require downstream Port 4 BAR0 to mirror BAR0 of the immediate upstream port. Firmware may apply this during boot, but Linux PCI resource assignment can move the upstream BAR0 and leave Port 4 without the required mirror. Diodes confirmed that Tile0/P4 is OS-visible as device 04, function 0 on the bus below the upstream port. Add a final and early resume quirk for that downstream function. The quirk verifies that the immediate upstream bridge is the same switch, then writes Port 4 BAR0 from the upstream BAR0 after resource assignment and during early resume. If BAR0 is configured as a 64-bit memory BAR, mirror BAR1 as the upper dword as well. Port 4 BAR0 may read back as zero even after a successful write, so the write must be validated by platform-specific means. Signed-off-by: Nirmoy Das Acked-by: Matthew R. Ochs Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/pci/quirks.c | 88 +++++++++++++++++++++++++++++++++++++++++ include/linux/pci_ids.h | 1 + 2 files changed, 89 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index b9d6c37272ddd..325f75dd51ce1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -6282,6 +6282,94 @@ DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0xb404, DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0xb404, pci_fixup_pericom_acs_store_forward); +/* + * Some Pericom/Diodes PI7C9X3G606GPC switches require downstream Port 4 + * BAR 0 to mirror BAR 0 of the immediate upstream port. Firmware may + * program this during boot, but Linux resource assignment can move the + * upstream BAR. + * + * Diodes confirmed Tile0/P4 appears to Linux as device 4, function 0 on + * the bus below the upstream port. Match that downstream function and + * re-apply the mirror after resource assignment and early resume. + */ +static void pci_fixup_pericom_pi7c9x3g606gpc_bar0_mirror(struct pci_dev *pdev) +{ + struct pci_dev *upstream; + bool bar0_64, disable_mem; + u16 cmd = 0; + u32 bar = 0, bar1 = 0, upstream_bar = 0, upstream_bar1 = 0; + + if (pci_pcie_type(pdev) != PCI_EXP_TYPE_DOWNSTREAM) + return; + + if (PCI_SLOT(pdev->devfn) != 4 || PCI_FUNC(pdev->devfn)) + return; + + upstream = pci_upstream_bridge(pdev); + if (!upstream || upstream->vendor != PCI_VENDOR_ID_PERICOM || + upstream->device != PCI_DEVICE_ID_PERICOM_PI7C9X3G606GPC || + pci_pcie_type(upstream) != PCI_EXP_TYPE_UPSTREAM) + return; + + pci_read_config_dword(upstream, PCI_BASE_ADDRESS_0, &upstream_bar); + if (upstream_bar & PCI_BASE_ADDRESS_SPACE_IO) + return; + + bar0_64 = (upstream_bar & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == + PCI_BASE_ADDRESS_MEM_TYPE_64; + if (bar0_64) + pci_read_config_dword(upstream, PCI_BASE_ADDRESS_1, + &upstream_bar1); + + if (!(upstream_bar & PCI_BASE_ADDRESS_MEM_MASK) && + (!bar0_64 || !upstream_bar1)) { + pci_warn(pdev, "skipping PI7C9X3G606GPC BAR 0 mirror workaround because upstream BAR 0 is unassigned\n"); + return; + } + + pci_read_config_dword(pdev, PCI_BASE_ADDRESS_0, &bar); + if (bar0_64) { + pci_read_config_dword(pdev, PCI_BASE_ADDRESS_1, &bar1); + if (bar == upstream_bar && bar1 == upstream_bar1) + return; + } else { + if (bar == upstream_bar) + return; + } + + /* + * Port 4 BAR 0 may read back as zero even after a successful write. + * If BAR 0 is configured as 64-bit, BAR 1 is the upper half. + * Disable memory decoding while updating both dwords, matching PCI + * core's 64-bit BAR update sequence. + */ + disable_mem = bar0_64 && !pdev->mmio_always_on; + if (disable_mem) { + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + pci_write_config_word(pdev, PCI_COMMAND, + cmd & ~PCI_COMMAND_MEMORY); + } + + pci_write_config_dword(pdev, PCI_BASE_ADDRESS_0, upstream_bar); + if (bar0_64) { + pci_write_config_dword(pdev, PCI_BASE_ADDRESS_1, + upstream_bar1); + if (disable_mem) + pci_write_config_word(pdev, PCI_COMMAND, cmd); + pci_info(pdev, "wrote upstream BAR 0/1 %#x/%#x to Port 4 BAR 0/1 for PI7C9X3G606GPC BAR 0 mirror workaround\n", + upstream_bar, upstream_bar1); + } else { + pci_info(pdev, "wrote upstream BAR 0 %#x to Port 4 BAR 0 for PI7C9X3G606GPC BAR 0 mirror workaround\n", + upstream_bar); + } +} +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_PERICOM, + PCI_DEVICE_ID_PERICOM_PI7C9X3G606GPC, + pci_fixup_pericom_pi7c9x3g606gpc_bar0_mirror); +DECLARE_PCI_FIXUP_RESUME_EARLY(PCI_VENDOR_ID_PERICOM, + PCI_DEVICE_ID_PERICOM_PI7C9X3G606GPC, + pci_fixup_pericom_pi7c9x3g606gpc_bar0_mirror); + static void nvidia_ion_ahci_fixup(struct pci_dev *pdev) { pdev->dev_flags |= PCI_DEV_FLAGS_HAS_MSI_MASKING; diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h index 406abf629be2e..b230d3b0a026e 100644 --- a/include/linux/pci_ids.h +++ b/include/linux/pci_ids.h @@ -1849,6 +1849,7 @@ #define PCI_DEVICE_ID_PERICOM_PI7C9X7952 0x7952 #define PCI_DEVICE_ID_PERICOM_PI7C9X7954 0x7954 #define PCI_DEVICE_ID_PERICOM_PI7C9X7958 0x7958 +#define PCI_DEVICE_ID_PERICOM_PI7C9X3G606GPC 0xc008 #define PCI_SUBVENDOR_ID_CHASE_PCIFAST 0x12E0 #define PCI_SUBDEVICE_ID_CHASE_PCIFAST4 0x0031 From 1f953f17330b7c43be5b62ee43467cd9fb69513e Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Tue, 9 Jun 2026 11:12:01 +0100 Subject: [PATCH 421/464] arm64: cputype: Add C1-Ultra definitions BugLink: https://bugs.launchpad.net/bugs/2156557 Add cputype definitions for C1-Ultra. These will be used for errata detection in subsequent patches. These values can be found in the C1-Ultra TRM: https://developer.arm.com/documentation/108014/0100/ ... in section A.5.1 ("MIDR_EL1, Main ID Register"). Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Will Deacon (cherry picked from commit 60349e64a6c65f9f0aa118af711b3c7e137f07ff linux-next) Signed-off-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- arch/arm64/include/asm/cputype.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 08860d482e600..70beef0712424 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -97,6 +97,7 @@ #define ARM_CPU_PART_CORTEX_X925 0xD85 #define ARM_CPU_PART_CORTEX_A725 0xD87 #define ARM_CPU_PART_CORTEX_A720AE 0xD89 +#define ARM_CPU_PART_C1_ULTRA 0xD8C #define ARM_CPU_PART_NEOVERSE_N3 0xD8E #define APM_CPU_PART_XGENE 0x000 @@ -188,6 +189,7 @@ #define MIDR_CORTEX_X925 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_X925) #define MIDR_CORTEX_A725 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A725) #define MIDR_CORTEX_A720AE MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A720AE) +#define MIDR_C1_ULTRA MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_C1_ULTRA) #define MIDR_NEOVERSE_N3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N3) #define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX) #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX) From 9ffcfa3198c515e7ecc1ce8b65181bba126baa8d Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Tue, 9 Jun 2026 11:12:02 +0100 Subject: [PATCH 422/464] arm64: cputype: Add C1-Premium definitions BugLink: https://bugs.launchpad.net/bugs/2156557 Add cputype definitions for C1-Premium. These will be used for errata detection in subsequent patches. These values can be found in the C1-Premium TRM: https://developer.arm.com/documentation/109416/0100/ ... in section A.5.1 ("MIDR_EL1, Main ID Register"). Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Will Deacon (backported from commit d28413bfc5a255957241f1df5d7fd0c2cd74fe18 linux-next) [mochs: Minor context adjustment due to absent definitions] Signed-off-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- arch/arm64/include/asm/cputype.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 70beef0712424..45f96ca01cea5 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -99,6 +99,7 @@ #define ARM_CPU_PART_CORTEX_A720AE 0xD89 #define ARM_CPU_PART_C1_ULTRA 0xD8C #define ARM_CPU_PART_NEOVERSE_N3 0xD8E +#define ARM_CPU_PART_C1_PREMIUM 0xD90 #define APM_CPU_PART_XGENE 0x000 #define APM_CPU_VAR_POTENZA 0x00 @@ -191,6 +192,7 @@ #define MIDR_CORTEX_A720AE MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A720AE) #define MIDR_C1_ULTRA MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_C1_ULTRA) #define MIDR_NEOVERSE_N3 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N3) +#define MIDR_C1_PREMIUM MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_C1_PREMIUM) #define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX) #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX) #define MIDR_THUNDERX_83XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_83XX) From 43aed3749a741096c528a328d87d85467bf30fbf Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Tue, 9 Jun 2026 11:12:03 +0100 Subject: [PATCH 423/464] arm64: errata: Mitigate TLBI errata on various Arm CPUs BugLink: https://bugs.launchpad.net/bugs/2156557 A number of CPUs developed by Arm suffer from errata whereby a broadcast TLBI;DSB sequence may complete before the global observation of writes which are translated by an affected TLB entry. These errata ONLY affect the completion of memory accesses which have been translated by an invalidated TLB entry, and these errata DO NOT affect the actual invalidation of TLB entries. TLB entries are removed correctly. This issue has been assigned CVE ID CVE-2025-10263. To mitigate this issue, Arm recommends that software follows any affected TLBI;DSB sequence with an additional TLBI;DSB, which will ensure that all memory write effects affected by the first TLBI have been globally observed. The additional TLBI can use any operation that is broadcast to affected CPUs, and the additional DSB can use any option that is sufficient to complete the additional TLBI. The ARM64_WORKAROUND_REPEAT_TLBI workaround is sufficient to mitigate the issue. Enable this workaround for affected CPUs, and update the silicon errata documentation accordingly. Note that due to the manner in which Arm develops IP and tracks errata, some CPUs share a common erratum number. Signed-off-by: Mark Rutland Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Will Deacon (backported from commit cfd391e74134db664feb499d43af286380b10ba8 linux-next) [mochs: Minor context adjustment due to absent definitions] Signed-off-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 42 +++++++++++++++++++++ arch/arm64/Kconfig | 36 ++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 32 +++++++++++++++- 3 files changed, 108 insertions(+), 2 deletions(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index 65ed6ea33751f..c853204585751 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -128,16 +128,28 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A76 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A76 | #4193800 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A76AE | #4193801 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1491015 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1508412 | ARM64_ERRATUM_1508412 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #3324348 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A77 | #4193798 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78 | #4193791 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78AE | #4193793 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78C | #3324346,3324347| ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78C | #4193794 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2119858 | ARM64_ERRATUM_2119858 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2054223 | ARM64_ERRATUM_2054223 | @@ -146,6 +158,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A710 | #4193788 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #2645198 | ARM64_ERRATUM_2645198 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #3456084 | ARM64_ERRATUM_3194386 | @@ -158,20 +172,32 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X1 | #4193791 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1C | #3324346 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X1C | #4193792 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #2119858 | ARM64_ERRATUM_2119858 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #2224489 | ARM64_ERRATUM_2224489 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X2 | #4193788 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X3 | #3324335 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X3 | #4193786 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X4 | #3194386 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X4 | #4118414 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X925 | #3324334 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X925 | #4193781 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1188873,1418040| ARM64_ERRATUM_1418040 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1349291 | N/A | @@ -182,6 +208,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N1 | #4193800 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2139208 | ARM64_ERRATUM_2139208 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2067961 | ARM64_ERRATUM_2067961 | @@ -190,18 +218,32 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #3324339 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N2 | #4193789 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N3 | #3456111 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #1619801 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #3324341 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V1 | #4193790 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V2 | #3324336 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V2 | #4193787 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V3 | #3312417 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V3 | #4193784 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V3AE | #3312417 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V3AE | #4193784 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | C1-Premium | #4193780 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | C1-Ultra | #4193780 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | MMU-500 | #841119,826419 | ARM_SMMU_MMU_500_CPRE_ERRATA| | | | #562869,1047329 | | +----------------+-----------------+-----------------+-----------------------------+ diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index bf706982e0335..126af13856bca 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1176,6 +1176,42 @@ config ARM64_ERRATUM_4311569 If unsure, say Y. +config ARM64_ERRATUM_4118414 + bool "Cortex-*/Neoverse-*/C1-*: Completion of affected memory accesses might not be guaranteed by completion of a TLBI" + default y + select ARM64_WORKAROUND_REPEAT_TLBI + help + This option adds a workaround for the following errata: + + * ARM C1-Premium erratum 4193780 + * ARM C1-Ultra erratum 4193780 + * ARM Cortex-A76 erratum 4193800 + * ARM Cortex-A76AE erratum 4193801 + * ARM Cortex-A77 erratum 4193798 + * ARM Cortex-A78 erratum 4193791 + * ARM Cortex-A78AE erratum 4193793 + * ARM Cortex-A78C erratum 4193794 + * ARM Cortex-A710 erratum 4193788 + * ARM Cortex-X1 erratum 4193791 + * ARM Cortex-X1C erratum 4193792 + * ARM Cortex-X2 erratum 4193788 + * ARM Cortex-X3 erratum 4193786 + * ARM Cortex-X4 erratum 4118414 + * ARM Cortex-X925 erratum 4193781 + * ARM Neoverse-N1 erratum 4193800 + * ARM Neoverse-N2 erratum 4193789 + * ARM Neoverse-V1 erratum 4193790 + * ARM Neoverse-V2 erratum 4193787 + * ARM Neoverse-V3 erratum 4193784 + * ARM Neoverse-V3AE erratum 4193784 + + On affected cores, some memory accesses might not be completed by + broadcast TLB invalidation. + + This issue is also known as CVE-2025-10263. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 059f5e322eb81..98f75750bf076 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -344,7 +344,35 @@ static const struct arm64_cpu_capabilities arm64_repeat_tlbi_list[] = { ERRATA_MIDR_RANGE(MIDR_CORTEX_A510, 0, 0, 1, 1), }, #endif - {}, +#ifdef CONFIG_ARM64_ERRATUM_4118414 + { + ERRATA_MIDR_RANGE_LIST(((const struct midr_range[]) { + MIDR_ALL_VERSIONS(MIDR_C1_PREMIUM), + MIDR_ALL_VERSIONS(MIDR_C1_ULTRA), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76AE), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78AE), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78C), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A710), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X1C), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X2), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X3), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X4), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X925), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3AE), + {} + })), + }, +#endif + {} }; #endif @@ -771,7 +799,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = { #endif #ifdef CONFIG_ARM64_WORKAROUND_REPEAT_TLBI { - .desc = "Qualcomm erratum 1009, or ARM erratum 1286807, 2441009", + .desc = "Broken broadcast TLBI completion", .capability = ARM64_WORKAROUND_REPEAT_TLBI, .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, .matches = cpucap_multi_entry_cap_matches, From fa2519467a1d9621343e703e3e1f102424ffc764 Mon Sep 17 00:00:00 2001 From: Shanker Donthineni Date: Tue, 9 Jun 2026 18:40:44 -0500 Subject: [PATCH 424/464] arm64: errata: Mitigate TLBI errata on NVIDIA Olympus CPU BugLink: https://bugs.launchpad.net/bugs/2156557 NVIDIA Olympus cores are affected by the TLBI completion issue tracked as CVE-2025-10263. The existing ARM64_ERRATUM_4118414 handling already uses ARM64_WORKAROUND_REPEAT_TLBI to issue an additional broadcast TLBI;DSB sequence and ensure affected memory write effects are globally observed. Add MIDR_NVIDIA_OLYMPUS to the repeat-TLBI match list so the same mitigation is enabled on affected Olympus systems. Also document the NVIDIA Olympus erratum in the arm64 silicon errata table and list it in the Kconfig help text. Signed-off-by: Shanker Donthineni Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Acked-by: Mark Rutland Signed-off-by: Will Deacon (cherry picked from commit ec7216f92e4ebd485b1c6dc6aa3f6064b71a5768 linux-next) Signed-off-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- Documentation/arch/arm64/silicon-errata.rst | 2 ++ arch/arm64/Kconfig | 3 ++- arch/arm64/kernel/cpu_errata.c | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index c853204585751..fa8b12e268712 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -290,6 +290,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM | +----------------+-----------------+-----------------+-----------------------------+ +| NVIDIA | Olympus core | T410-OLY-1029 | ARM64_ERRATUM_4118414 | ++----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | NVIDIA | T241 MPAM | T241-MPAM-1 | N/A | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 126af13856bca..ad5c8f7f5a005 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1177,7 +1177,7 @@ config ARM64_ERRATUM_4311569 If unsure, say Y. config ARM64_ERRATUM_4118414 - bool "Cortex-*/Neoverse-*/C1-*: Completion of affected memory accesses might not be guaranteed by completion of a TLBI" + bool "Various: Completion of affected memory accesses might not be guaranteed by completion of a TLBI" default y select ARM64_WORKAROUND_REPEAT_TLBI help @@ -1204,6 +1204,7 @@ config ARM64_ERRATUM_4118414 * ARM Neoverse-V2 erratum 4193787 * ARM Neoverse-V3 erratum 4193784 * ARM Neoverse-V3AE erratum 4193784 + * NVIDIA Olympus erratum T410-OLY-1029 On affected cores, some memory accesses might not be completed by broadcast TLB invalidation. diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 98f75750bf076..c5af584203762 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -368,6 +368,7 @@ static const struct arm64_cpu_capabilities arm64_repeat_tlbi_list[] = { MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2), MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3), MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3AE), + MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS), {} })), }, From 46f5fbb46ddce17885371ddb344587d445028192 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Thu, 11 Jun 2026 15:01:45 -0700 Subject: [PATCH 425/464] NVIDIA: [Config] Enable ARM64_ERRATUM_4118414 CVE-Enable ARM64_ERRATUM_4118414 to mitigate 2025-10263 on NVIDIA platforms. BugLink: https://bugs.launchpad.net/bugs/2156557 Signed-off-by: Matthew R. Ochs Acked-by: Nirmoy Das Acked-by: Carol L Soto Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- debian.nvidia-bos/config/annotations | 3 +++ 1 file changed, 3 insertions(+) diff --git a/debian.nvidia-bos/config/annotations b/debian.nvidia-bos/config/annotations index 152d6d69fc387..bcc6f1a3be0ab 100644 --- a/debian.nvidia-bos/config/annotations +++ b/debian.nvidia-bos/config/annotations @@ -36,6 +36,9 @@ CONFIG_ARM64_ERRATUM_2224489 note<'Required for Grace enablem CONFIG_ARM64_ERRATUM_2253138 policy<{'arm64': 'y'}> CONFIG_ARM64_ERRATUM_2253138 note<'Required for Grace enablement'> +CONFIG_ARM64_ERRATUM_4118414 policy<{'arm64': 'y'}> +CONFIG_ARM64_ERRATUM_4118414 note<'Required for Grace and Vera enablement'> + CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE policy<{'arm64': 'y'}> CONFIG_ARM64_WORKAROUND_NC_TO_NGNRE note<'Required for Vera enablement'> From c1bac171a68d163f3735b2aa239090ded86c4c70 Mon Sep 17 00:00:00 2001 From: "Matthew R. Ochs" Date: Tue, 26 May 2026 08:20:21 -0700 Subject: [PATCH 426/464] fuse: back uncached readdir buffers with pages BugLink: https://bugs.launchpad.net/bugs/2156632 Commit dabb90391028 ("fuse: increase readdir buffer size") changed fuse_readdir_uncached() to size its temporary buffer from ctx->count. This is useful for overlayfs and other in-kernel callers that use INT_MAX to indicate an unlimited directory read. The larger buffer is currently supplied as a kvec output argument. For virtiofs, kvec arguments are copied through req->argbuf, which is allocated with kmalloc(..., GFP_ATOMIC). A large uncached readdir buffer can therefore require a multi-megabyte contiguous atomic allocation before the request is queued. Avoid the large bounce-buffer allocation by backing uncached readdir output with pages and setting out_pages. Transports such as virtiofs can then pass the pages as scatter-gather entries instead of copying the output through argbuf. Map the pages with vm_map_ram() only while parsing the returned dirents. The existing parser can then continue to use a linear kernel mapping. [SzM: separate allocation of pages into a helper function] Fixes: dabb90391028 ("fuse: increase readdir buffer size") Cc: stable@vger.kernel.org Signed-off-by: Matthew R. Ochs Signed-off-by: Miklos Szeredi (cherry picked from commit 2fcb1dd15faba5657b825cc5d54423251c70e79a linux-next) Signed-off-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- fs/fuse/readdir.c | 85 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 67 insertions(+), 18 deletions(-) diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c index aae657fd56c0e..57a4d6254bc98 100644 --- a/fs/fuse/readdir.c +++ b/fs/fuse/readdir.c @@ -12,6 +12,7 @@ #include #include #include +#include static bool fuse_use_readdirplus(struct inode *dir, struct dir_context *ctx) { @@ -335,6 +336,43 @@ static int parse_dirplusfile(char *buf, size_t nbytes, struct file *file, return 0; } +static struct page **fuse_readdir_alloc_buf(struct fuse_args_pages *ap, size_t *bufsize) +{ + unsigned int i, nr_alloc, nr_pages = DIV_ROUND_UP(*bufsize, PAGE_SIZE); + struct page **pages = kcalloc(nr_pages, sizeof(*pages), GFP_KERNEL); + + if (!pages) + return NULL; + + nr_alloc = alloc_pages_bulk(GFP_KERNEL, nr_pages, pages); + if (!nr_alloc) + goto free_array; + + if (nr_alloc < nr_pages) { + nr_pages = nr_alloc; + *bufsize = (size_t) nr_pages << PAGE_SHIFT; + } + + ap->folios = fuse_folios_alloc(nr_pages, GFP_KERNEL, &ap->descs); + if (!ap->folios) + goto release_pages; + + for (i = 0; i < nr_pages; i++) { + ap->folios[i] = page_folio(pages[i]); + ap->descs[i].length = min_t(size_t, *bufsize - (size_t)i * PAGE_SIZE, PAGE_SIZE); + } + ap->num_folios = nr_pages; + ap->args.out_pages = true; + + return pages; + +release_pages: + release_pages(pages, nr_pages); +free_array: + kfree(pages); + return NULL; +} + static int fuse_readdir_uncached(struct file *file, struct dir_context *ctx) { int plus; @@ -343,18 +381,16 @@ static int fuse_readdir_uncached(struct file *file, struct dir_context *ctx) struct fuse_mount *fm = get_fuse_mount(inode); struct fuse_conn *fc = fm->fc; struct fuse_io_args ia = {}; - struct fuse_args *args = &ia.ap.args; + struct fuse_args_pages *ap = &ia.ap; void *buf; size_t bufsize = clamp((unsigned int) ctx->count, PAGE_SIZE, fc->max_pages << PAGE_SHIFT); u64 attr_version = 0, evict_ctr = 0; bool locked; + struct page **pages = fuse_readdir_alloc_buf(ap, &bufsize); - buf = kvmalloc(bufsize, GFP_KERNEL); - if (!buf) + if (!pages) return -ENOMEM; - args->out_args[0].value = buf; - plus = fuse_use_readdirplus(inode, ctx); if (plus) { attr_version = fuse_get_attr_version(fm->fc); @@ -364,24 +400,37 @@ static int fuse_readdir_uncached(struct file *file, struct dir_context *ctx) fuse_read_args_fill(&ia, file, ctx->pos, bufsize, FUSE_READDIR); } locked = fuse_lock_inode(inode); - res = fuse_simple_request(fm, args); + res = fuse_simple_request(fm, &ap->args); fuse_unlock_inode(inode, locked); - if (res >= 0) { - if (!res) { - struct fuse_file *ff = file->private_data; - - if (ff->open_flags & FOPEN_CACHE_DIR) - fuse_readdir_cache_end(file, ctx->pos); - } else if (plus) { - res = parse_dirplusfile(buf, res, file, ctx, attr_version, - evict_ctr); - } else { + if (res < 0) + goto out; + + if (!res) { + struct fuse_file *ff = file->private_data; + + if (ff->open_flags & FOPEN_CACHE_DIR) + fuse_readdir_cache_end(file, ctx->pos); + goto out; + } + + buf = vm_map_ram(pages, ap->num_folios, -1); + if (!buf) { + res = -ENOMEM; + } else { + if (plus) + res = parse_dirplusfile(buf, res, file, ctx, attr_version, evict_ctr); + else res = parse_dirfile(buf, res, file, ctx); - } + + vm_unmap_ram(buf, ap->num_folios); } +out: + kfree(ap->folios); + release_pages(pages, ap->num_folios); + kfree(pages); - kvfree(buf); fuse_invalidate_atime(inode); + return res; } From c303007f63ce4177c803eb4e3ecf21dbe73e74c8 Mon Sep 17 00:00:00 2001 From: Haiyang Zhang Date: Fri, 5 Jun 2026 14:22:56 -0700 Subject: [PATCH 427/464] net: mana: Add support for PF device 0x00C1 BugLink: https://bugs.launchpad.net/bugs/2156821 Update the device id table to include the new device id 0x00C1. This device's BAR layout is similar to VF's, update the function, mana_gd_init_registers(), accordingly. Signed-off-by: Haiyang Zhang Link: https://patch.msgid.link/20260605212302.2135499-1-haiyangz@linux.microsoft.com Signed-off-by: Jakub Kicinski (backported from commit 53a65db20a4f3fe6c01b1f789f9eae6b1244910f linux-next) [ltrager: minor merge conflicts] Signed-off-by: Lee Trager Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/net/ethernet/microsoft/mana/gdma_main.c | 7 +++++-- include/net/mana/gdma.h | 2 ++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c index c2e855ff3ca9a..76b6956f990a7 100644 --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c @@ -78,7 +78,7 @@ static void mana_gd_init_registers(struct pci_dev *pdev) { struct gdma_context *gc = pci_get_drvdata(pdev); - if (gc->is_pf) + if (gc->is_pf && !gc->is_pf2) mana_gd_init_pf_regs(pdev); else mana_gd_init_vf_regs(pdev); @@ -1956,7 +1956,7 @@ static void mana_gd_cleanup(struct pci_dev *pdev) static bool mana_is_pf(unsigned short dev_id) { - return dev_id == MANA_PF_DEVICE_ID; + return dev_id == MANA_PF_DEVICE_ID || dev_id == MANA_PF2_DEVICE_ID; } static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent) @@ -2003,6 +2003,8 @@ static int mana_gd_probe(struct pci_dev *pdev, const struct pci_device_id *ent) gc->numa_node = dev_to_node(&pdev->dev); gc->is_pf = mana_is_pf(pdev->device); + gc->is_pf2 = (pdev->device == MANA_PF2_DEVICE_ID); + gc->bar0_va = bar0_va; gc->dev = &pdev->dev; xa_init(&gc->irq_contexts); @@ -2173,6 +2175,7 @@ static void mana_gd_shutdown(struct pci_dev *pdev) static const struct pci_device_id mana_id_table[] = { { PCI_DEVICE(PCI_VENDOR_ID_MICROSOFT, MANA_PF_DEVICE_ID) }, + { PCI_DEVICE(PCI_VENDOR_ID_MICROSOFT, MANA_PF2_DEVICE_ID) }, { PCI_DEVICE(PCI_VENDOR_ID_MICROSOFT, MANA_VF_DEVICE_ID) }, { } }; diff --git a/include/net/mana/gdma.h b/include/net/mana/gdma.h index 766f4fb25e266..03e16b07a8bac 100644 --- a/include/net/mana/gdma.h +++ b/include/net/mana/gdma.h @@ -411,6 +411,7 @@ struct gdma_context { u32 test_event_eq_id; bool is_pf; + bool is_pf2; bool in_service; phys_addr_t bar0_pa; @@ -560,6 +561,7 @@ struct gdma_eqe { #define GDMA_SRIOV_REG_CFG_BASE_OFF 0x108 #define MANA_PF_DEVICE_ID 0x00B9 +#define MANA_PF2_DEVICE_ID 0x00C1 #define MANA_VF_DEVICE_ID 0x00BA struct gdma_posted_wqe_info { From bb6a4252985b762f8c4ee440ffc8fcaaac580264 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:18 -0700 Subject: [PATCH 428/464] iommu/arm-smmu-v3: Add arm_smmu_kdump_adopt_strtab() for kdump BugLink: https://bugs.launchpad.net/bugs/2156531 When transitioning to a kdump kernel, the primary kernel might have crashed while endpoint devices were actively bus-mastering DMA. Currently, the SMMU driver aggressively resets the hardware during probe by clearing CR0_SMMUEN and setting the Global Bypass Attribute (GBPA) to ABORT. In a kdump scenario, this aggressive reset is highly destructive: a) If GBPA is set to ABORT, in-flight DMA will be aborted, generating fatal PCIe AER or SErrors that may panic the kdump kernel b) If GBPA is set to BYPASS, in-flight DMA targeting some IOVAs will bypass the SMMU and corrupt the physical memory at those 1:1 mapped IOVAs. To safely absorb in-flight DMAs, a kdump kernel will have to leave SMMUEN=1 intact and avoid modifying STRTAB_BASE, allowing HW to continue translating in-flight DMAs reusing the crashed kernel's page tables until the endpoint device drivers probe and quiesce their respective hardware. However, the ARM SMMUv3 architecture specification states that updating the SMMU_STRTAB_BASE register while SMMUEN == 1 is UNPREDICTABLE or ignored. This leaves a kdump kernel no choice but to adopt the stream table from the crashed kernel. Introduce ARM_SMMU_OPT_KDUMP_ADOPT and adopt functions memremapping all the stream tables extracted from STRTAB_BASE and STRTAB_BASE_CFG. Note that the adoption of the crashed kernel's stream table follows certain strict rules, since the old stream table might be compromised. Thus, apply some basic validations against the values read from the registers. If tests fail, it means the stream table cannot be trusted, so toss it entirely. To avoid OOM due to a potentially corrupted stream table, the memremap for l2 tables is done on the kdump kernel's demand. The new option will be set in a following change. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Suggested-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 254 +++++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + 2 files changed, 252 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index c913fb55de0c7..676689bd4f111 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1807,16 +1807,70 @@ static void arm_smmu_init_initial_stes(struct arm_smmu_ste *strtab, } } +static int arm_smmu_kdump_adopt_l2_strtab(struct arm_smmu_device *smmu, u32 sid, + phys_addr_t base, u32 span, + struct arm_smmu_strtab_l2 **l2table) +{ + struct arm_smmu_strtab_l2 *table; + size_t size; + + /* + * Only a coherent SMMU is supported at this moment. For a non-coherent + * SMMU that wants to support ARM_SMMU_OPT_KDUMP_ADOPT, try MEMREMAP_WC. + */ + if (WARN_ON(!(smmu->features & ARM_SMMU_FEAT_COHERENCY))) + return -EOPNOTSUPP; + + /* + * Retest the span in case the L1 descriptor has been overwritten since + * the adopt. Reject this master's insert; panic or SMMU-disable would + * either lose the vmcore or cascade aborts. Do not try to fix it, as it + * would break all other SIDs in the same bus (PCI case). The corruption + * blast radius is already bounded to that bus range. + */ + if (span != STRTAB_SPLIT + 1) { + dev_err(smmu->dev, + "kdump: L1[%u] span %u changed since adopt (was %u)\n", + arm_smmu_strtab_l1_idx(sid), span, STRTAB_SPLIT + 1); + return -EINVAL; + } + + size = (1UL << (span - 1)) * sizeof(struct arm_smmu_ste); + + table = devm_memremap(smmu->dev, base, size, MEMREMAP_WB); + if (IS_ERR(table)) { + dev_err(smmu->dev, + "kdump: failed to adopt l2 stream table for SID %u\n", + sid); + return PTR_ERR(table); + } + + *l2table = table; + return 0; +} + static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) { dma_addr_t l2ptr_dma; struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg; struct arm_smmu_strtab_l2 **l2table; + u32 l1_idx = arm_smmu_strtab_l1_idx(sid); - l2table = &cfg->l2.l2ptrs[arm_smmu_strtab_l1_idx(sid)]; + l2table = &cfg->l2.l2ptrs[l1_idx]; if (*l2table) return 0; + /* Deferred adoption of the crashed kernel's L2 table */ + if (smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) { + u64 l2ptr = le64_to_cpu(cfg->l2.l1tab[l1_idx].l2ptr); + phys_addr_t base = l2ptr & STRTAB_L1_DESC_L2PTR_MASK; + u32 span = FIELD_GET(STRTAB_L1_DESC_SPAN, l2ptr); + + if (span && base) + return arm_smmu_kdump_adopt_l2_strtab(smmu, sid, base, + span, l2table); + } + *l2table = dmam_alloc_coherent(smmu->dev, sizeof(**l2table), &l2ptr_dma, GFP_KERNEL); if (!*l2table) { @@ -1828,8 +1882,7 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) arm_smmu_init_initial_stes((*l2table)->stes, ARRAY_SIZE((*l2table)->stes)); - arm_smmu_write_strtab_l1_desc(&cfg->l2.l1tab[arm_smmu_strtab_l1_idx(sid)], - l2ptr_dma); + arm_smmu_write_strtab_l1_desc(&cfg->l2.l1tab[l1_idx], l2ptr_dma); return 0; } @@ -4071,10 +4124,204 @@ static int arm_smmu_init_strtab_linear(struct arm_smmu_device *smmu) return 0; } +static int arm_smmu_kdump_adopt_strtab_2lvl(struct arm_smmu_device *smmu, + u32 cfg_reg, phys_addr_t base) +{ + u32 log2size = FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, cfg_reg); + u32 split = FIELD_GET(STRTAB_BASE_CFG_SPLIT, cfg_reg); + struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg; + u32 num_l1_ents; + size_t size; + int i; + + /* + * Only a coherent SMMU is supported at this moment. For a non-coherent + * SMMU that wants to support ARM_SMMU_OPT_KDUMP_ADOPT, try MEMREMAP_WC. + */ + if (WARN_ON(!(smmu->features & ARM_SMMU_FEAT_COHERENCY))) + return -EOPNOTSUPP; + + if (log2size < split || log2size > smmu->sid_bits) { + dev_err(smmu->dev, "kdump: log2size %u out of range [%u, %u]\n", + log2size, split, smmu->sid_bits); + return -EINVAL; + } + if (split != STRTAB_SPLIT) { + dev_err(smmu->dev, + "kdump: unsupported STRTAB_SPLIT %u (expected %u)\n", + split, STRTAB_SPLIT); + return -EINVAL; + } + + num_l1_ents = 1U << (log2size - split); + if (num_l1_ents > STRTAB_MAX_L1_ENTRIES) { + dev_err(smmu->dev, "kdump: l1 entries %u exceeds max %u\n", + num_l1_ents, STRTAB_MAX_L1_ENTRIES); + return -EINVAL; + } + + cfg->l2.num_l1_ents = num_l1_ents; + + size = num_l1_ents * sizeof(struct arm_smmu_strtab_l1); + cfg->l2.l1tab = memremap(base, size, MEMREMAP_WB); + if (!cfg->l2.l1tab) + return -ENOMEM; + + cfg->l2.l2ptrs = + kcalloc(num_l1_ents, sizeof(*cfg->l2.l2ptrs), GFP_KERNEL); + if (!cfg->l2.l2ptrs) + return -ENOMEM; + + for (i = 0; i < num_l1_ents; i++) { + u64 l2ptr = le64_to_cpu(cfg->l2.l1tab[i].l2ptr); + phys_addr_t l2_base = l2ptr & STRTAB_L1_DESC_L2PTR_MASK; + u32 span = FIELD_GET(STRTAB_L1_DESC_SPAN, l2ptr); + + if (!span || !l2_base) + continue; + + if (span != STRTAB_SPLIT + 1) { + dev_err(smmu->dev, + "kdump: L1[%u] unsupported span %u (vs %u)\n", + i, span, STRTAB_SPLIT + 1); + return -EINVAL; + } + + /* + * If the crashed kernel's l1 descriptors are deeply corrupted, + * blindly memremapping every l2 table here could lead to OOM. + * + * Defer the l2 memremap to arm_smmu_init_l2_strtab(), so peak + * memory is bounded by the kdump kernel's actual demand. + */ + } + + return 0; +} + +static int arm_smmu_kdump_adopt_strtab_linear(struct arm_smmu_device *smmu, + u32 cfg_reg, phys_addr_t base) +{ + u32 log2size = FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, cfg_reg); + struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg; + unsigned int max_log2size; + size_t size; + + /* + * Only a coherent SMMU is supported at this moment. For a non-coherent + * SMMU that wants to support ARM_SMMU_OPT_KDUMP_ADOPT, try MEMREMAP_WC. + */ + if (WARN_ON(!(smmu->features & ARM_SMMU_FEAT_COHERENCY))) + return -EOPNOTSUPP; + + /* Cap the size at what the kdump kernel itself would have allocated */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) + max_log2size = + ilog2(STRTAB_MAX_L1_ENTRIES * STRTAB_NUM_L2_STES); + else + max_log2size = smmu->sid_bits; + + /* cfg->linear.num_ents is unsigned int, so cap log2size at 31 */ + max_log2size = min(max_log2size, 31U); + if (log2size > max_log2size) { + dev_err(smmu->dev, "kdump: unsupported log2size %u (> %u)\n", + log2size, max_log2size); + return -EINVAL; + } + + /* + * We might end up with a num_ents != sid_bits, which is fine. In the + * ARM_SMMU_OPT_KDUMP_ADOPT case, arm_smmu_write_strtab() is bypassed. + */ + cfg->linear.num_ents = 1U << log2size; + + size = cfg->linear.num_ents * sizeof(struct arm_smmu_ste); + cfg->linear.table = memremap(base, size, MEMREMAP_WB); + if (!cfg->linear.table) + return -ENOMEM; + return 0; +} + +static void arm_smmu_kdump_adopt_cleanup(void *data) +{ + struct arm_smmu_device *smmu = data; + u32 cfg_reg = readl_relaxed(smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg; + u32 fmt = FIELD_GET(STRTAB_BASE_CFG_FMT, cfg_reg); + + if (fmt == STRTAB_BASE_CFG_FMT_2LVL) { + kfree(cfg->l2.l2ptrs); + if (cfg->l2.l1tab) + memunmap(cfg->l2.l1tab); + } else if (fmt == STRTAB_BASE_CFG_FMT_LINEAR) { + if (cfg->linear.table) + memunmap(cfg->linear.table); + } +} + +static int arm_smmu_kdump_adopt_strtab(struct arm_smmu_device *smmu) +{ + u32 cfg_reg = readl_relaxed(smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + u64 base_reg = readq_relaxed(smmu->base + ARM_SMMU_STRTAB_BASE); + u32 fmt = FIELD_GET(STRTAB_BASE_CFG_FMT, cfg_reg); + phys_addr_t base = base_reg & STRTAB_BASE_ADDR_MASK; + int ret; + + dev_info(smmu->dev, "kdump: adopting crashed kernel's stream table\n"); + + if (fmt == STRTAB_BASE_CFG_FMT_2LVL) { + /* + * Both kernels run on the same hardware, so it's impossible for + * kdump kernel to see the support for linear stream table only. + */ + if (WARN_ON(!(smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB))) + ret = -EINVAL; + else + ret = arm_smmu_kdump_adopt_strtab_2lvl(smmu, cfg_reg, + base); + } else if (fmt == STRTAB_BASE_CFG_FMT_LINEAR) { + /* + * In case that the old kernel for some reason used the linear + * format, enforce the same format to match the adopted table. + */ + ret = arm_smmu_kdump_adopt_strtab_linear(smmu, cfg_reg, base); + if (!ret) + smmu->features &= ~ARM_SMMU_FEAT_2_LVL_STRTAB; + } else { + dev_err(smmu->dev, "kdump: invalid STRTAB format %u\n", fmt); + ret = -EINVAL; + } + + if (ret) { + arm_smmu_kdump_adopt_cleanup(smmu); + goto err; + } + + ret = devm_add_action_or_reset(smmu->dev, arm_smmu_kdump_adopt_cleanup, + smmu); + /* devm_add_action_or_reset ran the cleanup upon failure */ + if (ret) { + dev_warn(smmu->dev, "kdump: failed to set up cleanup action\n"); + goto err; + } + + return 0; + +err: + dev_warn(smmu->dev, "kdump: falling back to full reset\n"); + memset(&smmu->strtab_cfg, 0, sizeof(smmu->strtab_cfg)); + smmu->options &= ~ARM_SMMU_OPT_KDUMP_ADOPT; + return ret; +} + static int arm_smmu_init_strtab(struct arm_smmu_device *smmu) { int ret; + if ((smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) && + !arm_smmu_kdump_adopt_strtab(smmu)) + goto out; + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) ret = arm_smmu_init_strtab_2lvl(smmu); else @@ -4082,6 +4329,7 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu) if (ret) return ret; +out: ida_init(&smmu->vmid_map); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 1d9306232e0fe..56afb4b8546d4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -785,6 +785,7 @@ struct arm_smmu_device { #define ARM_SMMU_OPT_MSIPOLL (1 << 2) #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3) #define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4) +#define ARM_SMMU_OPT_KDUMP_ADOPT (1 << 5) u32 options; struct arm_smmu_cmdq cmdq; From 7e7e653a99453c24921a0f1c3eb19029a7f15de3 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:19 -0700 Subject: [PATCH 429/464] iommu/arm-smmu-v3: Implement is_attach_deferred() for kdump BugLink: https://bugs.launchpad.net/bugs/2156531 Though the kdump kernel adopts the crashed kernel's stream table, the iommu core will still try to attach each probed device to a default domain, which overwrites the adopted STE and breaks in-flight DMA from that device. Implement an is_attach_deferred() callback to prevent this. For each device that has STE.V=1 and STE.Cfg!=Abort in the adopted table, defer the default domain attachment, until the device driver explicitly requests it. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) [jamien: Resolve context conflict around arm_smmu_remove_master() due to the different surrounding arm-smmu-v3 code in this tree.] Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 24 +++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 676689bd4f111..032e60c23074f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3664,6 +3664,29 @@ static void arm_smmu_remove_master(struct arm_smmu_master *master) kfree(master->streams); } +static bool arm_smmu_is_attach_deferred(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct arm_smmu_device *smmu = master->smmu; + int i; + + if (!(smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT)) + return false; + + for (i = 0; i < master->num_streams; i++) { + struct arm_smmu_ste *ste = + arm_smmu_get_step_for_sid(smmu, master->streams[i].id); + u64 ent0 = le64_to_cpu(ste->data[0]); + + /* Defer only when there might be in-flight DMAs */ + if ((ent0 & STRTAB_STE_0_V) && + FIELD_GET(STRTAB_STE_0_CFG, ent0) != STRTAB_STE_0_CFG_ABORT) + return true; + } + + return false; +} + static int arm_smmu_master_prepare_ats(struct arm_smmu_master *master) { bool s1p = master->smmu->features & ARM_SMMU_FEAT_TRANS_S1; @@ -3943,6 +3966,7 @@ static const struct iommu_ops arm_smmu_ops = { .hw_info = arm_smmu_hw_info, .domain_alloc_sva = arm_smmu_sva_domain_alloc, .domain_alloc_paging_flags = arm_smmu_domain_alloc_paging_flags, + .is_attach_deferred = arm_smmu_is_attach_deferred, .probe_device = arm_smmu_probe_device, .release_device = arm_smmu_release_device, .device_group = arm_smmu_device_group, From 99f106006929addc506839035bd4a861443c1c3c Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:20 -0700 Subject: [PATCH 430/464] iommu/arm-smmu-v3: Do not enable EVTQ/PRIQ interrupts in kdump kernel BugLink: https://bugs.launchpad.net/bugs/2156531 In kdump cases, the crashed kernel's CDs and page tables can be corrupted, which could trigger event spamming. Also, we cannot serve page requests. Skip the IRQ setup for EVTQ/PRIQ in arm_smmu_setup_irqs(). Skip their IRQ handler registration in unique-IRQ and combined-IRQ cases. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Reviewed-by: Kevin Tian Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 58 ++++++++++++++------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 032e60c23074f..df41d18b6ee21 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2231,7 +2231,11 @@ static irqreturn_t arm_smmu_combined_irq_thread(int irq, void *dev) static irqreturn_t arm_smmu_combined_irq_handler(int irq, void *dev) { - arm_smmu_gerror_handler(irq, dev); + irqreturn_t ret = arm_smmu_gerror_handler(irq, dev); + + /* In kdump, EVTQ/PRIQ are disabled and there is no thread to wake */ + if (is_kdump_kernel()) + return ret; return IRQ_WAKE_THREAD; } @@ -4478,6 +4482,21 @@ static void arm_smmu_setup_unique_irqs(struct arm_smmu_device *smmu) arm_smmu_setup_msis(smmu); /* Request interrupt lines */ + irq = smmu->gerr_irq; + if (irq) { + ret = devm_request_irq(smmu->dev, irq, arm_smmu_gerror_handler, + 0, "arm-smmu-v3-gerror", smmu); + if (ret < 0) + dev_warn(smmu->dev, "failed to enable gerror irq\n"); + } else { + dev_warn(smmu->dev, + "no gerr irq - errors will not be reported!\n"); + } + + /* No EVTQ/PRIQ interrupts in kdump -- queues are disabled */ + if (is_kdump_kernel()) + return; + irq = smmu->evtq.q.irq; if (irq) { ret = devm_request_threaded_irq(smmu->dev, irq, NULL, @@ -4490,16 +4509,6 @@ static void arm_smmu_setup_unique_irqs(struct arm_smmu_device *smmu) dev_warn(smmu->dev, "no evtq irq - events will not be reported!\n"); } - irq = smmu->gerr_irq; - if (irq) { - ret = devm_request_irq(smmu->dev, irq, arm_smmu_gerror_handler, - 0, "arm-smmu-v3-gerror", smmu); - if (ret < 0) - dev_warn(smmu->dev, "failed to enable gerror irq\n"); - } else { - dev_warn(smmu->dev, "no gerr irq - errors will not be reported!\n"); - } - if (smmu->features & ARM_SMMU_FEAT_PRI) { irq = smmu->priq.q.irq; if (irq) { @@ -4520,7 +4529,7 @@ static void arm_smmu_setup_unique_irqs(struct arm_smmu_device *smmu) static int arm_smmu_setup_irqs(struct arm_smmu_device *smmu) { int ret, irq; - u32 irqen_flags = IRQ_CTRL_EVTQ_IRQEN | IRQ_CTRL_GERROR_IRQEN; + u32 irqen_flags = IRQ_CTRL_GERROR_IRQEN; /* Disable IRQs first */ ret = arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_IRQ_CTRL, @@ -4535,19 +4544,30 @@ static int arm_smmu_setup_irqs(struct arm_smmu_device *smmu) /* * Cavium ThunderX2 implementation doesn't support unique irq * lines. Use a single irq line for all the SMMUv3 interrupts. + * + * In kdump, EVTQ/PRIQ are disabled, so no threaded handling. */ - ret = devm_request_threaded_irq(smmu->dev, irq, - arm_smmu_combined_irq_handler, - arm_smmu_combined_irq_thread, - IRQF_ONESHOT, - "arm-smmu-v3-combined-irq", smmu); + if (is_kdump_kernel()) + ret = devm_request_irq(smmu->dev, irq, + arm_smmu_combined_irq_handler, 0, + "arm-smmu-v3-combined-irq", + smmu); + else + ret = devm_request_threaded_irq( + smmu->dev, irq, arm_smmu_combined_irq_handler, + arm_smmu_combined_irq_thread, IRQF_ONESHOT, + "arm-smmu-v3-combined-irq", smmu); if (ret < 0) dev_warn(smmu->dev, "failed to enable combined irq\n"); } else arm_smmu_setup_unique_irqs(smmu); - if (smmu->features & ARM_SMMU_FEAT_PRI) - irqen_flags |= IRQ_CTRL_PRIQ_IRQEN; + /* No EVTQ/PRIQ IRQ generation in kdump -- queues are disabled */ + if (!is_kdump_kernel()) { + irqen_flags |= IRQ_CTRL_EVTQ_IRQEN; + if (smmu->features & ARM_SMMU_FEAT_PRI) + irqen_flags |= IRQ_CTRL_PRIQ_IRQEN; + } /* Enable interrupt generation on the SMMU */ ret = arm_smmu_write_reg_sync(smmu, irqen_flags, From 8e05f7a103203d8d610e2801faa114b1196654d1 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:21 -0700 Subject: [PATCH 431/464] iommu/arm-smmu-v3: Skip EVTQ/PRIQ setup in kdump kernel BugLink: https://bugs.launchpad.net/bugs/2156531 In kdump cases, the crashed kernel's CDs and page tables can be corrupted, which could trigger event spamming. Also, we cannot serve page requests. Skip the EVTQ/PRIQ setup entirely rather than enabling then disabling them. Also add some inline comments explaining that. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Suggested-by: Kevin Tian Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 43 +++++++++++++-------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index df41d18b6ee21..2bdd697a17cdc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4676,21 +4676,35 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu) cmd.opcode = CMDQ_OP_TLBI_NSNH_ALL; arm_smmu_cmdq_issue_cmd_with_sync(smmu, &cmd); - /* Event queue */ - writeq_relaxed(smmu->evtq.q.q_base, smmu->base + ARM_SMMU_EVTQ_BASE); - writel_relaxed(smmu->evtq.q.llq.prod, smmu->page1 + ARM_SMMU_EVTQ_PROD); - writel_relaxed(smmu->evtq.q.llq.cons, smmu->page1 + ARM_SMMU_EVTQ_CONS); - - enables |= CR0_EVTQEN; - ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, - ARM_SMMU_CR0ACK); - if (ret) { - dev_err(smmu->dev, "failed to enable event queue\n"); - return ret; + /* + * Event queue + * + * Do not enable in a kdump case, as the crashed kernel's CDs and page + * tables might be corrupted, triggering event spamming. + */ + if (!is_kdump_kernel()) { + writeq_relaxed(smmu->evtq.q.q_base, + smmu->base + ARM_SMMU_EVTQ_BASE); + writel_relaxed(smmu->evtq.q.llq.prod, + smmu->page1 + ARM_SMMU_EVTQ_PROD); + writel_relaxed(smmu->evtq.q.llq.cons, + smmu->page1 + ARM_SMMU_EVTQ_CONS); + + enables |= CR0_EVTQEN; + ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, + ARM_SMMU_CR0ACK); + if (ret) { + dev_err(smmu->dev, "failed to enable event queue\n"); + return ret; + } } - /* PRI queue */ - if (smmu->features & ARM_SMMU_FEAT_PRI) { + /* + * PRI queue + * + * Do not enable in a kdump case, as we cannot serve page requests. + */ + if (!is_kdump_kernel() && (smmu->features & ARM_SMMU_FEAT_PRI)) { writeq_relaxed(smmu->priq.q.q_base, smmu->base + ARM_SMMU_PRIQ_BASE); writel_relaxed(smmu->priq.q.llq.prod, @@ -4723,9 +4737,6 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu) return ret; } - if (is_kdump_kernel()) - enables &= ~(CR0_EVTQEN | CR0_PRIQEN); - /* Enable the SMMU interface */ enables |= CR0_SMMUEN; ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, From 395bc64d13eaf77c27496c94ad6e951de67e4f20 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:22 -0700 Subject: [PATCH 432/464] iommu/arm-smmu-v3: Retain CR0_SMMUEN during kdump device reset BugLink: https://bugs.launchpad.net/bugs/2156531 When ARM_SMMU_OPT_KDUMP_ADOPT is detected, do not disable SMMUEN and skip the CR1/CR2/STRTAB_BASE update sequence in arm_smmu_device_reset(). Those register writes are all CONSTRAINED UNPREDICTABLE while CR0_SMMUEN==1, so leaving them intact lets in-flight DMAs continue to be translated by the adopted stream table. Initialize 'enables' to 0 so it can carry CR0_SMMUEN in kdump case. Then, preserve that when enabling the command queue. Clear latched gerror bits if necessary. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Signed-off-by: Nicolin Chen Reviewed-by: Kevin Tian (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 47 +++++++++++++++++++-- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 2bdd697a17cdc..3c0faaaf36f1a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4616,11 +4616,28 @@ static void arm_smmu_write_strtab(struct arm_smmu_device *smmu) static int arm_smmu_device_reset(struct arm_smmu_device *smmu) { int ret; - u32 reg, enables; + u32 reg, enables = 0; struct arm_smmu_cmdq_ent cmd; - /* Clear CR0 and sync (disables SMMU and queue processing) */ reg = readl_relaxed(smmu->base + ARM_SMMU_CR0); + + /* + * In a kdump case (set when CR0_SMMUEN=1 and !GERROR_SFM_ERR), retain + * CR0_SMMUEN to avoid aborting in-flight DMA, and CR0_ATSCHK to carry + * on the ATS-check policy. + * + * According to spec, updating STRTAB_BASE/CR1/CR2 when CR0_SMMUEN=1 is + * CONSTRAINED UNPREDICTABLE. So, skip those register updates and rely + * on the adopted stream table from the crashed kernel. + */ + if (smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) { + dev_info(smmu->dev, + "kdump: retaining SMMUEN for in-flight DMA\n"); + enables = reg & (CR0_SMMUEN | CR0_ATSCHK); + goto reset_queues; + } + + /* Clear CR0 and sync (disables SMMU and queue processing) */ if (reg & CR0_SMMUEN) { dev_warn(smmu->dev, "SMMU currently enabled! Resetting...\n"); arm_smmu_update_gbpa(smmu, GBPA_ABORT, 0); @@ -4650,12 +4667,36 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu) /* Stream table */ arm_smmu_write_strtab(smmu); +reset_queues: + if (smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) { + /* Disable queues since arm_smmu_device_disable() was skipped */ + ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, + ARM_SMMU_CR0ACK); + if (ret) { + dev_err(smmu->dev, "failed to disable queues\n"); + return ret; + } + } + + /* + * GERROR bits are latched. Read after queue disabling so that unhandled + * errors would be visible. Ack everything prior to re-enabling the CMDQ + * as a stale CMDQ_ERR would halt the CMDQ and new command will timeout. + */ + if (is_kdump_kernel()) { + u32 gerror = readl_relaxed(smmu->base + ARM_SMMU_GERROR); + u32 gerrorn = readl_relaxed(smmu->base + ARM_SMMU_GERRORN); + + if ((gerror ^ gerrorn) & GERROR_ERR_MASK) + writel(gerror, smmu->base + ARM_SMMU_GERRORN); + } + /* Command queue */ writeq_relaxed(smmu->cmdq.q.q_base, smmu->base + ARM_SMMU_CMDQ_BASE); writel_relaxed(smmu->cmdq.q.llq.prod, smmu->base + ARM_SMMU_CMDQ_PROD); writel_relaxed(smmu->cmdq.q.llq.cons, smmu->base + ARM_SMMU_CMDQ_CONS); - enables = CR0_CMDQEN; + enables |= CR0_CMDQEN; ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0, ARM_SMMU_CR0ACK); if (ret) { From cbe84f051d0618e7cdb9a7cca8578dcf391116fa Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:23 -0700 Subject: [PATCH 433/464] iommu/arm-smmu-v3: Skip RMR bypass for kdump adoption BugLink: https://bugs.launchpad.net/bugs/2156531 RMR bypass STEs are installed during SMMUv3 probe for StreamIDs listed by IORT RMR nodes. A normal boot switches the driver to a fresh stream table whose initial STEs abort, so those RMR SIDs need bypass entries before it becomes live. This preserves firmware/guest-owned traffic, including vSMMU guest MSI cases built around RMR-described SIDs. ARM_SMMU_OPT_KDUMP_ADOPT is the opposite case: the driver keeps SMMUEN set and adopts the crashed kernel's stream table, so RMR SIDs already have the only translation state known to be safe for active in-flight DMA. Replacing an adopted STE with bypass can turn translated DMA into physical DMA, then point it at the wrong memory. arm_smmu_make_bypass_ste() also rewrites the STE in place after clearing it first. While the table is live, a concurrent hardware STE fetch can observe V=0 or mixed old/new state. Leaving the adopted STE unmodified keeps the kdump kernel using the crashed kernel's translation. That gives the endpoint driver a chance to probe and quiesce the device. If the old STE was already abort or invalid, installing bypass would create new DMA permission; leaving it alone is a safer failure mode. Later domain setup still gets the RMR direct mappings through the reserved-region path. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Assisted-by: Codex:gpt-5.5 Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 3c0faaaf36f1a..a181ad8675cdf 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -5263,6 +5263,14 @@ static void arm_smmu_rmr_install_bypass_ste(struct arm_smmu_device *smmu) struct list_head rmr_list; struct iommu_resv_region *e; + /* + * Kdump adoption keeps the crashed kernel's table live. Rewriting the + * adopted STE here could expose an in-flight fetch to a transient V=0 + * entry, or change Cfg=translate to Cfg=bypass. Must skip here. + */ + if (smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) + return; + INIT_LIST_HEAD(&rmr_list); iort_get_rmr_sids(dev_fwnode(smmu->dev), &rmr_list); @@ -5279,10 +5287,7 @@ static void arm_smmu_rmr_install_bypass_ste(struct arm_smmu_device *smmu) continue; } - /* - * STE table is not programmed to HW, see - * arm_smmu_initial_bypass_stes() - */ + /* The fresh stream table is not yet live. */ arm_smmu_make_bypass_ste(smmu, arm_smmu_get_step_for_sid(smmu, rmr->sids[i])); } From 0c53b5043509951c54d5014a7179f0e21de96d74 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Wed, 20 May 2026 10:03:24 -0700 Subject: [PATCH 434/464] iommu/arm-smmu-v3: Detect ARM_SMMU_OPT_KDUMP_ADOPT in probe() BugLink: https://bugs.launchpad.net/bugs/2156531 arm_smmu_device_hw_probe() runs before arm_smmu_init_structures(), so it's natural to decide whether the kdump kernel must adopt the crashed kernel's stream table. Given that memremap is used to adopt the old stream table, set this option only on a coherent SMMU. And make sure SMMU isn't in Service Failure Mode. Fixes: b63b3439b856 ("iommu/arm-smmu-v3: Abort all transactions if SMMU is enabled in kdump kernel") Cc: stable@vger.kernel.org # v6.12+ Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Nicolin Chen (backported from https://lore.kernel.org/linux-iommu/cover.1779265413.git.nicolinc@nvidia.com/#t) [jamien: Resolve context conflict around arm_smmu_device_hw_probe() due to the existing MPAM probe helper in this tree.] Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 31 +++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a181ad8675cdf..0332b8c741ed5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4858,6 +4858,33 @@ static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg) hw_features, fw_features); } +static void arm_smmu_device_hw_probe_kdump(struct arm_smmu_device *smmu) +{ + u32 gerror, gerrorn, active; + + /* No adoption if SMMU is disabled (i.e., there is no in-flight DMA) */ + if (!(readl_relaxed(smmu->base + ARM_SMMU_CR0) & CR0_SMMUEN)) + return; + + /* For now, only support a coherent SMMU that works with MEMREMAP_WB */ + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) { + dev_warn(smmu->dev, + "kdump: non-coherent SMMU unsupported; reset to block all DMAs\n"); + return; + } + + gerror = readl_relaxed(smmu->base + ARM_SMMU_GERROR); + gerrorn = readl_relaxed(smmu->base + ARM_SMMU_GERRORN); + active = gerror ^ gerrorn; + if (active & GERROR_SFM_ERR) { + dev_warn(smmu->dev, + "kdump: SMMU in Service Failure Mode, must reset\n"); + return; + } + + smmu->options |= ARM_SMMU_OPT_KDUMP_ADOPT; +} + static void arm_smmu_mpam_register_smmu(struct arm_smmu_device *smmu) { u16 partid_max; @@ -5099,6 +5126,10 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) dev_info(smmu->dev, "oas %lu-bit (features 0x%08x)\n", smmu->oas, smmu->features); + + if (is_kdump_kernel()) + arm_smmu_device_hw_probe_kdump(smmu); + return 0; } From 4cd73baec1d15d3244c2f428ac6a07b00c4719ad Mon Sep 17 00:00:00 2001 From: Jamie Nguyen Date: Thu, 11 Jun 2026 17:47:24 -0700 Subject: [PATCH 435/464] NVIDIA: SAUCE: iommu/arm-smmu-v3: Block kdump MPAM updates BugLink: https://bugs.launchpad.net/bugs/2156531 The kdump adoption backport preserves live STEs until device attach has been re-established. MPAM QoS updates can rewrite STE PARTID/PMG fields before deferred attach is resolved. That can modify an adopted live STE while DMA may still be in flight. Reject MPAM QoS updates while kdump adoption is active and the device still has attach_deferred set. The IOMMU core clears attach_deferred after a successful real attach, so normal MPAM updates are unchanged after the device is attached. Non-kdump boots are unchanged. Signed-off-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Seth Forshee Signed-off-by: Brad Figg --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0332b8c741ed5..be29914bbaad4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -3920,6 +3920,15 @@ static int arm_smmu_group_set_mpam(struct device *dev, u16 partid, return -EIO; smmu = master->smmu; + /* + * Do not rewrite an adopted STE before the deferred attach has + * replaced it. The kdump kernel may still be relying on the crashed + * kernel's live STE while endpoint DMA is being quiesced. + */ + if ((smmu->options & ARM_SMMU_OPT_KDUMP_ADOPT) && + dev->iommu->attach_deferred) + return -EBUSY; + arm_smmu_cmdq_batch_init(smmu, &cmds, &cmd); for (i = 0; i < master->num_streams; i++) { From c44aeff31cd32ab2a1cc6a8c31b8ba4ba7a0ab44 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 17 Jun 2026 08:48:00 -0500 Subject: [PATCH 436/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index fe8d960379aac..880215549563e 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2012.12) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Wed, 17 Jun 2026 08:48:00 -0500 + linux-nvidia-bos (7.0.0-2009.9) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2009.9 -proposed tracker (LP: #2155210) From 15c85bd263aee81f6282de0c982fa5f765f56e9d Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 18 Jun 2026 09:44:34 -0500 Subject: [PATCH 437/464] UBUNTU: [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel-versions (adhoc/d2026.06.15) BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/dkms-versions | 1 + 1 file changed, 1 insertion(+) diff --git a/debian.nvidia-bos/dkms-versions b/debian.nvidia-bos/dkms-versions index ba40c69369282..3f4cd049293c0 100644 --- a/debian.nvidia-bos/dkms-versions +++ b/debian.nvidia-bos/dkms-versions @@ -1,2 +1,3 @@ zfs-linux 2.4.1-1ubuntu5 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=riscv64 arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms off_series=true v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms off_series=true +nvidia-fs 2.29.4-1 modulename=nvidia-fs debpath=pool/universe/n/%package%/nvidia-fs-dkms_%version%_amd64.deb arch=amd64 arch=arm64 rprovides=nvidia-fs-modules rprovides=nvidia-fs-dkms type=standalone From 35448a52bd96cafa8a0074679461c312dff0e525 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 18 Jun 2026 20:18:11 -0500 Subject: [PATCH 438/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2156940 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index a53d8c75304b6..ebcee9aba3809 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2155210 d2026.06.03-1 +2156940 d2026.06.15-1 From 6c822a51c53b8ca6f65f964f1d3393a0c0d809b6 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Thu, 18 Jun 2026 20:24:04 -0500 Subject: [PATCH 439/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2012.12 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 633 +++++++++++++++++++++++++++++++++- debian.nvidia-bos/reconstruct | 2 + 2 files changed, 630 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 880215549563e..d9ed503570840 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,633 @@ -linux-nvidia-bos (7.0.0-2012.12) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2012.12) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2012.12 -proposed tracker (LP: #2156940) - -- Jacob Martin Wed, 17 Jun 2026 08:48:00 -0500 + * Packaging resync (LP: #1786013) + - [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel- + versions (adhoc/d2026.06.15) + + * Backport the arm-smmu-v3 kdump adoption series (LP: #2156531) + - iommu/arm-smmu-v3: Add arm_smmu_kdump_adopt_strtab() for kdump + - iommu/arm-smmu-v3: Implement is_attach_deferred() for kdump + - iommu/arm-smmu-v3: Do not enable EVTQ/PRIQ interrupts in kdump kernel + - iommu/arm-smmu-v3: Skip EVTQ/PRIQ setup in kdump kernel + - iommu/arm-smmu-v3: Retain CR0_SMMUEN during kdump device reset + - iommu/arm-smmu-v3: Skip RMR bypass for kdump adoption + - iommu/arm-smmu-v3: Detect ARM_SMMU_OPT_KDUMP_ADOPT in probe() + - NVIDIA: SAUCE: iommu/arm-smmu-v3: Block kdump MPAM updates + + * Backport mana support for PF device 0x00C1 (LP: #2156821) + - net: mana: Add support for PF device 0x00C1 + + * Backport: fuse: back uncached readdir buffers with pages (LP: #2156632) + - fuse: back uncached readdir buffers with pages + + * Backport: Mitigate TLBI errata on various Arm CPUs (LP: #2156557) // CVE- + Enable ARM64_ERRATUM_4118414 to mitigate 2025-10263 on NVIDIA platforms. + - NVIDIA: [Config] Enable ARM64_ERRATUM_4118414 + + * Backport: Mitigate TLBI errata on various Arm CPUs (LP: #2156557) // + CVE-2025-10263. The existing ARM64_ERRATUM_4118414 handling already uses + - arm64: errata: Mitigate TLBI errata on NVIDIA Olympus CPU + + * Backport: Mitigate TLBI errata on various Arm CPUs (LP: #2156557) + - arm64: cputype: Add C1-Ultra definitions + - arm64: cputype: Add C1-Premium definitions + - arm64: errata: Mitigate TLBI errata on various Arm CPUs + + * PCI: mirror PI7C9X3G606GPC Port 4 BAR0 (LP: #2154457) + - NVIDIA: SAUCE: PCI: quirks: mirror PI7C9X3G606GPC Port 4 BAR0 + + * fs/ntfs3: fix mount failure on 64K page-size kernels (LP: #2155467) + - fs/ntfs3: fix mount failure on 64K page-size kernels + + [ Ubuntu: 7.0.0-27.27 ] + + * resolute/linux: 7.0.0-27.27 -proposed tracker (LP: #2157114) + * Packaging resync (LP: #1786013) + - [Packaging] update annotations scripts + * Ubuntu 26.04 linux kernel has non-functional nova-core GPU driver enabled, + conflicting with nouveau (LP: #2150845) + - [Config] Disable NOVA_CORE + * CVE-2026-46316 + - KVM: arm64: vgic-its: Drop the translation cache reference only for the + erased entry + * CVE-2026-46244 + - netfilter: nft_inner: Fix IPv6 inner_thoff desync + * CVE-2026-46137 + - mptcp: pm: ADD_ADDR rtx: allow ID 0 + - mptcp: pm: ADD_ADDR rtx: fix potential data-race + * CVE-2026-46185 + - smb/client: fix out-of-bounds read in symlink_data() + * CVE-2026-46195 + - smb: client: validate dacloffset before building DACL pointers + * CVE-2026-46289 + - lib/scatterlist: fix length calculations in extract_kvec_to_sg + * CVE-2026-46119 + - libceph: Fix slab-out-of-bounds access in auth message processing + * CVE-2026-46135 + - nvmet-tcp: fix race between ICReq handling and queue teardown + * CVE-2026-46155 + - smb/client: fix out-of-bounds read in smb2_compound_op() + * CVE-2026-46115 + - block: add pgmap check to biovec_phys_mergeable + * CVE-2026-46243 + - smb: client: reject userspace cifs.spnego descriptions + + [ Ubuntu: 7.0.0-26.26 ] + + * resolute/linux: 7.0.0-26.26 -proposed tracker (LP: #2154530) + * Packaging resync (LP: #1786013) + - Revert "UBUNTU: SAUCE: import Huawei ES3000_V2 (2.1.0.23)" + - [Packaging] debian.master/dkms-versions -- remove dkms-versions + (main/2026.05.18) + * Fix mic mute led on a HP EliteBook 6 G2a platform (LP: #2150065) + - ALSA: hda/realtek: Add LED fixup for HP EliteBook 6 G2a Laptops + * ov08x40 module mounted upside down on a certain DELL platforms + (LP: #2146517) + - SAUCE: media: ipu-bridge: Add DMI quirk for new Dell XPS laptops with + upside down sensors + - SAUCE: media: ipu-bridge: Add DMI quirk for Dell 14 laptops with upside + down sensors + * Support additional 2888x1808@30fps 900MHz for OVTI05C1 camera sensor + (LP: #2147409) + - SAUCE: media: ipu-bridge: Add 900MHz for OV05C10 + - SAUCE: platform/x86: int3472: increase handshake delay to 50ms for + OV05C10 + * Support Samsung S5K3J1 sensor for Intel MIPI camera (LP: #2121852) + - SAUCE: media: ipu-bridge: Support s5k3j1 sensor + * [SRU] ASoC: enable rt1320 speaker amp and DMIC on PTL SoundWire platforms + (LP: #2150196) + - ASoC: Intel: soc-acpi-intel-ptl-match: drop rt722 monolithic match + tables + - ASoC: SOF: Intel: Add a is_amp flag to fix the wrong name prefix + - ASoC: sdw_utils: add rt1320 and rt1321 dmic dai in codec_info_list + * powerpc-build in ubuntu_kernel_selftests fails to build due to + uninitialized value (LP: #2129844) + - selftests/powerpc: Suppress -Wmaybe-uninitialized with GCC 15 + * Ubuntu 26.04 linux kernel has non-functional nova-core GPU driver enabled, + conflicting with nouveau (LP: #2150845) + - [Config] Disable DRM_NOVA + * Resolute update: v7.0.6 upstream stable release (LP: #2152558) + - Linux 7.0.6 + - Upstream stable to v7.0.6 + * Resolute update: v7.0.5 upstream stable release (LP: #2152556) + - Linux 7.0.5 + - Upstream stable to v7.0.5 + * Resolute update: v7.0.4 upstream stable release (LP: #2152552) + - ALSA: usb-audio: stop parsing UAC2 rates at MAX_NR_RATES + - ALSA: usb-audio: Avoid false E-MU sample-rate notifications + - ALSA: usb-audio: Fix Audio Advantage Micro II SPDIF switch + - usb: xhci: Make usb_host_endpoint.hcpriv survive endpoint_disable() + - usb: chipidea: otg: not wait vbus drop if use role_switch + - usb: chipidea: core: allow ci_irq_handler() handle both ID and VBUS + change + - ALSA: usb-audio: Evaluate packsize caps at the right place + - LoongArch: Add spectre boundry for syscall dispatch table + - drm/nouveau: fix u32 overflow in pushbuf reloc bounds check + - leds: qcom-lpg: Check for array overflow when selecting the high + resolution + - greybus: gb-beagleplay: bound bootloader receive buffering + - greybus: gb-beagleplay: fix sleep in atomic context in hdlc_tx_frames() + - misc: ibmasm: fix OOB MMIO read in ibmasm_handle_mouse_interrupt() + - ibmasm: fix OOB reads in command_file_write due to missing size checks + - ibmasm: fix heap over-read in ibmasm_send_i2o_message() + - sysfs: attribute_group: Respect is_visible_const() when changing owner + - driver core: Don't let a device probe until it's ready + - device property: Make modifications of fwnode "flags" thread safe + - drm/nouveau: fix nvkm_device leak on aperture removal failure + - rust: dma: remove DMA_ATTR_NO_KERNEL_MAPPING from public attrs + - kbuild: rust: allow `clippy::uninlined_format_args` + - fs: afs: revert mmap_prepare() change + - firmware: google: framebuffer: Do not mark framebuffer as busy + - lib: test_hmm: evict device pages on file close to avoid use-after-free + - arm64/mm: Enable batched TLB flush in unmap_hotplug_range() + - arm64: mm: Fix rodata=full block mapping support for realm guests + - mm: migrate: requeue destination folio on deferred split queue + - mm: prevent droppable mappings from being locked + - mm: fix deferred split queue races during migration + - ocfs2: split transactions in dio completion to avoid credit exhaustion + - Input: edt-ft5x06 - fix use-after-free in debugfs teardown + - zram: do not forget to endio for partial discard requests + - wifi: rtw88: check for PCI upstream bridge existence + - wifi: mwifiex: fix use-after-free in mwifiex_adapter_cleanup() + - vfio: selftests: Fix VLA initialisation in vfio_pci_irq_set() + - vfio/xe: Add a missing vfio_pci_core_release_dev() + - vfio/virtio: Convert list_lock from spinlock to mutex + - vfio/cdx: Serialize VFIO_DEVICE_SET_IRQS with a per-device mutex + - vfio/cdx: Fix NULL pointer dereference in interrupt trigger path + - um: drivers: call kernel_strrchr() explicitly in cow_user.c + - thermal: core: Fix thermal zone governor cleanup issues + - spi: imx: fix use-after-free on unbind + - spi: ch341: fix memory leaks on probe failures + - crypto: algif_aead - snapshot IV for async AEAD requests + - crypto: pcrypt - Fix handling of MAY_BACKLOG requests + - dt-bindings: display: ti, am65x-dss: Fix AM62L DSS reg and clock + constraints + - of: unittest: fix use-after-free in of_unittest_changeset() + - of: unittest: fix use-after-free in testdrv_probe() + - hwmon: (powerz) Fix missing usb_kill_urb() on signal interrupt + - EDAC/versalnet: Fix device_node leak in mc_probe() + - PCI: imx6: Skip waiting for L2/L3 Ready on i.MX6SX + - media: amphion: Fix race between m2m job_abort and device_run + - ALSA: control: Validate buf_len before strnlen() in + snd_ctl_elem_init_enum_names() + - net: caif: clear client service pointer on teardown + - net: strparser: fix skb_head leak in strp_abort_strp() + - media: mtk-jpeg: fix use-after-free in release path due to uncancelled + work + - crypto: atmel-sha204a - Fix OTP sysfs read and error handling + - PCI: endpoint: pci-epf-ntb: Remove duplicate resource teardown + - Revert "ALSA: usb: Increase volume range that triggers a warning" + - phy: qcom: m31-eusb2: clear PLL_EN during init + - PCI: epf-mhi: Return 0, not remaining timeout, when eDMA ops complete + - lib/ts_kmp: fix integer overflow in pattern length calculation + - media: i2c: imx219: Check return value of devm_gpiod_get_optional() in + imx219_probe() + - net: qrtr: ns: Fix use-after-free in driver remove() + - ext2: reject inodes with zero i_nlink and valid mode in ext2_iget() + - mm/zsmalloc: copy KMSAN metadata in zs_page_migrate() + - ALSA: aoa: i2sbus: clear stale prepared state + - ALSA: aoa: i2sbus: fix OF node lifetime handling + - ALSA: aoa: Skip devices with no codecs in i2sbus_resume() + - ALSA: ctxfi: Add fallback to default RSR for S/PDIF + - ALSA: seq_oss: return full count for successful SEQ_FULLSIZE writes + - erofs: fix the out-of-bounds nameoff handling for trailing dirents + - ipmi:ssif: Clean up kthread on errors + - jbd2: fix deadlock in jbd2_journal_cancel_revoke() + - KVM: selftests: Fix reserved value WRMSR testcase for multi-feature MSRs + - md/raid10: fix deadlock with check operation and nowait requests + - media: rc: igorplugusb: heed coherency rules + - media: rockchip: rkcif: fix off by one bugs + - media: rockchip: rkcif: comply with minimum number of buffers + requirement + - mfd: stpmic1: Attempt system shutdown twice in case PMIC is confused + - mm/alloc_tag: clear codetag for pages allocated before page_ext + initialization + - mm/damon/core: fix damon_call() vs kdamond_fn() exit race + - mm/damon/core: fix damos_walk() vs kdamond_fn() exit race + - mm/hugetlb: fix early boot crash on parameters without '=' separator + - mtd: docg3: fix use-after-free in docg3_release() + - nvme-pci: add NVME_QUIRK_DISABLE_WRITE_ZEROES for Kingston OM3SGP4 + - nvme: respect NVME_QUIRK_DISABLE_WRITE_ZEROES when wzsl is set + - parisc: _llseek syscall is only available for 32-bit userspace + - parisc: Drop ip_fast_csum() inline assembly implementation + - PCI: cadence: Use cdns_pcie_read_sz() for byte or word read access + - PCI: imx6: Fix reference clock source selection for i.MX95 + - perf annotate: Use jump__delete when freeing LoongArch jumps + - RDMA/mana_ib: Disable RX steering on RSS QP destroy + - remoteproc: xlnx: Only access buffer information if IPI is buffered + - reset: rzv2h-usb2phy: Keep PHY clock enabled for entire device lifetime + - sched: Use u64 for bandwidth ratio calculations + - selftests/mqueue: Fix incorrectly named file + - landlock: Fix LOG_SUBDOMAINS_OFF inheritance across fork() + - landlock: Allow TSYNC with LOG_SUBDOMAINS_OFF and fd=-1 + - selftests/landlock: Drain stale audit records on init + - selftests/landlock: Fix format warning for __u64 in net_test + - selftests/landlock: Fix snprintf truncation checks in audit helpers + - selftests/landlock: Skip stale records in audit_match_record() + - rbd: fix null-ptr-deref when device_add_disk() fails + - mm/zone_device: do not touch device folio after calling ->folio_free() + - block: fix zone write plugs refcount handling in + disk_zone_wplug_schedule_bio_work() + - io_uring/zcrx: return back two step unregistration + - io_uring/timeout: check unused sqe fields + - block: relax pgmap check in bio_add_page for compatible zone device + pages + - iio: adc: ti-ads7950: use iio_push_to_buffers_with_ts_unaligned() + - io_uring/register: fix ring resizing with mixed/large SQEs/CQEs + - io_uring/zcrx: fix user_struct uaf + - io_uring/poll: fix signed comparison in io_poll_get_ownership() + - io_uring/poll: ensure EPOLL_ONESHOT is propagated for EPOLL_URING_WAKE + - module.lds,codetag: force 0 sh_addr for sections + - module.lds.S: Fix modules on 32-bit parisc architecture + - ALSA: core: Fix potential data race at fasync handling + - ALSA: caiaq: Fix control_put() result and cache rollback + - ALSA: caiaq: Handle probe errors properly + - ALSA: 6fire: Fix input volume change detection + - ALSA: hda/realtek - Add mute LED support for HP Victus 15-fa2xxx + - ALSA: pcmtest: fix reference leak on failed device registration + - ALSA: pcmtest: Fix resource leaks in module init error paths + - iio: adc: ad7768-1: fix one-shot mode data acquisition + - iio: adc: ad7768-1: remove switch to one-shot mode + - rxrpc: Fix memory leaks in rxkad_verify_response() + - rxrpc: Fix rxkad crypto unalignment handling + - rxrpc: Fix error handling in rxgk_extract_token() + - rxrpc: Fix re-decryption of RESPONSE packets + - EDAC/versalnet: Fix memory leak in remove and probe error paths + - tools/accounting: handle truncated taskstats netlink messages + - net: txgbe: fix RTNL assertion warning when remove module + - arm64: dts: marvell: uDPU: add ethernet aliases + - net: qrtr: ns: Limit the maximum server registration per node + - net: qrtr: ns: Limit the maximum number of lookups + - net: qrtr: ns: Free the node during ctrl_cmd_bye() + - net: qrtr: ns: Limit the total number of nodes + - net: rds: fix MR cleanup on copy error + - net: txgbe: fix firmware version check + - net/smc: avoid early lgr access in smc_clc_wait_msg + - net: ks8851: Reinstate disabling of BHs around IRQ handler + - net: bridge: use a stable FDB dst snapshot in RCU readers + - netconsole: avoid out-of-bounds access on empty string in trim_newline() + - net: mctp: fix don't require received header reserved bits to be zero + - net: ks8851: Avoid excess softirq scheduling + - drm/arcpgu: fix device node leak + - slub: fix data loss and overflow in krealloc() + - tracing/fprobe: Reject registration of a registered fprobe before init + - RDMA/rxe: Validate pad and ICRC before payload_size() in rxe_rcv + - printf: Compile the kunit test with DISABLE_BRANCH_PROFILING + DISABLE_BRANCH_PROFILING + - ipv4: icmp: validate reply type before using icmp_pointers + - libceph: Prevent potential null-ptr-deref in ceph_handle_auth_reply() + - spi: fix resource leaks on device setup failure + - extract-cert: Wrap key_pass with '#ifdef USE_PKCS11_ENGINE' + - tpm: avoid -Wunused-but-set-variable + - LoongArch: Make arch_irq_work_has_interrupt() true only if IPI HW exist + - LoongArch: Show CPU vulnerabilites correctly + - fbdev: defio: Disconnect deferred I/O from the lifetime of struct + fb_info + - power: supply: axp288_charger: Do not cancel work before initializing it + - hwmon: (isl28022) Fix integer overflow in power calculation on 32-bit + - hwmon: (powerz) Avoid cacheline sharing for DMA buffer + - media: rzv2h-ivc: Revise default VBLANK formula + - media: rzv2h-ivc: Fix AXIRX_VBLANK register write + - fs: prepare for adding LSM blob to backing_file + - lsm: add backing_file LSM hooks + - selinux: fix overlayfs mmap() and mprotect() access checks + - hwmon: (pt5161l) Fix bugs in pt5161l_read_block_data() + - randomize_kstack: Maintain kstack_offset per task + - mmc: block: use single block write in retry + - mmc: sdhci-of-dwcmshc: Disable clock before DLL configuration + - arm64: dts: ti: am62-verdin: Enable pullup for eMMC data pins + - crypto: qat - fix IRQ cleanup on 6xxx probe failure + - xfs: start gc on zonegc_low_space attribute updates + - xfs: fix a resource leak in xfs_alloc_buftarg() + - firmware: google: framebuffer: Do not unregister platform device + - firmware: exynos-acpm: Drop fake 'const' on handle pointer + - crypto: talitos - fix SEC1 32k ahash request limitation + - crypto: talitos - rename first/last to first_desc/last_desc + - pwm: imx-tpm: Count the number of enabled channels in probe + - tpm2-sessions: Fix missing tpm_buf_destroy() in tpm2_read_public() + - tpm: Fix auth session leak in tpm2_get_random() error path + - tpm: Use kfree_sensitive() to free auth session in tpm_dev_release() + - tpm: tpm_tis: add error logging for data transfer + - tpm: tpm_tis: stop transmit if retries are exhausted + - rtc: ntxec: fix OF node reference imbalance + - mm/vmalloc: take vmap_purge_lock in shrinker + - mm/memfd_luo: fix physical address conversion in put_folios cleanup + - mm/mempolicy: fix memory leaks in weighted_interleave_auto_store() + - mm/damon/stat: fix memory leak on damon_start() failure in + damon_stat_start() + - mm/damon/core: validate damos_quota_goal->nid for + node_mem_{used,free}_bp + - mm/damon/core: validate damos_quota_goal->nid for + node_memcg_{used,free}_bp + - mm/damon/core: use time_in_range_open() for damos quota window start + - mm/damon/core: disallow time-quota setting zero esz + - mm/damon/core: disallow non-power of two min_region_sz on damon_start() + - userfaultfd: allow registration of ranges below mmap_min_addr + - LoongArch: KVM: Use CSR_CRMD_PLV in kvm_arch_vcpu_in_kernel() + - KVM: x86: Defer non-architectural deliver of exception payload to + userspace read + - KVM: nSVM: Mark all of vmcb02 dirty when restoring nested state + - KVM: nSVM: Sync NextRIP to cached vmcb12 after VMRUN of L2 + - KVM: nSVM: Sync interrupt shadow to cached vmcb12 after VMRUN of L2 + - KVM: SVM: Inject #UD for INVLPGA if EFER.SVME=0 + - KVM: SVM: Explicitly mark vmcb01 dirty after modifying VMCB intercepts + - KVM: nSVM: Ensure AVIC is inhibited when restoring a vCPU to guest mode + - KVM: nSVM: Always use NextRIP as vmcb02's NextRIP after first L2 VMRUN + - KVM: nSVM: Delay stuffing L2's current RIP into NextRIP until vCPU run + - KVM: nSVM: Use vcpu->arch.cr2 when updating vmcb12 on nested #VMEXIT + - KVM: arm64: Account for RESx bits in __compute_fgt() + - KVM: nSVM: Avoid clearing VMCB_LBR in vmcb12 + - KVM: nSVM: Delay setting soft IRQ RIP tracking fields until vCPU run + - KVM: SVM: Switch svm_copy_lbrs() to a macro + - KVM: SVM: Add missing save/restore handling of LBR MSRs + - KVM: nSVM: Always inject a #GP if mapping VMCB12 fails on nested VMRUN + - KVM: nSVM: Refactor checking LBRV enablement in vmcb12 into a helper + - KVM: nSVM: Refactor writing vmcb12 on nested #VMEXIT as a helper + - KVM: nSVM: Triple fault if restore host CR3 fails on nested #VMEXIT + - KVM: nSVM: Triple fault if mapping VMCB12 fails on nested #VMEXIT + - KVM: nSVM: Clear GIF on nested #VMEXIT(INVALID) + - KVM: nSVM: Clear EVENTINJ fields in vmcb12 on nested #VMEXIT + - KVM: nSVM: Clear tracking of L1->L2 NMI and soft IRQ on nested #VMEXIT + - KVM: nSVM: Add missing consistency check for EFER, CR0, CR4, and CS + - KVM: nSVM: Drop the non-architectural consistency check for NP_ENABLE + - KVM: nSVM: Add missing consistency check for nCR3 validity + - KVM: nSVM: Raise #UD if unhandled VMMCALL isn't intercepted by L1 + - KVM: nSVM: Always intercept VMMCALL when L2 is active + - ARM: 9472/1: fix race condition on PG_dcache_clean in + __sync_icache_dcache() + - ring-buffer: Do not double count the reader_page + - ext4: fix bounds check in check_xattrs() to prevent out-of-bounds access + - ext4: fix missing brelse() in ext4_xattr_inode_dec_ref_all() + - udf: fix partition descriptor append bookkeeping + - mtd: spi-nor: sst: Fix write enable before AAI sequence + - mtd: spinand: winbond: Declare the QE bit on W25NxxJW + - amdgpu/jpeg: fix deepsleep register for jpeg 5_0_0 and 5_0_2 + - md/md-llbitmap: skip reading rdevs that are not in_sync + - md/md-llbitmap: raise barrier before state machine transition + - md/raid5: fix soft lockup in retry_aligned_read() + - md/raid5: validate payload size before accessing journal metadata + - check-uapi: link into shared objects + - mm, swap: speed up hibernation allocation and writeout + - HID: apple: ensure the keyboard backlight is off if suspending + - inotify: fix watch count leak when fsnotify_add_inode_mark_locked() + fails + - x86/cpu: Disable FRED when PTI is forced on + - x86/shstk: Prevent deadlock during shstk sigreturn + - wifi: rtl8xxxu: fix potential use of uninitialized value + - tcp: call sk_data_ready() after listener migration + - taskstats: set version in TGID exit notifications + - mptcp: sync the msk->sndbuf at accept() time + - mfd: core: Preserve OF node when ACPI handle is present + - 9p: fix access mode flags being ORed instead of replaced + - Bluetooth: hci_event: fix potential UAF in SSP passkey handlers + - bus: mhi: host: pci_generic: Switch to async power up to avoid boot + delays + - can: ucan: fix devres lifetime + - crypto: acomp - fix wrong pointer stored by acomp_save_req() + - crypto: arm64/aes - Fix 32-bit aes_mac_update() arg treated as 64-bit + - crypto: atmel-aes - Fix 3-page memory leak in atmel_aes_buff_cleanup + - crypto: atmel-ecc - Release client on allocation failure + - crypto: hisilicon - Fix dma_unmap_single() direction + - crypto: ccree - fix a memory leak in cc_mac_digest() + - crypto: atmel-tdes - fix DMA sync direction + - crypto: atmel-sha204a - Fix error codes in OTP reads + - crypto: atmel-sha204a - Fix potential UAF and memory leak in remove path + - crypto: atmel-sha204a - Fix uninitialized data access on OTP read error + - crypto: nx - fix bounce buffer leaks in nx842_crypto_{alloc,free}_ctx + - crypto: nx - fix context leak in nx842_crypto_free_ctx + - crypto: nx - Fix packed layout in struct nx842_crypto_header + - dm mirror: fix integer overflow in create_dirty_log() + - erofs: fix unsigned underflow in z_erofs_lz4_handle_overlap() + - ceph: fix num_ops off-by-one when crypto allocation fails + - ceph: only d_add() negative dentries when they are unhashed + - gtp: disable BH before calling udp_tunnel_xmit_skb() + - IB/core: Fix zero dmac race in neighbor resolution + - ktest: Fix the month in the name of the failure directory + - NFSv4.1: Apply session size limits on clone path + - ntfs3: add buffer boundary checks to run_unpack() + - ntfs3: fix integer overflow in run_unpack() volume boundary check + - rtmutex: Use waiter::task instead of current in remove_waiter() + - rxgk: Fix potential integer overflow in length check + - sched_ext: Documentation: Clarify ops.dispatch() role in task lifecycle + - scsi: sd: fix missing put_disk() when device_add(&disk_dev) fails + - seg6: fix seg6 lwtunnel output redirect for L2 reduced encap mode + - perf loongarch: Fix build failure with CONFIG_LIBDW_DWARF_UNWIND + - iio: frequency: admv1013: add dev variable + - iio: frequency: admv1013: fix NULL pointer dereference on str + - wifi: mt76: mt792x: describe USB WFSYS reset with a descriptor + - wifi: mt76: mt792x: fix mt7925u USB WFSYS reset handling + - mm: various small mmap_prepare cleanups + - mm: avoid deadlock when holding rmap on mmap_prepare error + - mei: me: use PCI_DEVICE_DATA macro + - mei: me: add nova lake point H DID + - crypto: authencesn - reject short ahash digests during instance creation + - driver core: Add kernel-doc for DEV_FLAG_COUNT enum value + - ALSA: caiaq: Fix potentially leftover ep1_in_urb at error path + - ALSA: caiaq: Don't abort when no input device is available + - ipv6: rpl: reserve mac_len headroom when recompressed SRH grows + - drm/amdgpu: fix zero-size GDS range init on RDNA4 + - drm/imagination: Fix segfault when updating ftrace mask + - ALSA: caiaq: fix usb_dev refcount leak on probe failure + - ALSA: aloop: Fix peer runtime UAF during format-change stop + - vmalloc: fix buffer overflow in vrealloc_node_align() + - mm/page_alloc: return NULL early from alloc_frozen_pages_nolock() in NMI + on UP + - mm/slab: return NULL early from kmalloc_nolock() in NMI on UP + - net: ipv6: fix NOREF dst use in seg6 and rpl lwtunnels + - netfilter: reject zero shift in nft_bitwise + - ipmi:ssif: Remove unnecessary indention + - ipmi:ssif: NULL thread on error + - Linux 7.0.4 + - Upstream stable to v7.0.4 + * Resolute update: v7.0.3 upstream stable release (LP: #2152550) + - Buffer overflow in drivers/xen/sys-hypervisor.c + - xen/privcmd: fix double free via VMA splitting + - Linux 7.0.3 + - Upstream stable to v7.0.3 + * Resolute update: v7.0.2 upstream stable release (LP: #2150553) + - crypto: authencesn - Fix src offset when decrypting in-place + - pwm: th1520: fix `CLIPPY=1` warning + - drm/amdgpu: replace PASID IDR with XArray + - crypto: krb5enc - fix sleepable flag handling in encrypt dispatch + - crypto: krb5enc - fix async decrypt skipping hash verification + - ksmbd: fix use-after-free in __ksmbd_close_fd() via durable scavenger + - ksmbd: validate owner of durable handle on reconnect + - scripts: generate_rust_analyzer.py: define scripts + - scripts/dtc: Remove unused dts_version in dtc-lexer.l + - fs/ntfs3: validate rec->used in journal-replay file record check + - f2fs: fix to do sanity check on dcc->discard_cmd_cnt conditionally + - f2fs: fix UAF caused by decrementing sbi->nr_pages[] in + f2fs_write_end_io() + - f2fs: fix to avoid memory leak in f2fs_rename() + - f2fs: fix to avoid uninit-value access in f2fs_sanity_check_node_footer + - fuse: reject oversized dirents in page cache + - fuse: abort on fatal signal during sync init + - fuse: Check for large folio with SPLICE_F_MOVE + - fuse: quiet down complaints in fuse_conn_limit_write + - fuse: fuse_dev_ioctl_clone() should wait for device file to be + initialized + - ksmbd: require minimum ACE size in smb_check_perm_dacl() + - smb: server: fix active_num_conn leak on transport allocation failure + - smb: client: fix dir separator in SMB1 UNIX mounts + - smb: server: fix max_connections off-by-one in tcp accept path + - smb: client: require a full NFS mode SID before reading mode bits + - smb: client: validate the whole DACL before rewriting it in cifsacl + - smb: client: fix OOB read in smb2_ioctl_query_info QUERY_INFO path + - ksmbd: validate response sizes in ipc_validate_msg() + - ksmbd: validate num_aces and harden ACE walk in smb_inherit_dacl() + - ksmbd: fix out-of-bounds write in smb2_get_ea() EA alignment + - ksmbd: use check_add_overflow() to prevent u16 DACL size overflow + - ksmbd: reset rcount per connection in ksmbd_conn_wait_idle_sess_id() + - writeback: Fix use after free in inode_switch_wbs_work_fn() + - f2fs: fix use-after-free of sbi in f2fs_compress_write_end_io() + - ALSA: usb-audio: apply quirk for MOONDROP JU Jiu + - ALSA: hda/realtek: Add quirk for Legion S7 15IMH + - ALSA: caiaq: take a reference on the USB device in create_card() + - net/packet: fix TOCTOU race on mmap'd vnet_hdr in tpacket_snd() + - crypto: ccp: Don't attempt to copy CSR to userspace if PSP command + failed + - crypto: ccp: Don't attempt to copy PDH cert to userspace if PSP command + failed + - crypto: ccp: Don't attempt to copy ID to userspace if PSP command failed + - rxrpc: Fix missing validation of ticket length in non-XDR key preparsing + - mshv_vtl: Fix vmemmap_shift exceeding MAX_FOLIO_ORDER + - Linux 7.0.2 + * Resolute update: v7.0.1 upstream stable release (LP: #2150547) + - Revert "UBUNTU: SAUCE: cdc-acm: Exclude Exar USB serial ports" + - nfc: llcp: add missing return after LLCP_CLOSED checks + - x86/CPU: Fix FPDSS on Zen1 + - can: raw: fix ro->uniq use-after-free in raw_rcv() + - i2c: s3c24xx: check the size of the SMBUS message before using it + - staging: rtl8723bs: initialize le_tmp64 in rtw_BIP_verify() + - HID: alps: fix NULL pointer dereference in alps_raw_event() + - HID: core: clamp report_size in s32ton() to avoid undefined shift + - net: usb: cdc-phonet: fix skb frags[] overflow in rx_complete() + - NFC: digital: Bounds check NFC-A cascade depth in SDD response handler + - drm/vc4: platform_get_irq_byname() returns an int + - bnge: return after auxiliary_device_uninit() in error path + - ALSA: usx2y: us144mkii: fix NULL deref on missing interface 0 + - ALSA: fireworks: bound device-supplied status before string array lookup + - fbdev: tdfxfb: avoid divide-by-zero on FBIOPUT_VSCREENINFO + - usb: gadget: f_ncm: validate minimum block_len in ncm_unwrap_ntb() + - usb: gadget: f_phonet: fix skb frags[] overflow in pn_rx_complete() + - usb: gadget: renesas_usb3: validate endpoint index in standard request + handlers + - smb: client: fix off-by-8 bounds check in check_wsl_eas() + - smb: client: fix OOB reads parsing symlink error response + - ksmbd: validate EaNameLength in smb2_get_ea() + - ksmbd: require 3 sub-authorities before reading sub_auth[2] + - ksmbd: fix mechToken leak when SPNEGO decode fails after token alloc + - smb: client: avoid double-free in smbd_free_send_io() after + smbd_send_batch_flush() + - smb: server: avoid double-free in smb_direct_free_sendmsg after + smb_direct_flush_send_list() + - usbip: validate number_of_packets in usbip_pack_ret_submit() + - usb: typec: fusb302: Switch to threaded IRQ handler + - usb: storage: Expand range of matched versions for VL817 quirks entry + - USB: cdc-acm: Add quirks for Yoga Book 9 14IAH10 INGENIC touchscreen + - usb: gadget: f_hid: don't call cdev_init while cdev in use + - usb: port: add delay after usb_hub_set_port_power() + - fbdev: udlfb: avoid divide-by-zero on FBIOPUT_VSCREENINFO + - scripts/gdb/symbols: handle module path parameters + - scripts: generate_rust_analyzer.py: avoid FD leak + - wifi: rtw88: fix device leak on probe failure + - staging: sm750fb: fix division by zero in ps_to_hz() + - selftests/mm: hmm-tests: don't hardcode THP size to 2MB + - USB: serial: option: add Telit Cinterion FN990A MBIM composition + - Docs/admin-guide/mm/damon/reclaim: warn commit_inputs vs param updates + race + - Docs/admin-guide/mm/damon/lru_sort: warn commit_inputs vs param updates + race + - ALSA: ctxfi: Limit PTP to a single page + - dcache: Limit the minimal number of bucket to two + - vfio/xe: Reorganize the init to decouple migration from reset + - arm64: mm: Handle invalid large leaf mappings correctly + - media: vidtv: fix NULL pointer dereference in + vidtv_channel_pmt_match_sections + - ocfs2: fix possible deadlock between unlink and dio_end_io_write + - ocfs2: fix use-after-free in ocfs2_fault() when VM_FAULT_RETRY + - ocfs2: handle invalid dinode in ocfs2_group_extend + - PCI: endpoint: pci-epf-vntb: Stop cmd_handler work in + epf_ntb_epc_cleanup + - PCI: endpoint: pci-epf-vntb: Remove duplicate resource teardown + - KVM: selftests: Remove duplicate LAUNCH_UPDATE_VMSA call in SEV-ES + migrate test + - KVM: SEV: Reject attempts to sync VMSA of an already-launched/encrypted + vCPU + - KVM: SEV: Protect *all* of sev_mem_enc_register_region() with kvm->lock + - KVM: SEV: Disallow LAUNCH_FINISH if vCPUs are actively being created + - KVM: SEV: Lock all vCPUs when synchronzing VMSAs for SNP launch finish + - KVM: SEV: Drop WARN on large size for KVM_MEMORY_ENCRYPT_REG_REGION + - mm: call ->free_folio() directly in folio_unmap_invalidate() + - checkpatch: add support for Assisted-by tag + - x86-64: rename misleadingly named '__copy_user_nocache()' function + - x86: rename and clean up __copy_from_user_inatomic_nocache() + - x86-64/arm64/powerpc: clean up and rename __copy_from_user_flushcache + - KVM: x86: Use scratch field in MMIO fragment to hold small write values + - ASoC: qcom: q6apm: move component registration to unmanaged version + - mm/kasan: fix double free for kasan pXds + - mm: blk-cgroup: fix use-after-free in cgwb_release_workfn() + - media: vidtv: fix nfeeds state corruption on start_streaming failure + - media: mediatek: vcodec: fix use-after-free in encoder release path + - media: em28xx: fix use-after-free in em28xx_v4l2_open() + - hwmon: (powerz) Fix use-after-free on USB disconnect + - ALSA: 6fire: fix use-after-free on disconnect + - bcache: fix cached_dev.sb_bio use-after-free and crash + - wireguard: device: use exit_rtnl callback instead of manual rtnl_lock in + pre_exit + - media: as102: fix to not free memory after the device is registered in + as102_usb_probe() + - nilfs2: fix NULL i_assoc_inode dereference in + nilfs_mdt_save_to_shadow_map + - media: vidtv: fix pass-by-value structs causing MSAN warnings + - media: hackrf: fix to not free memory after the device is registered in + hackrf_probe() + - mm/userfaultfd: fix hugetlb fault mutex hash calculation + - clockevents: Add missing resets of the next_event_forced flag + - Linux 7.0.1 + * GRO managed-frag use-after-free leading to local privilege escalation + (LP: #2154172) + - net: gro: don't merge zcopy skbs + * AppArmor Vulnerabilities (LP: #2151747) + - SAUCE: apparmor: pass big_resp to handler + - SAUCE: apparmor: remove redundant kref_init for listener->count + - SAUCE: apparmor: fix NULL pointer dereference in unpack_pdb + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47337 + - SAUCE: apparmor: fix NULL pointer dereference in bind_map_addr + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47334 + - SAUCE: apparmor: fix sleep prone memory allocation under a spin_lock + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47333 + - SAUCE: apparmor: fix dfa unpacking size of the notification filter + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47332 + - SAUCE: apparmor: fix size check against type instead of pointer + * apparmor: LLVM/clang build failure due to uninitialized variable in + notify.c (LP: #2148809) // CVE-2026-47330 + - SAUCE: apparmor: initialize variable used in uninitialized context + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47329 + - SAUCE: apparmor: fix name validation bypass on notification + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47327 // + CVE-2026-47328 + - SAUCE: apparmor: fix glob memory leak after kstrdup + * AppArmor Vulnerabilities (LP: #2151747) // CVE-2026-47326 + - SAUCE: apparmor: fix inverted NULL check after aa_get_buffer + * CVE-2026-46300 + - net: skbuff: preserve shared-frag marker during coalescing + - net: skbuff: propagate shared-frag marker through frag-transfer helpers + * net/rds: reset op_nents when zerocopy page pin fails (LP: #2153962) + - net/rds: reset op_nents when zerocopy page pin fails + * CVE-2026-46333 + - ptrace: slightly saner 'get_dumpable()' logic + * CVE-2026-43500 + - rxrpc: Fix conn-level packet handling to unshare RESPONSE packets + - rxrpc: Fix potential UAF after skb_unshare() failure + - rxrpc: Fix rxrpc_input_call_event() to only unshare DATA packets + - rxrpc: Also unshare DATA/RESPONSE packets when paged frags are present + * CVE-2026-43284 + - xfrm: esp: avoid in-place decrypt on shared skb frags + + -- Jacob Martin Thu, 18 Jun 2026 20:24:04 -0500 linux-nvidia-bos (7.0.0-2009.9) resolute; urgency=medium diff --git a/debian.nvidia-bos/reconstruct b/debian.nvidia-bos/reconstruct index ef633a946371a..d3d1c7130e908 100644 --- a/debian.nvidia-bos/reconstruct +++ b/debian.nvidia-bos/reconstruct @@ -43,4 +43,6 @@ chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.c' chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.h' # Remove any files deleted from the orig. rm -f 'Documentation/admin-guide/perf/nvidia-pmu.rst' +rm -f 'arch/parisc/lib/checksum.c' +rm -f 'tools/testing/selftests/mqueue/setting' exit 0 From 3cb358b4608d7c8cb163fb00ce3a060686ae037d Mon Sep 17 00:00:00 2001 From: Farhan Ali Date: Mon, 22 Jun 2026 10:18:40 -0700 Subject: [PATCH 440/464] NVIDIA: SAUCE: PCI/MSI: Enable memory decoding before restoring MSI-X messages BugLink: https://bugs.launchpad.net/bugs/2158328 The current MSI-X restoration path assumes the Command register Memory bit is enabled when writing MSI-X messages. But its possible the last saved and restored state of device may not have the Memory bit enabled, even if a device driver later enables Memory bit and MSI-X. Attempting to access Memory space without Memory bit enabled can lead to Unsupported Request (UR) from the device. Fix this by enabling Memory bit and restore it afterwards. Signed-off-by: Farhan Ali Reviewed-by: Niklas Schnelle Reviewed-by: Thomas Gleixner (cherry picked from https://lore.kernel.org/all/20260622171840.1618-5-alifm@linux.ibm.com/) Signed-off-by: Matthew R. Ochs Acked-by: Seth Forshee Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/pci/msi/msi.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/pci/msi/msi.c b/drivers/pci/msi/msi.c index e2412175d7af2..5389b431d2e78 100644 --- a/drivers/pci/msi/msi.c +++ b/drivers/pci/msi/msi.c @@ -864,6 +864,7 @@ void __pci_restore_msix_state(struct pci_dev *dev) { struct msi_desc *entry; bool write_msg; + u16 cmd; if (!dev->msix_enabled) return; @@ -872,6 +873,8 @@ void __pci_restore_msix_state(struct pci_dev *dev) pci_intx_for_msi(dev, 0); pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_ENABLE | PCI_MSIX_FLAGS_MASKALL); + pci_read_config_word(dev, PCI_COMMAND, &cmd); + pci_write_config_word(dev, PCI_COMMAND, cmd | PCI_COMMAND_MEMORY); write_msg = arch_restore_msi_irqs(dev); @@ -883,6 +886,7 @@ void __pci_restore_msix_state(struct pci_dev *dev) } } + pci_write_config_word(dev, PCI_COMMAND, cmd); pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0); } From fc18cc1c472b551ae270a63f47d3778640ea25ba Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Jul 2026 13:49:38 -0500 Subject: [PATCH 441/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index d9ed503570840..cc2919a5c2054 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2014.14) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Wed, 01 Jul 2026 13:49:38 -0500 + linux-nvidia-bos (7.0.0-2012.12) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2012.12 -proposed tracker (LP: #2156940) From 948187a7d4dbdb9b0881f1633bb9e59e9625d970 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Jul 2026 13:51:06 -0500 Subject: [PATCH 442/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2158813 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index ebcee9aba3809..a4560ac96536d 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2156940 d2026.06.15-1 +2158813 d2026.06.30-1 From 3642b22d61677574dade1f7641dae9fac764f88b Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Jul 2026 13:51:52 -0500 Subject: [PATCH 443/464] UBUNTU: [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel-versions (adhoc/d2026.06.30) BugLink: https://bugs.launchpad.net/bugs/1786013 Signed-off-by: Jacob Martin --- debian.nvidia-bos/dkms-versions | 1 + 1 file changed, 1 insertion(+) diff --git a/debian.nvidia-bos/dkms-versions b/debian.nvidia-bos/dkms-versions index 3f4cd049293c0..4a941b2e62257 100644 --- a/debian.nvidia-bos/dkms-versions +++ b/debian.nvidia-bos/dkms-versions @@ -1,3 +1,4 @@ zfs-linux 2.4.1-1ubuntu5 modulename=zfs debpath=pool/universe/z/%package%/zfs-dkms_%version%_all.deb arch=amd64 arch=arm64 arch=ppc64el arch=riscv64 arch=s390x rprovides=spl-modules rprovides=spl-dkms rprovides=zfs-modules rprovides=zfs-dkms off_series=true v4l2loopback 0.15.3-1ubuntu2 modulename=v4l2loopback debpath=pool/universe/v/%package%/v4l2loopback-dkms_%version%_all.deb arch=amd64 rprovides=v4l2loopback-modules rprovides=v4l2loopback-dkms off_series=true nvidia-fs 2.29.4-1 modulename=nvidia-fs debpath=pool/universe/n/%package%/nvidia-fs-dkms_%version%_amd64.deb arch=amd64 arch=arm64 rprovides=nvidia-fs-modules rprovides=nvidia-fs-dkms type=standalone +mstflint 4.33.0+1-1.1 modulename=mstflint_access debpath=pool/universe/m/%package%/mstflint-dkms_%version%_all.deb arch=amd64 arch=arm64 rprovides=mstflint-modules rprovides=mstflint-dkms From a8601926731c539847aaf5d5d34c1e6c4ce3413b Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 1 Jul 2026 13:52:27 -0500 Subject: [PATCH 444/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2014.14 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index cc2919a5c2054..e989e5f11d970 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,17 @@ -linux-nvidia-bos (7.0.0-2014.14) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2014.14) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2014.14 -proposed tracker (LP: #2158813) - -- Jacob Martin Wed, 01 Jul 2026 13:49:38 -0500 + * Packaging resync (LP: #1786013) + - [Packaging] debian.nvidia-bos/dkms-versions -- update from kernel- + versions (adhoc/d2026.06.30) + + * Backport PCI/MSI: Enable memory decoding before restoring MSI-X messages + (LP: #2158328) + - NVIDIA: SAUCE: PCI/MSI: Enable memory decoding before restoring MSI-X + messages + + -- Jacob Martin Wed, 01 Jul 2026 13:52:27 -0500 linux-nvidia-bos (7.0.0-2012.12) resolute; urgency=medium From c859a5ab2179714bf03004a6b1ea04beb748f929 Mon Sep 17 00:00:00 2001 From: Shrikanth Hegde Date: Fri, 15 May 2026 22:54:53 +0530 Subject: [PATCH 445/464] topology: Introduce cpu_smt_mask for CONFIG_SCHED_SMT=n BugLink: https://bugs.launchpad.net/bugs/2158811 Define cpu_smt_mask in case of CONFIG_SCHED_SMT=n as cpumask_of that CPU. With that config, it is expected that kernel treats each CPU as individual core. Using cpumask_of(cpu) reflects that. This would help to get rid of the ifdeffery that is spread across the codebase since cpu_smt_mask is defined only in case of CONFIG_SCHED_SMT=y. Note: There is no arch today which defines cpu_smt_mask unconditionally. So likely defining the cpu_smt_mask shouldn't lead redefinition errors. Signed-off-by: Shrikanth Hegde Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Phil Auld Reviewed-by: Valentin Schneider Tested-by: K Prateek Nayak Link: https://patch.msgid.link/20260515172456.542799-2-sshegde@linux.ibm.com (cherry picked from commit 8d3ad4bd922e94834d5a5e1c46df625fd21a4acb) Signed-off-by: Andrea Righi Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- include/linux/topology.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/include/linux/topology.h b/include/linux/topology.h index 6575af39fd10f..709a2dcf4c738 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -230,11 +230,24 @@ static inline int cpu_to_mem(int cpu) #define topology_drawer_cpumask(cpu) cpumask_of(cpu) #endif -#if defined(CONFIG_SCHED_SMT) && !defined(cpu_smt_mask) +/* + * Defining cpu_smt_mask as cpumask_of that CPU helps to get + * rid of lot of ifdeffery all around the codebase in case of + * CONFIG_SCHED_SMT=n. It just means there are no other siblings, which + * is what is expected. + */ +#if defined(CONFIG_SCHED_SMT) +# if !defined(cpu_smt_mask) static inline const struct cpumask *cpu_smt_mask(int cpu) { return topology_sibling_cpumask(cpu); } +# endif +#else /* !CONFIG_SCHED_SMT */ +static inline const struct cpumask *cpu_smt_mask(int cpu) +{ + return cpumask_of(cpu); +} #endif #ifndef topology_is_primary_thread From 7cb635dfc8ada8f3008e665670f85f5b6af0b0ad Mon Sep 17 00:00:00 2001 From: Andrea Righi Date: Sun, 28 Jun 2026 08:47:06 +0200 Subject: [PATCH 446/464] NVIDIA: VR: SAUCE: sched/fair: Stabilize idle SMT core selection with asym-capacity BugLink: https://bugs.launchpad.net/bugs/2158811 select_idle_capacity() scans all logical CPUs also when it is looking for a fully idle SMT core. Two concurrent wakeups can therefore observe the same core as idle, encounter different siblings first, and place one task on each sibling while another core remains unused. Make every logical CPU of a selected idle core resolve to the same stable CPU representative within the scan's existing affinity and scheduling-domain mask. If the first task is enqueued before the next scan examines the core, that scan rejects the now-busy core. If both scans observe the core as idle, they select the same runqueue even if the first enqueue becomes visible before the second scan finishes, exposing the imbalance to the load balancer. The symmetric CPU idle selection path is subject to the same race, but normally returns as soon as select_idle_core() finds a fully idle core, reducing the conflict window. The per-CPU capacity scan can retain an idle-core candidate while evaluating other CPUs, giving concurrent wakeups more opportunity to select different siblings of the same SMT core. Therefore, limit the normalization to the asym-capacity path, where this behavior has a measurable impact. On NVIDIA Vera Rubin (arm64, 176 CPUs/88 cores per NUMA node), a CPU-intensive NVPL SGEMM workload restricted to 88 threads (one per core) showed a consistent 23% increase in mean throughput across multiple runs. For comparison, DCPerf MediaWiki running at system saturation (with all SMT siblings busy) showed neither a benefit nor a regression: throughput and Nginx request latency remained within measurement error. Likewise, schbench under partially idle conditions showed no material change in wakeup latency, request latency, or throughput (within 0.1%). Tail wakeup latency was more consistent across runs with this change applied. Signed-off-by: Andrea Righi [ arighi: adjust context for v7.0 ] (backported from https://lore.kernel.org/all/20260630152747.128746-1-arighi@nvidia.com) Signed-off-by: Andrea Righi Acked-by: Jamie Nguyen Acked-by: Matthew R. Ochs Signed-off-by: Brad Figg --- kernel/sched/fair.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index def97f2d6a470..6054931ac774b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7773,6 +7773,16 @@ enum asym_fits_state { ASYM_IDLE_CORE_BIAS = -3, }; +/* + * Return a stable CPU representative of @cpu's SMT core within @cpus. + */ +static int select_idle_core_cpu(int cpu, const struct cpumask *cpus) +{ + int sibling = cpumask_first_and(cpu_smt_mask(cpu), cpus); + + return sibling < nr_cpu_ids ? sibling : cpu; +} + /* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which * the task fits. If no CPU is big enough, but there are idle ones, try to @@ -7782,6 +7792,7 @@ static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) { bool prefers_idle_core = sched_smt_active() && test_idle_cores(target); + bool best_idle_core = false; unsigned long task_util, util_min, util_max, best_cap = 0; int fits, best_fits = ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu = -1; @@ -7807,7 +7818,8 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) } for_each_cpu_wrap(cpu, cpus, target) { - bool preferred_core = !prefers_idle_core || is_core_idle(cpu); + bool idle_core = !sched_smt_active() || is_core_idle(cpu); + bool preferred_core = !prefers_idle_core || idle_core; unsigned long cpu_cap = capacity_of(cpu); /* @@ -7824,7 +7836,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) /* This CPU fits with all requirements */ if (fits > 0 && preferred_core) - return cpu; + return idle_core ? select_idle_core_cpu(cpu, cpus) : cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. * Look for the CPU with best capacity. @@ -7865,6 +7877,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) best_cap = cpu_cap; best_cpu = cpu; best_fits = fits; + best_idle_core = idle_core; } } @@ -7880,6 +7893,8 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) */ if (prefers_idle_core && best_fits > ASYM_IDLE_CORE_BIAS) set_idle_cores(target, false); + else if (best_idle_core) + best_cpu = select_idle_core_cpu(best_cpu, cpus); return best_cpu; } From 8179cb9c6134d2b150956ec1310345741999b0d3 Mon Sep 17 00:00:00 2001 From: David Thompson Date: Thu, 28 May 2026 16:50:17 +0000 Subject: [PATCH 447/464] net: lan743x: avoid netdev-based logging before netdev registration BugLink: https://bugs.launchpad.net/bugs/2156928 This patch updates the lan743x driver to prevent the use of netdev-based logging APIs (such as netdev_dbg) before the network device has been successfully registered. Using netdev-based logging prior to registration results in log messages referencing "(unnamed net_device) (uninitialized)", which can be confusing and less informative. The driver must use netif_msg_ APIs and device-based logging (e.g. dev_dbg) until netdev registration is complete. This ensures log entries are associated with the correct device context and improves log clarity. After registration, netdev-based logging APIs can be used safely. Signed-off-by: David Thompson Link: https://patch.msgid.link/20260528165017.421576-1-davthompson@nvidia.com Signed-off-by: Jakub Kicinski (cherry picked from commit e3c6508a46f56ece0c1550a4fdf1e005afe3d563) Signed-off-by: David Thompson Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- drivers/net/ethernet/microchip/lan743x_main.c | 48 ++++++++----------- 1 file changed, 21 insertions(+), 27 deletions(-) diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c index f3332417162e6..d40c277af112b 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.c +++ b/drivers/net/ethernet/microchip/lan743x_main.c @@ -108,9 +108,9 @@ static int lan743x_pci_init(struct lan743x_adapter *adapter, if (ret) goto return_error; - netif_info(adapter, probe, adapter->netdev, - "PCI: Vendor ID = 0x%04X, Device ID = 0x%04X\n", - pdev->vendor, pdev->device); + dev_dbg(&adapter->pdev->dev, + "PCI: Vendor ID = 0x%04X, Device ID = 0x%04X\n", + pdev->vendor, pdev->device); bars = pci_select_bars(pdev, IORESOURCE_MEM); if (!test_bit(0, &bars)) goto disable_device; @@ -192,10 +192,10 @@ static int lan743x_csr_init(struct lan743x_adapter *adapter) csr->id_rev = lan743x_csr_read(adapter, ID_REV); csr->fpga_rev = lan743x_csr_read(adapter, FPGA_REV); - netif_info(adapter, probe, adapter->netdev, - "ID_REV = 0x%08X, FPGA_REV = %d.%d\n", - csr->id_rev, FPGA_REV_GET_MAJOR_(csr->fpga_rev), - FPGA_REV_GET_MINOR_(csr->fpga_rev)); + dev_dbg(&adapter->pdev->dev, + "ID_REV = 0x%08X, FPGA_REV = %d.%d\n", + csr->id_rev, FPGA_REV_GET_MAJOR_(csr->fpga_rev), + FPGA_REV_GET_MINOR_(csr->fpga_rev)); if (!ID_REV_IS_VALID_CHIP_ID_(csr->id_rev)) return -ENODEV; @@ -953,8 +953,8 @@ int lan743x_sgmii_read(struct lan743x_adapter *adapter, u8 mmd, u16 addr) u32 val; if (mmd > 31) { - netif_err(adapter, probe, adapter->netdev, - "%s mmd should <= 31\n", __func__); + dev_err(&adapter->pdev->dev, + "%s mmd should <= 31\n", __func__); return -EINVAL; } @@ -983,8 +983,8 @@ static int lan743x_sgmii_write(struct lan743x_adapter *adapter, int ret; if (mmd > 31) { - netif_err(adapter, probe, adapter->netdev, - "%s mmd should <= 31\n", __func__); + dev_err(&adapter->pdev->dev, + "%s mmd should <= 31\n", __func__); return -EINVAL; } mutex_lock(&adapter->sgmii_rw_lock); @@ -1215,8 +1215,7 @@ static void lan743x_mac_set_address(struct lan743x_adapter *adapter, lan743x_csr_write(adapter, MAC_RX_ADDRH, addr_hi); ether_addr_copy(adapter->mac_address, addr); - netif_info(adapter, drv, adapter->netdev, - "MAC address set to %pM\n", addr); + dev_dbg(&adapter->pdev->dev, "MAC address set to %pM\n", addr); } static int lan743x_mac_init(struct lan743x_adapter *adapter) @@ -1370,8 +1369,8 @@ static void lan743x_phy_interface_select(struct lan743x_adapter *adapter) else adapter->phy_interface = PHY_INTERFACE_MODE_RGMII; - netif_dbg(adapter, drv, adapter->netdev, - "selected phy interface: 0x%X\n", adapter->phy_interface); + dev_dbg(&adapter->pdev->dev, + "selected phy interface: 0x%X\n", adapter->phy_interface); } static void lan743x_rfe_open(struct lan743x_adapter *adapter) @@ -3168,7 +3167,7 @@ static int lan743x_phylink_create(struct lan743x_adapter *adapter) } adapter->phylink = pl; - netdev_dbg(netdev, "lan743x phylink created"); + dev_dbg(&adapter->pdev->dev, "lan743x phylink created"); return 0; } @@ -3581,30 +3580,26 @@ static int lan743x_mdiobus_init(struct lan743x_adapter *adapter) adapter->mdiobus->priv = (void *)adapter; if (adapter->is_pci11x1x) { if (adapter->is_sgmii_en) { - netif_dbg(adapter, drv, adapter->netdev, - "SGMII operation\n"); + dev_dbg(&adapter->pdev->dev, "SGMII operation\n"); adapter->mdiobus->read = lan743x_mdiobus_read_c22; adapter->mdiobus->write = lan743x_mdiobus_write_c22; adapter->mdiobus->read_c45 = lan743x_mdiobus_read_c45; adapter->mdiobus->write_c45 = lan743x_mdiobus_write_c45; adapter->mdiobus->name = "lan743x-mdiobus-c45"; - netif_dbg(adapter, drv, adapter->netdev, - "lan743x-mdiobus-c45\n"); + dev_dbg(&adapter->pdev->dev, "lan743x-mdiobus-c45\n"); } else { - netif_dbg(adapter, drv, adapter->netdev, - "RGMII operation\n"); + dev_dbg(&adapter->pdev->dev, "RGMII operation\n"); // Only C22 support when RGMII I/F adapter->mdiobus->read = lan743x_mdiobus_read_c22; adapter->mdiobus->write = lan743x_mdiobus_write_c22; adapter->mdiobus->name = "lan743x-mdiobus"; - netif_dbg(adapter, drv, adapter->netdev, - "lan743x-mdiobus\n"); + dev_dbg(&adapter->pdev->dev, "lan743x-mdiobus\n"); } } else { adapter->mdiobus->read = lan743x_mdiobus_read_c22; adapter->mdiobus->write = lan743x_mdiobus_write_c22; adapter->mdiobus->name = "lan743x-mdiobus"; - netif_dbg(adapter, drv, adapter->netdev, "lan743x-mdiobus\n"); + dev_dbg(&adapter->pdev->dev, "lan743x-mdiobus\n"); } snprintf(adapter->mdiobus->id, MII_BUS_ID_SIZE, @@ -3696,8 +3691,7 @@ static int lan743x_pcidev_probe(struct pci_dev *pdev, ret = lan743x_phylink_create(adapter); if (ret < 0) { - netif_err(adapter, probe, netdev, - "failed to setup phylink (%d)\n", ret); + dev_err(&pdev->dev, "failed to setup phylink (%d)\n", ret); goto cleanup_mdiobus; } From fd15b2cbcf5cd7bc0eee741e5fe50bedb9c489ea Mon Sep 17 00:00:00 2001 From: David Thompson Date: Fri, 29 May 2026 21:03:00 +0000 Subject: [PATCH 448/464] net: lan743x: permit VLAN-tagged packets up to configured MTU BugLink: https://bugs.launchpad.net/bugs/2156928 VLAN-tagged interfaces on lan743x devices were previously unreachable via SSH and failed to respond to large ping packets (e.g. "ping -s 1469" given MTU=1500). In these scenarios, "ethtool -S" reports non-zero "RX Oversize Frame Errors". According to Microchip AN2948, the MAC_RX FSE (VLAN field size enforcement) bit determines whether frames with VLAN tags exceeding the base MTU plus tag length are discarded. The driver must set the MAC_RX.FSE bit before setting MAC_RX.RXEN to allow VLAN-tagged frames up to the interface MTU, preventing them from being treated as oversized. As a result, both the base and VLAN-tagged interfaces can use the same MTU without receive errors. Fixes: 23f0703c125b ("lan743x: Add main source files for new lan743x driver") Signed-off-by: David Thompson Reviewed-by: Thangaraj Samynathan Reviewed-by: Nicolai Buchwitz Tested-by: Nicolai Buchwitz # lan7430 on arm64 (RevPi Link: https://patch.msgid.link/20260529210300.433135-1-davthompson@nvidia.com Signed-off-by: Jakub Kicinski (cherry picked from commit 8173d22b211f615015f7b35f48ab11a6dd78dc99) Signed-off-by: David Thompson Acked-by: Matthew R. Ochs Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- drivers/net/ethernet/microchip/lan743x_main.c | 32 +++++++++++++++++++ drivers/net/ethernet/microchip/lan743x_main.h | 1 + 2 files changed, 33 insertions(+) diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c index d40c277af112b..1cdce35e14239 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.c +++ b/drivers/net/ethernet/microchip/lan743x_main.c @@ -1218,6 +1218,36 @@ static void lan743x_mac_set_address(struct lan743x_adapter *adapter, dev_dbg(&adapter->pdev->dev, "MAC address set to %pM\n", addr); } +static void lan743x_mac_rx_enable_fse(struct lan743x_adapter *adapter) +{ + u32 mac_rx; + bool rxen; + + mac_rx = lan743x_csr_read(adapter, MAC_RX); + if (mac_rx & MAC_RX_FSE_) + return; + + rxen = mac_rx & MAC_RX_RXEN_; + if (rxen) { + mac_rx &= ~MAC_RX_RXEN_; + lan743x_csr_write(adapter, MAC_RX, mac_rx); + lan743x_csr_wait_for_bit(adapter, MAC_RX, MAC_RX_RXD_, + 1, 1000, 20000, 100); + } + + /* Per AN2948, hardware prevents modification of the FSE bit while the + * MAC receiver is enabled (RXEN bit set). Use separate register write + * to assert the FSE bit before enabling the RXEN bit in MAC_RX + */ + mac_rx |= MAC_RX_FSE_; + lan743x_csr_write(adapter, MAC_RX, mac_rx); + + if (rxen) { + mac_rx |= MAC_RX_RXEN_; + lan743x_csr_write(adapter, MAC_RX, mac_rx); + } +} + static int lan743x_mac_init(struct lan743x_adapter *adapter) { bool mac_address_valid = true; @@ -1257,6 +1287,8 @@ static int lan743x_mac_init(struct lan743x_adapter *adapter) lan743x_mac_set_address(adapter, adapter->mac_address); eth_hw_addr_set(netdev, adapter->mac_address); + lan743x_mac_rx_enable_fse(adapter); + return 0; } diff --git a/drivers/net/ethernet/microchip/lan743x_main.h b/drivers/net/ethernet/microchip/lan743x_main.h index 160d94a7cee66..1573c8f9c9937 100644 --- a/drivers/net/ethernet/microchip/lan743x_main.h +++ b/drivers/net/ethernet/microchip/lan743x_main.h @@ -182,6 +182,7 @@ #define MAC_RX (0x104) #define MAC_RX_MAX_SIZE_SHIFT_ (16) #define MAC_RX_MAX_SIZE_MASK_ (0x3FFF0000) +#define MAC_RX_FSE_ BIT(2) #define MAC_RX_RXD_ BIT(1) #define MAC_RX_RXEN_ BIT(0) From 94220739a54718f01c85144064957d4f7b2264df Mon Sep 17 00:00:00 2001 From: Nirmoy Das Date: Fri, 3 Jul 2026 07:02:34 -0700 Subject: [PATCH 449/464] NVIDIA: VR: SAUCE: cxl: Guard unlinked memdev endpoints BugLink: https://bugs.launchpad.net/bugs/2153819 cxlmd->endpoint starts as ERR_PTR(-ENXIO) until endpoint port registration links the memdev to a real cxl_port. Treat NULL and error pointers as "endpoint not linked" before dereferencing cxlmd->endpoint in CXL helper paths. The BOS region-management backport exposes these helpers before endpoint linkage. This backports commit aff4ccee4530 ("NVIDIA: VR: SAUCE: cxl: Guard unlinked memdev endpoints"). Its PCI hunk is omitted because BOS already guards cxl_reset_done(). Fixes: 29317f8dc6ed ("cxl/mem: Introduce cxl_memdev_attach for CXL-dependent operation") Signed-off-by: Nirmoy Das Acked-by: Andrea Righi Acked-by: Jamie Nguyen Signed-off-by: Brad Figg --- drivers/cxl/core/hdm.c | 6 +++++- drivers/cxl/core/region.c | 6 ++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c index 603cc746aa497..e0d084267a7aa 100644 --- a/drivers/cxl/core/hdm.c +++ b/drivers/cxl/core/hdm.c @@ -3,6 +3,7 @@ #include #include #include +#include #include "cxlmem.h" #include "core.h" @@ -617,6 +618,9 @@ cxl_find_free_decoder(struct cxl_memdev *cxlmd) struct cxl_port *endpoint = cxlmd->endpoint; struct device *dev; + if (IS_ERR_OR_NULL(endpoint)) + return NULL; + guard(rwsem_read)(&cxl_rwsem.dpa); dev = device_find_child(&endpoint->dev, NULL, find_free_decoder); if (!dev) @@ -759,7 +763,7 @@ struct cxl_endpoint_decoder *cxl_get_committed_decoder(struct cxl_memdev *cxlmd, struct cxl_endpoint_decoder *cxled; struct device *cxled_dev; - if (!endpoint) + if (IS_ERR_OR_NULL(endpoint)) return NULL; guard(rwsem_read)(&cxl_rwsem.dpa); diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index e820fb63b95aa..a7062a502f88a 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -814,7 +815,7 @@ struct cxl_root_decoder *cxl_get_hpa_freespace(struct cxl_memdev *cxlmd, struct cxl_port *endpoint; endpoint = cxlmd->endpoint; - if (!endpoint) { + if (IS_ERR_OR_NULL(endpoint)) { dev_dbg(&cxlmd->dev, "endpoint not linked to memdev\n"); return ERR_PTR(-ENXIO); } @@ -3172,7 +3173,8 @@ struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa) .dpa = dpa, }; port = cxlmd->endpoint; - if (port && is_cxl_endpoint(port) && cxl_num_decoders_committed(port)) + if (!IS_ERR_OR_NULL(port) && is_cxl_endpoint(port) && + cxl_num_decoders_committed(port)) device_for_each_child(&port->dev, &ctx, __cxl_dpa_to_region); return ctx.cxlr; From 985d78260ff1e7cd661a69b7b83bae7a48ed3b07 Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Fri, 24 Apr 2026 18:15:21 -0700 Subject: [PATCH 450/464] iommu: Fix kdocs of pci_dev_reset_iommu_done() BugLink: https://bugs.launchpad.net/bugs/2159852 Remove the duplicated word. No functional change. Fixes: c279e83953d9 ("iommu: Introduce pci_dev_reset_iommu_prepare/done()") Reviewed-by: Shuai Xue Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian Reviewed-by: Lu Baolu Signed-off-by: Nicolin Chen Signed-off-by: Joerg Roedel (cherry picked from commit 834ab85aa96656f87bb215a8285d34af870f4b66) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/iommu/iommu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 877eaef1ca7f5..e32e4fb2f4899 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -4145,9 +4145,9 @@ EXPORT_SYMBOL_GPL(pci_dev_reset_iommu_prepare); * @pdev: PCI device that has finished a reset routine * * After a PCIe device finishes a reset routine, it wants to restore its IOMMU - * IOMMU activity, including new translation as well as cache invalidation, by - * re-attaching all RID/PASID of the device's back to the domains retained in - * the core-level structure. + * activity, including new translation and cache invalidation, by re-attaching + * all RID/PASID of the device back to the domains retained in the core-level + * structure. * * Caller must pair it with a successful pci_dev_reset_iommu_prepare(). * From fc095857bb400890ead9351932761a3d8659b2ae Mon Sep 17 00:00:00 2001 From: Nicolin Chen Date: Fri, 24 Apr 2026 18:15:27 -0700 Subject: [PATCH 451/464] iommu: Warn on premature unblock during DMA aliased sibling reset BugLink: https://bugs.launchpad.net/bugs/2159852 When two aliased siblings are in the same iommu_group, they might share the same RID. The reset functions don't support this case, though it is unclear whether there is a real case of having an ATS capable device on a PCI/PCI-X bus. Theoretically, however, if two aliased devices are resetting concurrently, one might be unblocked prematurely in the middle of the reset by the other sibling who completes the reset first. This isn't a regression from this series but it's better to spit a warning, so we can know if such use case is common enough for us to make subsequent patches for its coverage. Signed-off-by: Nicolin Chen Reviewed-by: Kevin Tian Signed-off-by: Joerg Roedel (cherry picked from commit 15dd29ca620648a18a0334a3f6b26af181154a23) Signed-off-by: Jiandi An Acked-by: Jamie Nguyen Acked-by: Carol L Soto Acked-by: Nirmoy Das Signed-off-by: Brad Figg --- drivers/iommu/iommu.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e32e4fb2f4899..558442222f9db 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -4140,6 +4140,41 @@ int pci_dev_reset_iommu_prepare(struct pci_dev *pdev) } EXPORT_SYMBOL_GPL(pci_dev_reset_iommu_prepare); +static int __group_device_cmp_dma_alias(struct pci_dev *dev, u16 alias, + void *data) +{ + return alias == *(u16 *)data; +} + +static int group_device_cmp_dma_alias(struct pci_dev *dev, u16 alias, + void *data) +{ + return pci_for_each_dma_alias(data, __group_device_cmp_dma_alias, + &alias); +} + +static bool group_device_dma_alias_is_blocked(struct iommu_group *group, + struct group_device *gdev) +{ + struct group_device *sibling; + + lockdep_assert_held(&group->mutex); + + if (!dev_is_pci(gdev->dev)) + return false; + + for_each_group_device(group, sibling) { + if (sibling == gdev || !sibling->blocked || + !dev_is_pci(sibling->dev)) + continue; + if (pci_for_each_dma_alias(to_pci_dev(gdev->dev), + group_device_cmp_dma_alias, + to_pci_dev(sibling->dev))) + return true; + } + return false; +} + /** * pci_dev_reset_iommu_done() - Restore IOMMU after a PCI device reset is done * @pdev: PCI device that has finished a reset routine @@ -4179,6 +4214,20 @@ void pci_dev_reset_iommu_done(struct pci_dev *pdev) if (WARN_ON(!group->blocking_domain)) return; + if (group_device_dma_alias_is_blocked(group, gdev)) { + /* + * FIXME: DMA aliased devices share the same RID, which would be + * convoluted to handle, as "gdev->blocked" is not sufficient: + * - "blocked" state is effectively shared across these devices + * - if the core skipped the blocking on the second device, the + * IOMMU driver's attachment state would diverge from the HW + * state + * For now, just warn and see whether real ATS use cases hit it. + */ + pci_warn(pdev, + "DMA-aliased sibling may be prematurely unblocked\n"); + } + /* * Re-attach RID domain back to group->domain * From e4d50bf8131deea3c3213dfbac61c610e64cb449 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Jul 2026 14:44:42 -0500 Subject: [PATCH 452/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index e989e5f11d970..568c284af3136 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2015.15) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Tue, 07 Jul 2026 14:44:42 -0500 + linux-nvidia-bos (7.0.0-2014.14) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2014.14 -proposed tracker (LP: #2158813) From 682f1f11ab15b0c9b92cca92268ac1c4146206f6 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Jul 2026 15:36:29 -0500 Subject: [PATCH 453/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2160067 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index a4560ac96536d..e589c9c126cd0 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2158813 d2026.06.30-1 +2160067 d2026.06.30-2 From 58da2e1fe4a0dfc1d79561895f276ad254c8e56a Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Tue, 7 Jul 2026 15:43:30 -0500 Subject: [PATCH 454/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2015.15 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 568c284af3136..8f2a49fd06ce0 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,30 @@ -linux-nvidia-bos (7.0.0-2015.15) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2015.15) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2015.15 -proposed tracker (LP: #2160067) - -- Jacob Martin Tue, 07 Jul 2026 14:44:42 -0500 + * Backport iommu: Fix pci_dev_reset_iommu_prepare/done() (LP: #2159852) + - iommu: Fix NULL group->domain dereference in pci_dev_reset_iommu_done() + - iommu: Fix kdocs of pci_dev_reset_iommu_done() + - iommu: Replace per-group resetting_domain with per-gdev blocked flag + - iommu: Fix pasid attach in pci_dev_reset_iommu_prepare/done() + - iommu: Fix nested pci_dev_reset_iommu_prepare/done() + - iommu: Fix ATS invalidation timeouts during __iommu_remove_group_pasid() + - iommu: Fix WARN_ON in __iommu_group_set_domain_nofail() due to reset + - iommu: Warn on premature unblock during DMA aliased sibling reset + + * CXL: Backport Type-2, state save/restore, and reset support (LP: #2153819) + - NVIDIA: VR: SAUCE: cxl: Guard unlinked memdev endpoints + + * Backport vlan and log fix for lan743x driver (LP: #2156928) + - net: lan743x: avoid netdev-based logging before netdev registration + - net: lan743x: permit VLAN-tagged packets up to configured MTU + + * Reduce SMT contention on Vera (LP: #2158811) + - topology: Introduce cpu_smt_mask for CONFIG_SCHED_SMT=n + - NVIDIA: VR: SAUCE: sched/fair: Stabilize idle SMT core selection with + asym-capacity + + -- Jacob Martin Tue, 07 Jul 2026 15:43:30 -0500 linux-nvidia-bos (7.0.0-2014.14) resolute; urgency=medium From 4657f9f13c7a64c0893f3f777ae7b2ca687cd272 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 22 Jul 2026 09:54:00 -0500 Subject: [PATCH 455/464] UBUNTU: Start new release Ignore: yes Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 8f2a49fd06ce0..283fa90227ef4 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,3 +1,11 @@ +linux-nvidia-bos (7.0.0-2016.16) UNRELEASED; urgency=medium + + CHANGELOG: Do not edit directly. Autogenerated at release. + CHANGELOG: Use the printchanges target to see the current changes. + CHANGELOG: Use the insertchanges target to create the final log. + + -- Jacob Martin Wed, 22 Jul 2026 09:53:59 -0500 + linux-nvidia-bos (7.0.0-2015.15) resolute; urgency=medium * resolute/linux-nvidia-bos: 7.0.0-2015.15 -proposed tracker (LP: #2160067) From 7e26681930e1d58ed0e6f5406e907aaf654c1815 Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 22 Jul 2026 09:55:48 -0500 Subject: [PATCH 456/464] UBUNTU: link-to-tracker: update tracking bug BugLink: https://bugs.launchpad.net/bugs/2161467 Properties: no-test-build Signed-off-by: Jacob Martin --- debian.nvidia-bos/tracking-bug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian.nvidia-bos/tracking-bug b/debian.nvidia-bos/tracking-bug index e589c9c126cd0..0f6a28f89b498 100644 --- a/debian.nvidia-bos/tracking-bug +++ b/debian.nvidia-bos/tracking-bug @@ -1 +1 @@ -2160067 d2026.06.30-2 +2161467 d2026.07.21-1 From 27bce41aa9ab0964dad6aa10f9af8175ff26bfbc Mon Sep 17 00:00:00 2001 From: Jacob Martin Date: Wed, 22 Jul 2026 10:02:25 -0500 Subject: [PATCH 457/464] UBUNTU: Ubuntu-nvidia-bos-7.0.0-2016.16 Signed-off-by: Jacob Martin --- debian.nvidia-bos/changelog | 3130 ++++++++++++++++++++++++++++++++- debian.nvidia-bos/reconstruct | 6 + 2 files changed, 3131 insertions(+), 5 deletions(-) diff --git a/debian.nvidia-bos/changelog b/debian.nvidia-bos/changelog index 283fa90227ef4..9bd181ac73479 100644 --- a/debian.nvidia-bos/changelog +++ b/debian.nvidia-bos/changelog @@ -1,10 +1,3130 @@ -linux-nvidia-bos (7.0.0-2016.16) UNRELEASED; urgency=medium +linux-nvidia-bos (7.0.0-2016.16) resolute; urgency=medium - CHANGELOG: Do not edit directly. Autogenerated at release. - CHANGELOG: Use the printchanges target to see the current changes. - CHANGELOG: Use the insertchanges target to create the final log. + * resolute/linux-nvidia-bos: 7.0.0-2016.16 -proposed tracker (LP: #2161467) - -- Jacob Martin Wed, 22 Jul 2026 09:53:59 -0500 + [ Ubuntu: 7.0.0-29.29 ] + + * resolute/linux: 7.0.0-29.29 -proposed tracker (LP: #2160781) + * Packaging resync (LP: #1786013) + - [Packaging] update annotations scripts + * CVE-2026-46331 + - net/sched: fix pedit partial COW leading to page cache corruption + * CVE-2026-53212 + - netfilter: nft_tunnel: fix use-after-free on object destroy + * CVE-2026-53359 + - KVM: x86: Fix shadow paging use-after-free due to unexpected role + * CVE-2026-53131 + - netfilter: require Ethernet MAC header before using eth_hdr() + * CVE-2026-53151 + - rxrpc: Fix the ACK parser to extract the SACK table for parsing + * CVE-2026-53175 + - inet: frags: fix use-after-free caused by the fqdir_pre_exit() flush + * CVE-2026-53176 + - IB/isert: Reject login PDUs shorter than ISER_HEADERS_LEN + * CVE-2026-53186 + - RDMA/srp: bound SRP_RSP sense copy by the received length + * CVE-2026-53215 + - net: mvpp2: refill RX buffers before XDP or skb use + * CVE-2026-53216 + - net: mvpp2: limit XDP frame size to the RX buffer + * CVE-2026-53221 + - ip6_vti: fix incorrect tunnel matching in vti6_tnl_lookup() + * CVE-2026-53224 + - sctp: validate embedded INIT chunk and address list lengths in cookie + * CVE-2026-53225 + - sctp: fix uninit-value in __sctp_rcv_asconf_lookup() + * CVE-2026-53228 + - ipv6: sit: reload inner IPv6 header after GSO offloads + * CVE-2026-52924 + - sctp: purge outqueue on stale COOKIE-ECHO handling + * CVE-2026-53246 + - sctp: validate cached peer INIT chunk length in COOKIE_ECHO processing + * CVE-2026-53247 + - net: ethernet: mtk_eth_soc: Fix use-after-free in metadata dst teardown + * CVE-2026-53260 + - tcp: Add preempt_{disable,enable}_nested() in reqsk_queue_hash_req(). + * Performance regression causes SDXL inference slowdown (~42x) (LP: #2158267) + - drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages + + [ Ubuntu: 7.0.0-28.28 ] + + * resolute/linux: 7.0.0-28.28 -proposed tracker (LP: #2157520) + * Backport ASoC SDCA, AMD SoundWire, and RT722 audio fixes (LP: #2154418) + - ASoC: amd: acp-sdw-legacy: rename the dmic component name + - SAUCE: ASoC: rt722-sdca: add FU06 Playback Switch for speaker mute + control + * MIPI camera of a BBG809N3A_B sensor SKU of the DELL Pro 14 Premium PA14260 + renders upside-down (LP: #2155837) + - SAUCE: media: ipu-bridge: correct platform handling for DELL Pro 14 + Premium PA14260 + * resolute ubuntu_kernel_selftests:seccomp_build test compilation issue + (LP: #2154174) + - SAUCE: selftests/seccomp fix compilation issue for amd64 + * [Ubuntu 26.04] Severe Performance Degradation on kernel 7.0.0-15 + (LP: #2154748) + - s390: Remove GENERIC_LOCKBREAK Kconfig option + - UBUNTU [Config]: Remove GENERIC_LOCKBREAK in s390x + * Fix no sound output device on Dell GhostRider PTL no camera SKU + (LP: #2156559) + - ASoC: Intel: sof_sdw: append dai type to dai link name unconditionally + * Fix no audio from right built-in speaker on HP ZBook with TAS2781 + amplifier (LP: #2156556) + - ALSA: hda/tas2781: Fix device-0 reset issue and handle -EXDEV in block + data processing + * Installer fails internally with a RSync error due to page fault + (LP: #2150640) + - ovl: keep err zero after successful ovl_cache_get() + * Internal display black screen on Intel Lunar Lake with eDP panel + (LP: #2156312) + - drm/i915/alpm: Allow LOBF only for platform that have Always on VRR TG + * Thunderbolt DP tunnel torn down during boot with LUKS Full Disk Encryption + (LP: #2155096) + - SAUCE: thunderbolt: Defer DP tunnel teardown until display driver is + ready + * watchdog: lenovo_se10_wdt: Add SE10 Gen 2 support (LP: #2154715) + - watchdog: lenovo_se10_wdt: Add support for SE10 Gen 2 platform + - watchdog: lenovo_se10_wdt: Fix use-after-free and resource leak risk + * [Ubuntu 26.04] KVM: IBM Test Accelerator for Z (TAZ) not working properly + (LP: #2153159) + - KVM: s390: only deliver service interrupt with payload + - KVM: s390: vsie: Allow non-zarch guests + - KVM: s390: vsie: Disable some bits when in ESA mode + - KVM: s390: vsie: Accommodate ESA prefix pages + - KVM: s390: Add KVM capability for ESA mode guests + * ubuntu_bpf failed to build on Resolute (error: expected ‘:’, ‘,’, ‘;’, ‘}’ + or ‘__attribute__’ before ‘__counted_by’) (LP: #2150071) + - tools/headers: Regenerate stddef.h to fix BPF selftests + * ubuntu_bpf: FTBFS with clang 23 / gcc 15 (LP: #2154343) + - selftests/bpf: Fix const qualifier warning in fexit_bpf2bpf.c + * ALSA: hda/tas2781 Audio Fix for the front-right speacker (LP: #2149770) + - ALSA: hda/tas2781: Fix sound abnormal issue on some SPI device + * Fix bad audio record quality when volume above 50% on Framework PTL + (LP: #2153155) + - ALSA: hda/realtek: fix mic boost on Framework PTL + * Patchset for TUXEDO devices (LP: #2152570) + - drm/i915/vbt: Add edp pipe joiner enable/disable bits + - drm/i915/dp: Avoid joiner for eDP if not enabled in VBT + - drm/amd/display: Add Idle state manager(ISM) + - drm/i915/backlight: Remove try_vesa_interface + - drm/i915/backlight: Use intel_panel variable instead of intel_connector + - drm/i915/backlight: Take luminance_set into account for VESA backlight + - drm/i915/backlight: Check luminance_set when disabling PWM via AUX VESA + backlight + - drm/i915/backlight: Short circuit intel_dp_aux_supports_hdr_backlight + - drm/i915/backlight: Update debug log during backlight setup + - drm/i915/backlight: Check if VESA backlight is possible + - drm/i915/backlight: Provide clear description on how backlight level is + controlled + - drm/i915/backlight: Fix VESA backlight possible check condition + * Resolute update: v7.0.12 upstream stable release (LP: #2156636) + - Input: usbtouchscreen - clamp NEXIO data_len/x_len to URB buffer size + - ACPI: button: Fix ACPI GPE handler leak during removal + - ACPI: button: Enable wakeup GPEs for ACPI buttons at probe time + - xfrm: move policy_bydst RCU sync from per-netns .exit to .pre_exit + - net/sched: sch_sfb: Replace direct dequeue call with peek and + qdisc_dequeue_peeked + - bcache: fix uninitialized closure object + - nfc: llcp: Fix use-after-free in llcp_sock_release() + - nfc: llcp: Fix use-after-free race in nfc_llcp_recv_cc() + - xfrm: Check for underflow in xfrm_state_mtu + - nfc: nxp-nci: i2c: use rising-edge IRQ on ACPI systems + - tools/bootconfig: Fix buf leaks in apply_xbc + - HID: remove duplicate hid_warn_ratelimited definition + - kunit: fix use-after-free in debugfs when using kunit.filter + - accel/rocket: fix UAF via dangling GEM handle in create_bo + - netfilter: synproxy: refresh tcphdr after skb_ensure_writable + - netfilter: xt_cpu: prefer raw_smp_processor_id + - netfilter: ebtables: fix OOB read in compat_mtw_from_user + - netfilter: nf_tables: fix dst corruption in same register operation + - vsock: keep poll shutdown state consistent + - net: netlink: fix sending unassigned nsid after assigned one + - net: netlink: don't set nsid on local notifications + - net/smc: Do not re-initialize smc hashtables + - net/iucv: fix locking in .getsockopt + - scsi: core: Run queues for all non-SDEV_DEL devices from + scsi_run_host_queues + - scsi: scsi_debug: Add missing newline in scsi_debug_device_reset() + - ipv4: free net->ipv4.sysctl_local_reserved_ports after + unregister_net_sysctl_table() + - ALSA: hda: cs35l56: Fix system name string leaks + - ALSA: pcm: oss: Fix setup list UAF on proc write error + - ASoC: Intel: bytcht_es8316: Fix MCLK leak on init errors + - net/mlx5: HWS: Reject unsupported remove-header action + - net: hsr: fix potential OOB access in supervision frame handling + - accel/ivpu: prevent uninitialized data bug in debugfs + - gpio: mxc: fix irq_high handling + - drm/i915/aux: use polling when irqs are unavailable + - net: Avoid checksumming unreadable skb tail on trim + - ethtool: rss: avoid modifying the RSS context response + - ethtool: rss: add missing errno on RSS context delete + - ethtool: rss: fix falsely ignoring indir table updates + - ethtool: rss: fix indir_table and hkey leak on get_rxfh failure + - ethtool: rss: fix hkey leak when indir_size is 0 + - ethtool: rss: avoid device context leak on reply-build failure + - ethtool: module: call ethnl_ops_complete() on module flash errors + - ethtool: module: avoid leaking a netdev ref on module flash errors + - ethtool: module: avoid racy updates to dev->ethtool bitfield + - ethtool: module: check fw_flash_in_progress under rtnl_lock + - ethtool: module: fix cleanup if socket used for flashing multiple + devices + - ethtool: cmis: require exact CDB reply length + - ethtool: cmis: fix u16-to-u8 truncation of msleep_pre_rpl + - ethtool: cmis: validate start_cmd_payload_size from module + - ethtool: cmis: validate fw->size against start_cmd_payload_size + - cxl/test: Update mock dev array before calling platform_device_add() + - blk-mq: reinsert cached request to the list + - tunnels: load network headers after skb_cow() in + iptunnel_pmtud_build_icmp[v6]() + - vxlan: do not reuse cached ip_hdr() value after skb_tunnel_check_pmtu() + - tunnels: do not assume transport header in iptunnel_pmtud_check_icmp() + - ksmbd: fix FSCTL permission bypass by adding a permission check for + FSCTL_SET_SPARSE + - ASoC: codecs: simple-mux: Fix enum control bounds check + - drm/xe: Restore IDLEDLY regiter on engine reset + - Bluetooth: 6lowpan: check skb_clone() return value in send_mcast_pkt() + - bonding: refuse to enslave CAN devices + - bridge: Fix sleep in atomic context in netlink path + - bridge: Fix sleep in atomic context in sysfs path + - ethtool: coalesce: cap profile updates at NET_DIM_PARAMS_NUM_PROFILES + - ethtool: tsconfig: fix reply error handling + - ethtool: linkstate: fix unbalanced ethnl_ops_complete() on PHY lookup + error + - ethtool: pse-pd: fix missing ethnl_ops_complete() + - ethtool: tsconfig: fix missing ethnl_ops_complete() + - ethtool: tsinfo: fix uninitialized stats on the by-PHC path + - ethtool: tsinfo: don't pass ERR_PTR to genlmsg_cancel on prepare failure + - ethtool: strset: fix header attribute index in ethnl_req_get_phydev() + - ethtool: eeprom: add missing ethnl_ops_begin() / _complete() during + fallback + - ethtool: eeprom: add more safeties to EEPROM Netlink fallback + - ipv6: rpl: fix hdrlen overflow in ipv6_rpl_srh_decompress() + - net/sched: Revert "net/sched: Restrict conditions for adding duplicating + netems to qdisc tree" + - net/sched: fix packet loop on netem when duplicate is on + - net: Introduce skb tc depth field to track packet loops + - net/sched: Fix ethx:ingress -> ethy:egress -> ethx:ingress mirred loop + - net/sched: act_mirred: Fix blockcast recursion bypass leading to stack + overflow + - net/sched: act_mirred: Fix return code in early mirred redirect error + paths + - net: hibmcge: disable Relaxed Ordering to fix RX packet corruption + - net: hibmcge: move dma_rmb() after dma_sync_single_for_cpu() in RX path + - net/handshake: Use spin_lock_bh for hn_lock + - nvme-tcp: store negative errno in queue->tls_err + - net/handshake: Pass negative errno through handshake_complete() + - net/handshake: hand off the pinned file reference to accept_doit + - net/handshake: Take a long-lived file reference at submit + - net/handshake: Drain pending requests at net namespace exit + - dpll: zl3073x: detect DPLL channel count from chip ID at runtime + - dpll: zl3073x: add die temperature reporting for supported chips + - dpll: export __dpll_device_change_ntf() for use under dpll_lock + - dpll: zl3073x: use __dpll_device_change_ntf() and remove change_work + - Bluetooth: l2cap: clear chan->ident on ECRED reconfiguration success + - Bluetooth: L2CAP: Fix possible crash on l2cap_ecred_conn_rsp + - Bluetooth: hci_sync: Set HCI_CMD_DRAIN_WORKQUEUE during device close + - Bluetooth: hci_sync: Reset device counters in hci_dev_close_sync() + - gpio: adnp: fix flow control regression caused by scoped_guard() + - gpio: virtuser: Fix uninitialized data bug in + gpio_virtuser_direction_do_write() + - gpio: rockchip: convert bank->clk to devm_clk_get_enabled() + - gpio: rockchip: teardown bugs and resource leaks + - net: mana: Add NULL guards in teardown path to prevent panic on attach + failure + - net: mana: Skip redundant detach on already-detached port + - sctp: fix race between sctp_wait_for_connect and peeloff + - net: pcs: pcs-mtk-lynxi: fix bpi-r3 serdes configuration + - vsock/virtio: bind uarg before filling zerocopy skb + - ipv6: fix possible infinite loop in rt6_fill_node() + - ipv6: fix possible infinite loop in fib6_select_path() + - net: skbuff: fix pskb_carve leaking zcopy pages + - Revert "ipv6: preserve insertion order for same-scope addresses" + - Revert "x86/fpu: Refine and simplify the magic number check during + signal return" + - drm/i915/psr: Add defininitions for INTEL_WA_REGISTER_CAPS DPCD register + - drm/i915/psr: Read Intel DPCD workaround register + - drm/i915/psr: Apply Intel DPCD workaround when SDP on prior line used + - iio: imu: st_lsm6dsx: fix stack leak in tagged FIFO buffer + - iio: imu: adis16550: fix stack leak in trigger handler + - iio: pressure: bmp280: fix stack leak in bmp580 trigger handler + - usb: typec: ucsi: ccg: reject firmware images without a ':' record + header + - usb: typec: tcpm: validate VDO count in Discover Identity ACK handlers + - usb: typec: tcpm: bound altmode_desc[] per iteration in + svdm_consume_modes() + - usb: typec: ucsi: displayport: NAK DP_CMD_CONFIGURE without a payload + VDO + - usb: typec: altmodes/displayport: validate count before reading Status + Update VDO + - usb: typec: wcove: don't write past struct pd_message in + wcove_read_rx_buffer() + - usb: typec: tcpm/tcpci_maxim: validate header NDO against RX_BYTE_CNT + - usb: typec: ucsi: validate connector number in ucsi_connector_change() + - USB: serial: safe_serial: fix memory corruption with small endpoint + - media: rc: igorplugusb: fix control request setup packet + - Input: ims-pcu - fix usb_free_coherent() size in ims_pcu_buffers_free() + - USB: serial: cypress_m8: fix memory corruption with small endpoint + - HID: quirks: Add ALWAYS_POLL quirk for SIGMACHIP USB mouse + - Bluetooth: btusb: Allow firmware re-download when version matches + - mm/vmalloc: do not trigger BUG() on BH disabled context + - hpfs: fix a crash if hpfs_map_dnode_bitmap fails + - mm/damon/sysfs-schemes: delete tried region in regions_rmdirs() + - ipc: limit next_id allocation to the valid ID range + - mm: memcontrol: propagate NMI slab stats to memcg vmstats + - mm/migrate_device: fix pgtable leak in migrate_vma_insert_huge_pmd_page + - memfd: deny writeable mappings when implying SEAL_WRITE + - zram: fix use-after-free in zram_writeback_endio + - mm/rmap: initialize nr_pages to 1 at loop start in try_to_unmap_one + - auxdisplay: line-display: fix OOB read on zero-length message_store() + - smb: client: fix uninitialized variable in smb2_writev_callback + - Bluetooth: L2CAP: use chan timer to close channels in cleanup_listen() + - Bluetooth: L2CAP: fix chan ref leak in l2cap_chan_timeout() on !conn + - Bluetooth: HIDP: fix missing length checks in hidp_input_report() + - Bluetooth: ISO: fix UAF in iso_recv_frame + - Bluetooth: ISO: serialize iso_sock_clear_timer with socket lock + - Bluetooth: hci_conn: Fix memory leak in hci_le_big_terminate() + - Bluetooth: hci_qca: Use 100 ms SSR delay for rampatch and NVM loading + - Bluetooth: hci_sync: fix UAF in hci_le_create_cis_sync + - Input: xpad - fix out-of-bounds access for Share button + - parport: Fix race between port and client registration + - rust_binder: Avoid holding lock when dropping delivered_death + - rust_binder: avoid calling pending_oneway_finished() on TF_UPDATE_TXN + - USB: cdc-acm: Fix bit overlap and move quirk definitions to header + - KVM: arm64: Correctly cap ZCR_EL2 provided by a guest hypervisor + - KVM: arm64: PMU: Preserve AArch32 counter low bits + - KVM: SVM: Flush the current TLB when transitioning from xAVIC => x2AVIC + - KVM: SEV: Require in-GHCB scratch area if GHCB v2+ is in use + - KVM: SEV: Ignore Port I/O requests of length '0' + - KVM: SEV: Use the size of the PSC header as the minimum size for PSC + requests + - KVM: SEV: WARN if KVM attempts to setup scratch area with min_len==0 + - KVM: SEV: Compute the correct max length of the in-GHCB scratch area + - KVM: SEV: Check PSC request indices against the actual size of the + buffer + - KVM: SEV: Use READ_ONCE() when reading entries/indices from PSC buffer + - KVM: SEV: Don't explicitly pass PSC buffer to snp_begin_psc() + - gpio: shared: undo the vote of the proxy on GPIO free + - gpio: shared: fix deadlock on shared proxy's parent removal + - gpio: shared: fix lockdep false positive by removing unneeded lock + - Disable -Wattribute-alias for clang-23 and newer + - iio: adc: xilinx-xadc: Fix sequencer mode in postdisable for dual mux + - iio: adc: npcm: fix unbalanced clk_disable_unprepare() + - iio: dac: ad3530r: Fix AD3531/AD3531R powerdown mode strings + - iio: dac: max5821: fix return value check in powerdown sync + - iio: dac: ad5686: fix ref bit initialization for single-channel parts + - iio: dac: ad5686: fix input raw value check + - iio: dac: ad5686: acquire lock when doing powerdown control + - iio: dac: ad5686: fix powerdown control on dual-channel devices + - iio: adc: mt6359: fix unchecked return value in mt6358_read_imp + - iio: adc: viperboard: Fix error handling in vprbrd_iio_read_raw + - iio: adc: ad4695: Fix call ordering in offload buffer postenable + - iio: adc: nxp-sar-adc: fix division by zero in write_raw + - iio: adc: nxp-sar-adc: Avoid division by zero + - iio: adc: nxp-sar-adc: zero-initialize dma_slave_config + - iio: gyro: itg3200: fix i2c read into the wrong stack location + - iio: gyro: adis16260: fix division by zero in write_raw + - iio: ssp_sensors: cancel delayed work_refresh on remove + - iio: temperature: tsys01: fix broken PROM checksum validation + - iio: magnetometer: st_magn: fix default DRDY pin selection for LIS2MDL + - iio: light: veml6070: Fix resource leak in probe error path + - iio: Fix iio_multiply_value use in iio_read_channel_processed_scale + - iio: chemical: mhz19b: reject oversized serial replies + - iio: chemical: scd30: fix division by zero in write_raw + - iio: light: cm3323: fix reg_conf not being initialized correctly + - iio: buffer: hw-consumer: fix use-after-free in error path + - iio: buffer: Fix DMA fence leak in iio_buffer_enqueue_dmabuf() + - USB: serial: omninet: fix memory corruption with small endpoint + - usb: cdns3: gadget: fix request skipping after clearing halt + - usb: cdns3: plat: fix leaked usb2_phy initialization on usb3_phy + acquisition failure + - usb: cdns3: plat: fix unbalanced pm_runtime_forbid() call permanently + leaks the runtime PM usage counter across bind/unbind cycles + - usb: dwc2: Fix use after free in debug code + - Input: elan_i2c - validate firmware size before use + - i2c: davinci: fix division by zero on missing clock-frequency + - x86/ftrace: Relocate %rip-relative percpu refs in dynamic trampolines + - wireguard: send: append trailer after expanding head + - bpf: sockmap: fix tail fragment offset in bpf_msg_push_data + - macsec: fix replay protection at XPN lower-PN wrap + - ipv6: exthdrs: refresh nh pointer after ipv6_hop_jumbo() + - ASoC: qcom: q6asm-dai: fix error handling in prepare and set_params + - octeontx2-af: validate body pcifunc in rvu_mbox_handler_rep_event_notify + - ipv6: exthdrs: refresh nh after handling HAO option + - ip6: vti: Use ip6_tnl.net in vti6_siocdevprivate(). + - ipv6: validate extension header length before copying to cmsg + - xfrm: input: hold netns during deferred transport reinjection + - l2tp: use refcount_inc_not_zero in l2tp_session_get_by_ifname + - ip6: vti: Use ip6_tnl.net in vti6_changelink(). + - net: skbuff: fix missing zerocopy reference in pskb_carve helpers + - spi: spi-mem: avoid mutating op template in spi_mem_supports_op() + - HID: wacom: Fix OOB write in wacom_hid_set_device_mode() + - iommu, debugobjects: avoid gcc-16.1 section mismatch warnings + - nfc: hci: fix out-of-bounds read in HCP header parsing + - xfrm: route MIGRATE notifications to caller's netns + - xfrm: ipcomp: Free destination pages on acomp errors + - xfrm: ah: use skb_to_full_sk in async output callbacks + - ALSA: scarlett2: Fix 2i2 Gen 4 direct monitor gain on firmware 2417 + - ALSA: firewire-motu: Protect register DSP event queue positions + - netfilter: conntrack: tcp: do not force CLOSE on invalid-seq RST without + direction check + - ASoC: qcom: q6asm-dai: close stream only when running + - ASoC: qcom: q6asm-dai: do not set stream state in event and trigger + callbacks + - xfrm: esp: restore combined single-frag length gate + - ALSA: hda/realtek: Fix speaker output on ASUS ROG Strix G615LP + - xfrm: iptfs: reset runtime state when cloning SAs + - dma-buf: fix UAF in dma_buf_fd() tracepoint + - Input: xpad - add "Nova 2 Lite" from GameSir + - Input: xpad - add support for ASUS ROG RAIKIRI II + - ksmbd: OOB read regression in smb_check_perm_dacl() ACE-walk loops + - misc: rp1: Send IACK on IRQ activate to fix kdump/kexec + - Input: atmel_mxt_ts - fix boundary check in mxt_prepare_cfg_mem + - Input: synaptics - add LEN2058 to SMBus passlist for ThinkPad E490 + - dt-bindings: usb: Fix EIC7700 USB reset's issue + - comedi: comedi_test: fix check for valid scan_begin_src in + waveform_ai_cmdtest() + - comedi: comedi_test: Fix limiting of convert_arg in + waveform_ai_cmdtest() + - counter: Fix refcount leak in counter_alloc() error path + - tty: serial: pch_uart: add check for dma_alloc_coherent() + - tty: serial: samsung: Remove redundant port lock acquisition in rx + helpers + - uio: uio_pci_generic_sva: fix double free of devm_kzalloc() memory + - usb: chipidea: core: convert ci_role_switch to local variable + - usb: core: Fix up Interrupt IN endpoints with bogus wBytesPerInterval + - usb: dwc3: xilinx: fix error handling in zynqmp init error paths + - usb: musb: omap2430: Fix use-after-free in omap2430_probe() + - USB: quirks: add NO_LPM for Lenovo ThinkPad USB-C Dock Gen2 hub + controllers + - usb: storage: Add quirks for PNY Elite Portable SSD + - usbip: vudc: Fix use after free bug in vudc_remove due to race condition + - usb: usbtmc: check URB actual_length for interrupt-IN notifications + - usb: usbtmc: reject interrupt endpoints with small wMaxPacketSize + - usb: typec: tipd: Fix error code in tps6598x_probe() + - usb: typec: tcpm: improve handling of DISCOVER_MODES failures + - usb: typec: ucsi: Check if power role change actually happened before + handling + - usb: typec: ucsi: Don't update power_supply on power role change if not + connected + - USB: serial: option: add MeiG SRM813Q + - USB: serial: option: add missing RSVD(5) flag for Rolling RW135R-GL + - USB: serial: belkin_sa: validate interrupt status length + - USB: serial: cypress_m8: validate interrupt packet headers + - USB: serial: digi_acceleport: fix memory corruption with small endpoints + - USB: serial: keyspan: fix missing indat transfer sanity check + - USB: serial: mxuport: fix memory corruption with small endpoint + - USB: serial: mct_u232: fix memory corruption with small endpoint + - USB: serial: mct_u232: fix missing interrupt-in transfer sanity check + - usb: gadget: uvc: hold opts->lock across XU walks in uvc_function_bind + - usb: gadget: net2280: Fix double free in probe error path + - usb: gadget: f_hid: fix device reference leak in hidg_alloc() + - usb: gadget: composite: fix integer underflow in WebUSB GET_URL handling + - usb: gadget: dummy_hcd: Reject hub port requests for non-existent ports + - usb: gadget: f_fs: copy only received bytes on short ep0 read + - usb: gadget: f_fs: serialize DMABUF cancel against request completion + - thunderbolt: property: Reject u32 wrap in tb_property_entry_valid() + - thunderbolt: property: Reject dir_len < 4 to prevent size_t underflow + - thunderbolt: property: Cap recursion depth in __tb_property_parse_dir() + - scsi: fcoe: Reject FIP descriptors with zero fip_dlen in CVL walker + - scsi: scsi_transport_fc: Widen FPIN pname walker counter to u32 + - scsi: target: iscsi: Fix CRC overread and double-free in + iscsit_handle_text_cmd() + - scsi: target: iscsi: Bound iscsi_encode_text_output() appends to rsp_buf + - scsi: target: iscsi: Validate CHAP_R length before base64 decode + - drm/hyperv: validate resolution_count and fix WIN8 fallback + - drm/hyperv: validate VMBus packet size in receive callback + - drm/gem: fix race between change_handle and handle_delete + - drm/i915/color: Fix HDR pre-CSC LUT programming loop + - drm/i915/psr: Block DC states on vblank enable when Panel Replay + supported + - drm/i915/psr: Use DC_OFF wake reference to block DC6 on vblank enable + - drm/i915: Fix potential UAF in TTM object purge + - drm/amd/pm/si: Disregard vblank time when no displays are connected + - serial: altera_jtaguart: handle uart_add_one_port() failures + - serial: qcom-geni: fix UART_RX_PAR_EN bit position + - serial: qcom_geni: fix kfifo underflow when flush precedes DMA + completion IRQ + - serial: sh-sci: fix memory region release in error path + - serial: zs: Fix swapped RI/DSR modem line transition counting + - serial: fsl_lpuart: fix rx buffer and DMA map leaks in start_rx_dma + - drm/amdkfd: fix NULL pointer bug in svm_range_set_attr + - drm/amdkfd: fix a vulnerability of integer overflow in kfd debugger + - drm/amdkfd: Check for pdd drm file first in CRIU restore path + - drm/amdgpu: fix lock leak on ENOMEM in AMDGPU_GEM_OP_GET_MAPPING_INFO + - drm/amdgpu: fix calling VM invalidation in amdgpu_hmm_invalidate_gfx + - drm/amdgpu: fix amdgpu_hmm_range_get_pages + - drm/amdgpu: check num_entries in GEM_OP GET_MAPPING_INFO + - serial: dz: Fix bootconsole message clobbering at chip reset + - serial: dz: Fix bootconsole handover lockup + - serial: dz: Convert to use a platform device + - serial: zs: Fix bootconsole handover lockup + - serial: zs: Switch to using channel reset + - serial: zs: Convert to use a platform device + - serial: core: introduce guard(uart_port_lock_check_sysrq_irqsave) + - serial: 8250: dispatch SysRq character in serial8250_handle_irq() + - serial: 8250_dw: dispatch SysRq character in dw8250_handle_irq() + - platform/x86/intel/vsec: Refactor base_addr handling + - platform/x86/intel/vsec: Make driver_data info const + - platform/x86/intel/vsec: Fix enable_cnt imbalance on PCIe error recovery + - rxrpc: Fix RESPONSE packet verification to extract skb to a linear + buffer + - ALSA: hda/realtek: Fix mute and mic-mute LEDs for HP Envy X360 15-fh0xxx + - ALSA: hda/realtek: Fix mute and mic-mute LEDs for HP 16 Piston OmniBook + X + - arm64: tlb: Flush walk cache when unsharing PMD tables + - i2c: tegra: make tegra_i2c_mutex_unlock() return void + - hwmon: (pmbus) Add support for guarded PMBus lock + - hwmon: (pmbus/adm1266) serialize sequencer_state debugfs read with + pmbus_lock + - hwmon: (pmbus/adm1266) serialize GPIO PMBus accesses with pmbus_lock + - net: phy: micrel: fix LAN8814 QSGMII soft reset + - xhci: tegra: Fix ghost USB device on dual-role port unplug + - mailbox: Fix NULL message support in mbox_send_message() + - usb: core: Fix SuperSpeed root hub wMaxPacketSize + - tools: ynl: add scope qualifier for definitions + - Linux 7.0.12 + * Resolute update: v7.0.12 upstream stable release (LP: #2156636) // + CVE-2026-46318 + - Revert "mm/hugetlbfs: update hugetlbfs to use mmap_prepare" + * Resolute update: v7.0.12 upstream stable release (LP: #2156636) // + CVE-2026-46322 + - tun: free page on build_skb failure in tun_xdp_one() + * Resolute update: v7.0.12 upstream stable release (LP: #2156636) // + CVE-2026-46320 + - tap: free page on error paths in tap_get_user_xdp() + * Resolute update: v7.0.12 upstream stable release (LP: #2156636) // + CVE-2026-46321 + - tun: free page on short-frame rejection in tun_xdp_one() + * CVE-2026-46316 // CVE-2026-46317 + - KVM: arm64: Reassign nested_mmus array behind mmu_lock + * CVE-2026-46316 + - KVM: arm64: vgic-its: Drop the translation cache reference only for the + erased entry + - KVM: arm64: Take the SRCU lock for page table walks in fault injection + and AT emulation + * Resolute update: v7.0.11 upstream stable release (LP: #2156390) + - iommu/amd: Fix illegal cap/mmio access in IOMMU debugfs + - iommu/amd: Remove latent out-of-bounds access in IOMMU debugfs + - ksmbd: close durable scavenger races against m_fp_list lookups + - ata: libata-scsi: improve readability of ata_scsi_qc_issue() + - ata: libata-scsi: do not use the deferred QC feature for ATA_DEFER_PORT + - ata: libata-scsi: do not use the deferred QC feature on PMPs with CBS + - ata: libata-scsi: do not needlessly defer commands when using PMP with + FBS + - sysfs: don't remove existing directory on update failure + - mm/damon/sysfs-schemes: call missing mem_cgroup_iter_break() + - ksmbd: fix null pointer dereference in compare_guid_key() + - ksmbd: fix null pointer dereference in proc_show_files() + - ksmbd: fix SID memory leak in set_posix_acl_entries_dacl() on overflow + - ksmbd: validate SID in parent security descriptor during ACL inheritance + - regulator: tps65219: fix irq_data.rdev not being assigned + - x86/mm: Disable broadcast TLB flush when PCID is disabled + - scripts/gdb: mm: cast untyped symbols in x86_page_ops + - smb: client: require net admin for CIFS SWN netlink + - smb: client: protect tc_count increment in + smb2_find_smb_sess_tcon_unlocked() + - smb: client: use data_len for SMB2 READ encrypted folioq copy + - smb/server: promote S_DEL_ON_CLS to S_DEL_PENDING when close + - hwmon: (pmbus/adm1266) widen blackbox-info buffer to I2C_SMBUS_BLOCK_MAX + - ALSA: ua101: Reject too-short USB descriptors + - ALSA: pcm: Don't setup bogus iov_iter for silencing + - ALSA: asihpi: Fix potential OOB array access at reading cache + - ALSA: scarlett2: Allow flash writes ending at segment boundary + - ACPI: battery: Fix system wakeup on critical battery status + - efi: Allocate runtime workqueue before ACPI init + - spi: amd: Set correct bus number in ACPI probe path + - io_uring/waitid: clear waitid info before copying it to userspace + - drivers/base/memory: fix memory block reference leak in poison + accounting + - ipv6: ioam: refresh hdr pointer before ioam6_event() + - mm/memory: fix spurious warning when unmapping device-private/exclusive + pages + - mm: fix __vm_normal_page() to handle missing support for + pmd_special()/pud_special() + - mm/memory_hotplug: fix memory block reference leak on remove + - mm/page_alloc: fix initialization of tags of the huge zero folio with + init_on_free + - mm/migrate_device: fix spinlock leak in migrate_vma_insert_huge_pmd_page + - selftests/mm: run_vmtests.sh: fix destructive tests invocation + - mm/damon: fix damos_stat tracepoint format for sz_applied + - net: wwan: iosm: fix potential memory leaks in ipc_imem_init() + - Bluetooth: fix UAF in l2cap_sock_cleanup_listen() vs l2cap_conn_del() + - Bluetooth: ISO: drop ISO_END frames received without prior ISO_START + - Bluetooth: bnep: Fix UAF read of dev->name + - Bluetooth: hci_uart: fix UAFs and race conditions in close and init + paths + - Bluetooth: L2CAP: ecred_reconfigure: send packed pdu, not stack pointer + - Bluetooth: hci_qca: Convert timeout from jiffies to ms + - Bluetooth: MGMT: validate Add Extended Advertising Data length + - Bluetooth: serialize accept_q access + - phonet/pep: disable BH around forwarded sk_receive_skb() + - net: bcmgenet: keep RBUF EEE/PM disabled + - net: devmem: reject dma-buf bind with non-page-aligned size or SG length + - net: phy: skip EEE advertisement write when autoneg is disabled + - net: hsr: defer node table free until after RCU readers + - net/mlx5e: Fix use-after-free in mlx5e_tx_reporter_timeout_recover + - net: ifb: report ethtool stats over num_tx_queues + - net: pse-pd: fix sign on -ENOENT check in of_load_pse_pis() + - netfilter: ip6t_hbh: reject oversized option lists + - netfilter: nf_queue: hold bridge skb->dev while queued + - netfilter: ipset: stop hash:* range iteration at end + - net: ethtool: fix NULL pointer dereference in phy_reply_size + - net: ethtool: phy: avoid NULL deref when PHY driver is unbound + - ACPI: driver: Check ACPI_COMPANION() against NULL during probe + - sched_ext: Fix missing warning in scx_set_task_state() default case + - sched_ext: Avoid UAF in scx_root_enable_workfn() init failure path + - l2tp: use list_del_rcu in l2tp_session_unhash + - qed: fix double free in qed_cxt_tables_alloc() + - ring-buffer: Fix reporting of missed events in iterator + - ring-buffer: Flush and stop persistent ring buffer on panic + - wifi: mac80211: capture fast-RX rate before mesh reuses skb->cb + - ipv6: ioam: add NULL check for idev in ipv6_hop_ioam() + - selftests: mptcp: drop nanoseconds width specifier + - mptcp: pm: fix ADD_ADDR timer infinite retry on option space + insufficient + - vsock/vmci: fix UAF when peer resets connection during handshake + - vsock/virtio: reset connection on receiving queue overflow + - ice: fix VF queue configuration with low MTU values + - wifi: ath11k: clear shared SRNG pointer state on restart + - wifi: iwlwifi: mvm: fix driver-set TX rates on old devices + - wifi: iwlwifi: mld: stop TX during firmware restart + - ipv4: raw: reject IP_HDRINCL packets with ihl < 5 + - ixgbevf: fix use-after-free in VEPA multicast source pruning + - rbd: eliminate a race in lock_dwork draining on unmap + - mptcp: do not drop partial packets + - mptcp: reset rcv wnd on disconnect + - lsm: hold cred_guard_mutex for lsm_set_self_attr() + - octeontx2-af: CGX: add bounds check to cgx_speed_mbps index + - octeontx2-pf: fix double free in rvu_rep_rsrc_init() + - igc: fix potential skb leak in igc_fpe_xmit_smd_frame() + - ice: fix locking around wait_event_interruptible_locked_irq + - ice: fix setting promisc mode while adding VID filter + - ice: restore PTP Rx timestamp config after ethtool set-channels + - wifi: cfg80211: advance loop vars in cfg80211_merge_profile() + - af_unix: Fix UAF read of tail->len in unix_stream_data_wait() + - wifi: mac80211: consume only present negotiated TTLM maps + - octeontx2-pf: avoid double free of pool->stack on AQ init failure + - cifs: Fix busy dentry used after unmounting + - tracing: Do not call map->ops->elt_free() if elt_alloc() fails + - ASoC: codecs: pcm512x: fix null-ptr dereference in + pcm512x_overclock_xxx_put() + - arm64: probes: Handle probes on hinted conditional branch instructions + - KVM: arm64: vgic-its: Reject restored DTE with out-of-range + num_eventid_bits + - KVM: arm64: vgic: Free private_irqs when init fails after allocation + - KVM: SVM: Disable AVIC IPI virtualization on Hygon Family 18h (erratum + #1235) + - riscv: kvm: return SBI_ERR_FAILURE for pmu_snapshot_set_shmem() when OOM + - riscv: kvm: return SBI_ERR_FAILURE for pmu_event_info() when OOM + - virt: sev-guest: Explicitly leak pages in unknown state + - i2c: tegra: fix pm_runtime leak on mutex_lock failure + - drm/bridge: chipone-icn6211: use devm_drm_bridge_add in i2c probe + - spi: qup: fix error pointer deref after DMA setup failure + - phy: exynos5-usbdrd: fix USB 2.0 HS PHY tuning values for Exynos7870 + - phy: tegra: xusb: Fix per-pad high-speed termination calibration + - phy: qcom-qmp-ufs: Fix kaanapali PHY PLL lock failure after SM8650 G4 + fix + - phy: qcom: edp: Unify generic DP/eDP swing and pre-emphasis tables + - phy: qcom: edp: Add eDP/DP mode switch support + - phy: qcom: edp: Fix AUX_CFG8 programming for DP mode + - scsi: isci: Fix use-after-free in device removal path + - spi: ep93xx: fix error pointer deref after DMA setup failure + - spi: sprd: fix error pointer deref after DMA setup failure + - spi: ti-qspi: fix use-after-free after DMA setup failure + - mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache() + - RDMA/siw: Reject MPA FPDU length underflow before signed receive math + - s390/cio: Restore GFP_DMA for CHSC allocation + - s390/pai: Disable duplicate read of kernel PAI counter value + - s390/pai: Fix missing PAI counter increments under heavy load + - fwctl: pds: Validate RPC input size before parsing + - LoongArch: kprobes: Use larch_insn_text_copy() to patch instructions + - LoongArch: Remove unused code to avoid build warning + - cpufreq: intel_pstate: Use correct scaling factor on Raptor Lake-E + - device property: set fwnode->secondary to NULL in fwnode_init() + - drm/i915/display: Copy color pipeline from plane in the primary joiner + pipe + - drm/msm: Fix shrinker deadlock + - drm/v3d: Fix use-after-free of CPU job query arrays on error path + - drm/v3d: Release indirect CSD GEM reference on CPU job free + - drm/virtio: use uninterruptible resv lock for plane updates + - drm/xe/multi_queue: Fix secondary queue error case + - drm/amdgpu/vpe: Force collaborate sync after TRAP + - drm/bridge: it66121: acquire reset GPIO in probe + - drm/bridge: megachips: remove bridge when irq request fails + - drm/amd/display: Fix integer overflow in bios_get_image() + - drm/amd/display: Validate GPIO pin LUT table size before iterating + - drm/amd/display: Validate payload length and link_index in + dc_process_dmub_aux_transfer_async + - batman-adv: v: stop OGMv2 on disabled interface + - batman-adv: tvlv: abort OGM send on tvlv append failure + - batman-adv: tvlv: reject oversized TVLV packets + - batman-adv: iv: recover OGM scheduling after forward packet error + - batman-adv: mcast: fix use-after-free in orig_node RCU release + - batman-adv: clear current gateway during teardown + - batman-adv: dat: handle forward allocation error + - batman-adv: fix fragment reassembly length accounting + - batman-adv: fix tp_meter counter underflow during shutdown + - batman-adv: frag: disallow unicast fragment in fragment + - batman-adv: bla: fix report_work leak on backbone_gw purge + - batman-adv: bla: avoid double decrement of bla.num_requests + - batman-adv: bla: avoid NULL-ptr deref for claim via dropped interface + - batman-adv: tp_meter: avoid use of uninit sender vars + - batman-adv: tp_meter: directly shut down timer on cleanup + - batman-adv: tp_meter: fix tp_vars reference leak in receiver shutdown + - batman-adv: tp_meter: fix race condition in send error reporting + - batman-adv: tp_meter: avoid role confusion in tp_list + - batman-adv: tt: fix TOCTOU race for reported vlans + - batman-adv: tt: reject oversized local TVLV buffers + - batman-adv: tt: avoid empty VLAN responses + - batman-adv: tt: fix negative last_changeset_len + - batman-adv: tt: fix negative tt_buff_len + - batman-adv: tt: prevent TVLV entry number overflow + - hwmon: (pmbus/adm1266) seed timestamp from the real-time clock + - hwmon: (pmbus/adm1266) reject implausible blackbox record_count + - hwmon: (pmbus/adm1266) include PEC byte in pmbus_block_xfer read buffer + - hwmon: (pmbus/adm1266) bounce blackbox records through a protocol-sized + buffer + - hwmon: (pmbus/adm1266) cap PDIO scan in get_multiple at ADM1266_PDIO_NR + - hwmon: (pmbus/adm1266) don't clobber GPIO bits before PDIO read in + get_multiple + - hwmon: (pmbus/adm1266) register the gpio_chip after pmbus_do_probe() + - hwmon: (pmbus/adm1266) register the nvmem device after pmbus_do_probe() + - hwmon: (pmbus/adm1266) reject short block-read responses in the GPIO + accessors + - pinctrl: mediatek: moore: implement gpio_chip::get_direction() + - pinctrl: qcom: ipq4019: mark gpio as a GPIO pin function + - arm64: dts: renesas: r8a78000: Fix SCIF brg_int clocks + - ARM: dts: renesas: genmai: Drop superfluous cells + - ARM: dts: renesas: rskrza1: Drop superfluous cells + - pinctrl: renesas: rzg2l: Fix incorrect PUPD register offset for high + pins during suspend/resume + - pinctrl: renesas: rzg2l: Fix SMT register cache handling + - pinctrl: meson: amlogic-a4: fix deadlock issue + - pinctrl: qcom: Fix GPIO to PDC wake irq map for qcs615 + - kho: skip KHO for crash kernel + - mm/memfd_luo: report error when restoring a folio fails mid-loop + - HID: intel-thc-hid: Intel-quickspi: Fix some error codes + - HID: uclogic: Fix regression of input name assignment + - firmware: arm_ffa: Check for NULL FF-A ID table while driver + registration + - firmware: arm_ffa: Skip free_pages on RX buffer alloc failure + - firmware: arm_ffa: Fix per-vcpu self notifications handling in workqueue + - firmware: arm_ffa: Unregister bus notifier on teardown for FF-A v1.0 + - riscv: errata: Fix bitwise vs logical AND in MIPS errata patching + - riscv: Fix register corruption from uninitialized cregs on error + - riscv: mm: Fixup no5lvl failure when vaddr is invalid + - kunit: config: Enable KUNIT_DEBUGFS by default + - kunit: config: KUNIT_DEBUGFS should depend on DEBUG_FS + - pinctrl: qcom: Fix wakeirq map by removing disconnected irqs for sm8150 + - firmware: arm_ffa: Bound PARTITION_INFO_GET_REGS copies + - firmware: arm_ffa: Keep framework RX release under lock + - firmware: arm_ffa: Validate framework notification message layout + - firmware: arm_ffa: Align RxTx buffer size before mapping + - firmware: arm_ffa: Snapshot notifier callbacks under lock + - firmware: arm_ffa: Fix sched-recv callback partition lookup + - ARM: integrator: Fix early initialization + - ALSA: hda: cs35l56: Put ACPI device after setting companion + - ALSA: hda: cs35l41: Put ACPI device on missing physical node + - btrfs: tracepoints: fix sleep while in atomic context in + btrfs_sync_file() + - netfilter: x_tables: allow initial table replace without emitting audit + log message + - netfilter: x_tables: allocate hook ops while under mutex + - netfilter: x_tables: unregister the templates first + - netfilter: x_tables: add and use xt_unregister_table_pre_exit + - netfilter: x_tables: add and use xtables_unregister_table_exit + - netfilter: ebtables: move to two-stage removal scheme + - netfilter: ebtables: close dangling table module init race + - netfilter: x_tables: close dangling table module init race + - netfilter: bridge: eb_tables: close module init race + - netfilter: nf_conntrack_expect: restore helper propagation via + expectation + - kprobes: skip non-symbol addresses in kprobe_add_ksym_blacklist() + - test_kprobes: clear kprobes between test runs + - tcp: Fix imbalanced icsk_accept_queue count. + - net: napi: Avoid gro timer misfiring at end of busypoll + - net: shaper: Reject reparenting of existing nodes + - idpf: fix read_dev_clk_lock spinlock init in idpf_ptp_init() + - ice: fix setting RSS VSI hash for E830 + - ice: fix locking in ice_dcb_rebuild() + - ice: dpll: fix rclk pin state get for E810 + - ice: dpll: fix misplaced header macros + - net: lan966x: avoid unregistering netdev on register failure + - net: ti: icssm-prueth: fix eth_ports_node leak in probe + - phy: marvell: mvebu-a3700-utmi: fix incorrect USB2_PHY_CTRL register + access + - phy: spacemit: Remove incorrect clk_disable() in spacemit_usb2phy_init() + - NFSD: Fix infinite loop in layout state revocation + - ASoC: sdw_utils: Add quirk to ignore RT712 CODEC_MIC + - ASoC: sdw_utils: Add quirk to ignore RT721 CODEC_MIC + - fprobe: Fix unregister_fprobe() to wait for RCU grace period + - fs/statmount: fix slab out-of-bounds write in statmount_mnt_idmap + - fs: Fix return in jfs_mkdir and orangefs_mkdir + - irqchip/ath79-cpu: Remove unused function + - fs: fix forced iversion increment on lazytime timestamp updates + - ublk: reject max_sectors smaller than PAGE_SECTORS in parameter + validation + - nsfs: fix wrong error code returned for pidns ioctls + - irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT + - nvme: fix bio leak on mapping failure + - nvme-pci: fix use-after-free in nvme_free_host_mem() + - zonefs: handle integer overflow in zonefs_fname_to_fno + - tcp: Fix out-of-bounds access for twsk in tcp_ao_established_key(). + - ASoC: SOF: amd: Fix error code handling in psp_send_cmd() + - powerpc: 82xx: fix uninitialized pointers with free attribute + - powerpc: fix dead default for GUEST_STATE_BUFFER_TEST + - powerpc/hv-gpci: fix preempt count leak in sysfs show paths + - netfs: Fix cancellation of a DIO and single read subrequests + - netfs: Fix missing locking around retry adding new subreqs + - netfs: Fix missing barriers when accessing stream->subrequests + locklessly + - netfs: Fix netfs_read_to_pagecache() to pause on subreq failure + - netfs: Fix potential for tearing in ->remote_i_size and ->zero_point + - netfs: Fix zeropoint update where i_size > remote_i_size + - netfs: fix VM_BUG_ON_FOLIO() issue in netfs_write_begin() call + - netfs: Fix overrun check in netfs_extract_user_iter() + - netfs: Fix netfs_invalidate_folio() to clear dirty bit if all changes + gone + - netfs: Defer the emission of trace_netfs_folio() + - netfs: Fix streaming write being overwritten + - netfs: Fix potential deadlock in write-through mode + - netfs: Fix read-gaps to remove netfs_folio from filled folio + - netfs: Fix write streaming disablement if fd open O_RDWR + - netfs: Fix early put of sink folio in netfs_read_gaps() + - netfs: Fix leak of request in netfs_write_begin() error handling + - netfs: Fix potential UAF in netfs_unlock_abandoned_read_pages() + - netfs: Fix partial invalidation of streaming-write folio + - netfs: Fix folio->private handling in netfs_perform_write() + - netfs: Fix netfs_read_folio() to wait on writeback + - netfs, afs: Fix write skipping in dir/link writepages + - afs: Fix the locking used by afs_get_link() + - net: ethernet: cortina: Make RX SKB per-port + - net: ethernet: cortina: Drop half-assembled SKB + - net: ethernet: cortina: Carry over frag counter + - net: ethernet: cs89x0: remove stale CONFIG_MACH_MX31ADS reference + - wifi: ath11k: fix error path leaks in some WMI WOW calls + - wifi: ath11k: fix error path leak in ath11k_tm_cmd_wmi_ftm() + - wifi: ath10k: skip WMI and beacon transmission when device is wedged + - net: shaper: flip the polarity of the valid flag + - net: shaper: fix trivial ordering issue in net_shaper_commit() + - net: shaper: reject duplicate leaves in GROUP request + - net: shaper: set ret to -ENOMEM when genlmsg_new() fails in group_doit + - net: shaper: fix undersized reply skb allocation in GROUP command + - net: shaper: enforce singleton NETDEV scope with id 0 + - net: shaper: reject QUEUE scope handle with missing id + - block: don't overwrite bip_vcnt in bio_integrity_copy_user() + - block: recompute nr_integrity_segments in blk_insert_cloned_request + - HID: quirks: really enable the intended work around for appledisplay + - block: bio-integrity: Fix null-ptr-deref in bio_integrity_map_user() + - accel/qaic: Add overflow check to remap_pfn_range during mmap + - net/smc: avoid NULL deref of conn->lnk in smc_msg_event tracepoint + - ethtool: fix ethnl_bitmap32_not_zero() bit interval semantics + - drm/msm/dpu: fix UV scanlines calculation for YUV UBWC formats + - drm/msm/dpu: Fix Kaanapali CWB register configuration + - drm/msm/dsi: don't dump registers past the mapped region + - drm/msm/dpu: don't mix devm and drmm functions + - block: rename struct gendisk zone_wplugs_lock field + - block: allow submitting all zone writes from a single context + - block: fix handling of dead zone write plugs + - selftests: ublk: cap nthreads to kernel's actual nr_hw_queues + - x86/mce: Restore MCA polling interval halving + - Documentation: intel_pstate: Fix description of asymmetric packing with + SMT + - drm/msm: Fix GMEM_BASE for A650 + - drm/msm/a6xx: Add soft fuse detection support + - drm/msm/adreno: Fix a reference leak in a6xx_gpu_init() + - drm/msm/adreno: fix userspace-triggered crash on a2xx-a4xx + - drm/msm/a6xx: Restore sysprof_active + - drm/msm: Fix iommu_map_sgtable() return value check and avoid WARN + - drm/msm/a6xx: Check kzalloc return in a8xx_hfi_send_perf_table + - ASoC: intel: sof_sdw: Prepare for configuration without a jack + - ASoC: sdw_utils: cs42l43: allow spk component names to be combined + - ASoC: sdw_utils: Check speaker component string allocation + - riscv: Docs: fix unmatched quote warning + - powerpc/time: Remove redundant preempt_disable|enable() calls from + arch_irq_work_raise() + - net/smc: reject CHID-0 ACCEPT that matches an empty ism_dev slot + - net: tls: fix off-by-one in sg_chain entry count for wrapped sk_msg ring + - net: tls: prevent chain-after-chain in plain text SG + - net: phy: DP83TC811: add reading of abilities + - ovpn: tcp - use cached peer pointer in ovpn_tcp_close() + - ovpn: respect peer refcount in CMD_NEW_PEER error path + - ovpn: fix race between deleting interface and adding new peer + - cifs: client: stage smb3_reconfigure() updates and restore ctx on + failure + - phy: apple: atc: Fix typec switch/mux leak on unbind + - gcc-plugins: Always define CONST_CAST_GIMPLE and CONST_CAST_TREE + - x86/xen: Fix xen_e820_swap_entry_with_ram() + - vfio/pci: Check BAR resources before exporting a DMABUF + - ovpn: disable BHs when updating device stats + - tls: Preserve sk_err across recvmsg() when data has been copied + - net/mlx5: Do not restore destination-less TC rules + - net/mlx5: Skip disabled vports when setting max TX speed + - scsi: sd: Fix return code handling in sd_spinup_disk() + - ASoC: codecs: fs210x: fix possible buffer overflow + - iommupt: Directly call iommupt's unmap_range() + - iommupt: Avoid rewalking during map + - iommu: Fix loss of errno on map failure for classic ops + - iommu: Fix up map/unmap debugging for iommupt domains + - iommu: Handle unmap error when iommu_debug is enabled + - iommupt: Check for missing PAGE_SIZE in the pgsize_bitmap + - iommupt: Fix the end_index calculation in __map_range_leaf() + - ALSA: scarlett2: Add missing error check when initialise Autogain Status + - ALSA: hda/ca0132: Disable auto-detect on manual output select + - cachefiles: Fix error return when vfs_mkdir() fails + - io_uring/net: punt IORING_OP_BIND async if it needs file create + - vsock/virtio: fix zerocopy completion for multi-skb sends + - btrfs: check for subvolume before deleting squota qgroup + - btrfs: fix squota accounting during enable generation + - ASoC: amd: acp-sdw-legacy: check CPU DAI name before logging + - spi: mtk-snfi: Fix resource leak in mtk_snand_read_page_cache() + - netfilter: nft_inner: release local_lock before re-enabling softirqs + - ALSA: hda/realtek: Use ALC287_FIXUP_TXNW2781_I2C for ASUS Strix Gxx5 + - drm/msm/snapshot: fix dumping of the unaligned regions + - hwmon: (lm90) Stop work before releasing hwmon device + - hwmon: (lm90) Add lock protection to lm90_alert + - wifi: iwlwifi: mld: fix TSO segmentation explosion when AMSDU is + disabled + - wifi: iwlwifi: mld: don't dereference a pointer before NULL checking it + - dma-mapping: move dma_map_resource() sanity check into debug code + - drm/gem: Make the GEM LRU lock part of drm_device + - drm/xe/gsc: Fix double-free of managed BO in error path + - drm/xe/vf: Fix signature of print functions + - drm/xe/pf: Fix CFI failure in debugfs access + - drm/xe: Consolidate workaround entries for Wa_14019988906 + - drm/xe: Consolidate workaround entries for Wa_18033852989 + - drm/xe: Define and use MCR version of COMMON_SLICE_CHICKEN1 + - drm/xe/tuning: Apply windower hardware filtering setting on Xe3 and Xe3p + - drm/xe: Define and use MCR version of COMMON_SLICE_CHICKEN4 + - wifi: ath11k: fix peer resolution on rx path when peer_id=0 + - wifi: ath12k: fix EHT TX MCS limitation due to wrong 20 MHz-only parsing + - drm/mediatek: mtk_hdmi_ddc_v2: Fix non-static global variable + - drm/mediatek: mtk_hdmi_v2: Fix non-static global variable + - drm/mediatek: mtk_cec: Fix non-static global variable + - drm/mediatek: mtk_hdmi_ddc: Fix non-static global variable + - io_uring: propagate array_index_nospec opcode into req->opcode + - srcu: Don't queue workqueue handlers to never-online CPUs + - cgroup/rstat: validate cpu before css_rstat_cpu() access + - net/mlx5e: xsk: Fix unlocked writing to ICOSQ + - cifs: Fix undefined variables + - ice: ptp: serialize E825 PHY timer start with PTP lock + - ice: ptp: use primary NAC semaphore on E825 + - igc: set tx buffer type for SMD frames + - drm/i915/dp: Fix readback for target_rr in Adaptive Sync SDP + - phy: qcom: qmp-usbc: Fix out-of-bounds array access in dp swing config + - kbuild: pacman-pkg: make "rc" releases adhere to pacman versioning + scheme + - net: dsa: mt7530: fix FDB entries not aging out with short timeout + - net: dsa: mt7530: preserve VLAN tags on trapped link-local frames + - net: mana: Fix TOCTOU double-fetch of hwc_msg_id from DMA buffer + - platform/surface: aggregator_registry: omit battery & AC nodes on + Surface Laptop 7 + - platform/x86: adv_swbutton: Check ACPI_HANDLE() against NULL + - platform/x86: hp_accel: Check ACPI_COMPANION() against NULL + - platform/x86: intel-hid: Check ACPI_HANDLE() against NULL + - platform/x86: intel_sar: Check ACPI_HANDLE() against NULL + - platform/x86: intel-vbtn: Check ACPI_HANDLE() against NULL + - platform/x86: uniwill-laptop: Properly initialize charging threshold + - platform/x86: uniwill-laptop: Accept charging threshold of 0 + - platform/x86: uniwill-laptop: Fix behavior of "force" module param + - platform/x86: asus-armoury: fix mini-LED mode get/set on MODE2 devices + - ASoC: soc-utils: Add missing va_end in snd_soc_ret() + - drm/amdgpu: Align amdgpu_gtt_mgr entries to TLB size on Tahiti (v2) + - drm/amdgpu/vce1: Check that the GPU address is < 128 MiB + - drm/amdgpu/vce1: Fix VCE 1 firmware size and offsets + - RDMA/mana_ib: Report max_msg_sz in mana_ib_query_port + - RDMA/rtrs: Fix use-after-free in path file creation cleanup + - bridge: mcast: Fix a possible use-after-free when removing a bridge port + - net: phy: honor eee_disabled_modes in phy_support_eee() + - net: phy: honor eee_disabled_modes in phy_advertise_eee_all() + - net: airoha: Fix NPU RX DMA descriptor bits + - pds_core: fix error handling in pdsc_devcmd_wait + - pds_core: fix debugfs_lookup dentry leak and error handling + - erofs: fix managed cache race for unaligned extents + - erofs: harden h_shared_count in erofs_init_inode_xattrs() + - erofs: fix metabuf leak in inode xattr initialization + - wifi: mac80211: bounds-check link_id in ieee80211_ml_epcs + - wifi: mac80211: fix MLE defragmentation + - wifi: mac80211: fix multi-link element inheritance + - wifi: wilc1000: fix dma_buffer leak on bus acquire failure + - ALSA: seq: Serialize UMP output teardown with event_input + - cgroup: rstat: relax NMI guard after switch to try_cmpxchg + - tracing: Avoid NULL return from hist_field_name() on truncation + - Bluetooth: hci_sync: Fix not setting mask for + HCI_EVT_LE_ALL_REMOTE_FEATURES_COMPLETE + - Bluetooth: btintel_pcie: Fix incorrect MAC access programming + - Bluetooth: btmtk: fix urb->setup_packet leak in error paths + - udp: gso: Fix handling checksum in __udp_gso_segment + - udp: Fix UDP length on last GSO_PARTIAL segment + - net/mlx5e: Fix eswitch mode block underflow on IPsec acquire SA + - net: shaper: annotate the data races + - net: shaper: rework the VALID marking (again) + - crypto/krb5, rxrpc: Fix lack of pre-decrypt/pre-verify length checks + - rxrpc: Fix DATA decrypt vs splice() by copying data to buffer in recvmsg + - net: ag71xx: check error for platform_get_irq + - bpf, skmsg: fix verdict sk_data_ready racing with ktls rx + - tcp: fix stale per-CPU tcp_tw_isn leak enabling ISN prediction + - net: stmmac: eswin: fix HSP CSR init ordering after clock enable + - net: stmmac: eswin: clear TXD and RXD delay registers during + initialization + - net: stmmac: eswin: correct RGMII delay granularity to 20 ps + - net: stmmac: eswin: validate RGMII delay values + - gpio: cdev: check if uAPI v2 config attributes are correctly zeroed + - gpio: aggregator: fix a potential use-after-free + - gpio: aggregator: stop using dev-sync-probe + - gpio: aggregator: remove the software node when deactivating the + aggregator + - gpio: aggregator: lock device when calling device_is_bound() + - ASoC: cs35l56: Fix flushing of IRQ work in cs35l56_sdw_remove() + - drm/xe/oa: Fix exec_queue leak on width check in stream open + - ASoC: cs-amp-lib: Fix wrong sizeof() in + _cs_amp_set_efi_calibration_data() + - ASoC: cs-amp-lib: Fix missing dput() after debugfs_lookup() + - selftests: net: Fix checksums in xdp_native + - nvme-pci: fix dma_vecs leak on p2p memory + - nvme-pci: fix dma mapping leak on data setup error + - octeontx2-af: npc: Fix allmulticast skip logic for LBK and SDP VFs + - net: mana: validate rx_req_idx to prevent out-of-bounds array access + - tap: fix stack info leak in tap_ioctl() SIOCGIFHWADDR + - net: airoha: Disable GDM2 forwarding before configuring GDM2 loopback + - pds_core: ensure null-termination for firmware version strings + - net: enetc: fix missing error code when pf->vf_state allocation fails + - io_uring/nop: pass all errors to userspace + - blk-mq: pop cached request if it is usable + - ksmbd: fix durable reconnect error path file lifetime + - LoongArch: kprobes: Fix handling of fatal unrecoverable recursions + - block: avoid use-after-free in disk_free_zone_resources() + - Documentation: laptops: Update documentation for uniwill laptops + - platform/x86: uniwill-laptop: Do not enable the charging limit even when + forced + - drm/msm: Restore second parameter name in purge() and evict() + - security/keys: fix missed RCU read section on lookup + - Linux 7.0.11 + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) + - blk-cgroup: wait for blkcg cleanup before initializing new disk + - md: suppress spurious superblock update error message for dm-raid + - fs/omfs: reject s_sys_blocksize smaller than OMFS_DIR_START + - fs/mbcache: cancel shrink work before destroying the cache + - md/raid1: fix the comparing region of interval tree + - fs: fix archiecture-specific compat_ftruncate64 + - drbd: Balance RCU calls in drbd_adm_dump_devices() + - loop: fix partition scan race between udev and loop_reread_partitions() + - block: fix zones_cond memory leak on zone revalidation error paths + - nilfs2: reject zero bd_oblocknr in nilfs_ioctl_mark_blocks_dirty() + - blk-cgroup: fix disk reference leak in blkcg_maybe_throttle_current() + - pstore/ram: fix resource leak when ioremap() fails + - erofs: include the trailing NUL in FS_IOC_GETFSLABEL + - md: fix array_state=clear sysfs deadlock + - ublk: reset per-IO canceled flag on each fetch + - blk-wbt: remove WARN_ON_ONCE from wbt_init_enable_default() + - erofs: handle 48-bit blocks/uniaddr for extra devices + - md: remove unused static md_wq workqueue + - md: wake raid456 reshape waiters before suspend + - dcache: permit dynamic_dname()s up to NAME_MAX + - btrfs: fix the inline compressed extent check in inode_need_compress() + - btrfs: fix deadlock between reflink and transaction commit when using + flushoncommit + - btrfs: do not reject a valid running dev-replace + - OPP: debugfs: Use performance level if available to distinguish between + rates + - OPP: Move break out of scoped_guard in dev_pm_opp_xlate_required_opp() + - ACPI: x86: cmos_rtc: Clean up address space handler driver + - ACPI: x86: cmos_rtc: Improve coordination with ACPI TAD driver + - devres: fix missing node debug info in devm_krealloc() + - thermal/drivers/spear: Fix error condition for reading st,thermal-flags + - debugfs: check for NULL pointer in debugfs_create_str() + - debugfs: fix placement of EXPORT_SYMBOL_GPL for debugfs_create_str() + - soundwire: debugfs: initialize firmware_file to empty string + - amd-pstate: Fix memory leak in amd_pstate_epp_cpu_init() + - amd-pstate: Update cppc_req_cached in fast_switch case + - cpufreq: Pass the policy to cpufreq_driver->adjust_perf() + - PCI: use generic driver_override infrastructure + - platform/wmi: use generic driver_override infrastructure + - vdpa: use generic driver_override infrastructure + - s390/cio: use generic driver_override infrastructure + - s390/ap: use generic driver_override infrastructure + - bus: fsl-mc: use generic driver_override infrastructure + - locking/mutex: Rename mutex_init_lockep() + - locking/mutex: Fix wrong comment for CONFIG_DEBUG_LOCK_ALLOC + - irqchip/irq-pic32-evic: Address warning related to wrong printf() + formatter + - hrtimer: Avoid pointless reprogramming in __hrtimer_start_range_ns() + - hrtimer: Reduce trace noise in hrtimer_start() + - locking: Fix rwlock and spinlock lock context annotations + - signal: Fix the lock_task_sighand() annotation + - ww-mutex: Fix the ww_acquire_ctx function annotations + - perf/amd/ibs: Account interrupt for discarded samples + - perf/amd/ibs: Preserve PhyAddrVal bit when clearing PhyAddr MSR + - perf/amd/ibs: Avoid calling perf_allow_kernel() from the IBS NMI handler + - x86/tdx: Fix the typo in TDX_ATTR_MIGRTABLE + - rust: sync: atomic: Remove bound `T: Sync` for `Atomic::from_ptr()` + - sparc64: vdso: Link with -z noexecstack + - scripts/gdb: timerlist: Adapt to move of tk_core + - locking: Fix rwlock support in + - sched/topology: Compute sd_weight considering cpuset partitions + - x86/irqflags: Preemptively move include paravirt.h directive where it + belongs + - sched/topology: Fix sched_domain_span() + - irqchip/renesas-rzg2l: Fix error path in rzg2l_irqc_common_probe() + - ASoC: Intel: avs: Check maximum valid CPUID leaf + - ASoC: Intel: avs: Include CPUID header at file scope + - x86/vdso: Clean up remnants of VDSO32_NOTE_MASK + - firmware: dmi: Correct an indexing error in dmi.h + - fs/resctrl: Report invalid domain ID when parsing io_alloc_cbm + - sched: Make class_schedulers avoid pushing current, and get rid of + proxy_tag_curr() + - sched/rt: Skip group schedulable check with rt_group_sched=0 + - wifi: ath11k: fix memory leaks in beacon template setup + - wifi: mwifiex: Fix memory leak in mwifiex_11n_aggregate_pkt() + - wifi: rtlwifi: pci: fix possible use-after-free caused by unfinished + irq_prepare_bcn_tasklet + - bpf: test_run: Fix the null pointer dereference issue in + bpf_lwt_xmit_push_encap + - wifi: ath12k: account TX stats only when ACK/BA status is present + - wifi: ath12k: Fix legacy rate mapping for monitor mode capture + - selftests/bpf: Handle !CONFIG_SMC in bpf_smc.c + - wifi: ieee80211: fix definition of EHT-MCS 15 in MRU + - dpaa2: add independent dependencies for FSL_DPAA2_SWITCH + - [Config] Disable CONFIG_FSL_DPAA2_SWITCH on armhf and ppc64el + - dpaa2: compile dpaa2 even CONFIG_FSL_DPAA2_ETH=n + - s390/bpf: Zero-extend bpf prog return values and kfunc arguments + - powerpc/pgtable-frag: Fix bad page state in pte_frag_destroy + - powerpc/64s: Fix unmap race with PMD migration entries + - module: Fix freeing of charp module parameters when CONFIG_SYSFS=n + - wifi: libertas: use USB anchors for tracking in-flight URBs + - wifi: libertas: don't kill URBs in interrupt context + - bpf: Do not allow deleting local storage in NMI + - selftests/nolibc: fix test_file_stream() on musl libc + - selftests/nolibc: Fix build with host headers and libc + - tools/nolibc/printf: Change variables 'c' to 'ch' and 'tmpbuf[]' to + 'outbuf[]' + - tools/nolibc/printf: Move snprintf length check to callback + - tools/nolibc: MIPS: fix clobbers of 'lo' and 'hi' registers on different + ISAs + - tools/nolibc: avoid -Wundef warning for __STDC_VERSION__ + - wifi: mt76: mt7996: fix the behavior of radar detection + - wifi: mt76: mt7996: fix iface combination for different chipsets + - wifi: mt76: mt7996: Set mtxq->wcid just for primary link + - wifi: mt76: mt7996: Reset mtxq->idx if primary link is removed in + mt7996_vif_link_remove() + - wifi: mt76: mt7996: Switch to the secondary link if the default one is + removed + - wifi: mt76: mt7996: Clear wcid pointer in mt7996_mac_sta_deinit_link() + - wifi: mt76: mt7996: Reset ampdu_state state in case of failure in + mt7996_tx_check_aggr() + - wifi: mt76: mt7921: Reset ampdu_state state in case of failure in + mt76_connac2_tx_check_aggr() + - wifi: mt76: mt7925: Fix incorrect MLO mode in firmware control + - wifi: mt76: mt7615: fix use_cts_prot support + - wifi: mt76: mt7915: fix use_cts_prot support + - wifi: mt76: mt7925: prevent NULL pointer dereference in + mt7925_tx_check_aggr() + - wifi: mt76: mt7925: prevent NULL vif dereference in + mt7925_mac_write_txwi + - wifi: mt76: mt7996: fix FCS error flag check in RX descriptor + - wifi: mt76: mt7921: Place upper limit on station AID + - wifi: mt76: Fix memory leak destroying device + - wifi: mt76: mt7996: Fix NPU stop procedure + - wifi: mt76: npu: Add missing rx_token_size initialization + - wifi: mt76: mt7925: drop puncturing handling from BSS change path + - wifi: mt76: mt7925: fix potential deadlock in mt7925_roc_abort_sync + - wifi: mt76: Fix memory leak after mt76_connac_mcu_alloc_sta_req() + - wifi: mt76: mt7925: fix tx power setting failure after chip reset + - wifi: mt76: mt7921: fix potential deadlock in mt7921_roc_abort_sync + - wifi: mt76: fix deadlock in remain-on-channel + - wifi: mt76: fix backoff fields and max_power calculation + - arm64: cpufeature: Make PMUVer and PerfMon unsigned + - bpf: Switch CONFIG_CFI_CLANG to CONFIG_CFI + - wifi: mt76: mt7996: fix wrong DMAD length when using MAC TXP + - wifi: mt76: mt7996: fix struct mt7996_mcu_uni_event + - wifi: mt76: mt7915: fix use-after-free bugs in mt7915_mac_dump_work() + - wifi: mt76: mt7996: fix use-after-free bugs in mt7996_mac_dump_work() + - wifi: mt76: mt7921: fix 6GHz regulatory update on connection + - wifi: mt76: mt7996: Add missing CHANCTX_STA_CSA property + - wifi: mt76: mt7996: Remove link pointer dependency in + mt7996_mac_sta_remove_links() + - wifi: mt76: mt7996: Decrement sta counter removing the link in + mt7996_mac_reset_sta_iter() + - wifi: mt76: fix multi-radio on-channel scanning + - wifi: mt76: support upgrading passive scans to active + - wifi: mt76: mt7996: fix RRO EMU configuration + - bpf: Fix refcount check in check_struct_ops_btf_id() + - selftests/bpf: Fix sockmap_multi_channels reliability + - bpf: Use RCU-safe iteration in dev_map_redirect_multi() SKB path + - bpf: Fix variable length stack write over spilled pointers + - arm_mpam: Ensure in_reset_state is false after applying configuration + - arm_mpam: Reset when feature configuration bit unset + - bpf,arc_jit: Fix missing newline in pr_err messages + - wifi: rtw89: phy: fix uninitialized variable access in + rtw89_phy_cfo_set_crystal_cap() + - drivers/vfio_pci_core: Change PXD_ORDER check from switch case to + if/else block + - r8152: fix incorrect register write to USB_UPHY_XTAL + - selftests/tracing: Fix to make --logdir option work again + - selftests/tracing: Fix to check awk supports non POSIX strtonum() + - powerpc/crash: fix backup region offset update to elfcorehdr + - powerpc/crash: Update backup region offset in elfcorehdr on memory + hotplug + - bpf: Fix abuse of kprobe_write_ctx via freplace + - macvlan: annotate data-races around port->bc_queue_len_used + - bpf: Use copy_map_value_locked() in alloc_htab_elem() for BPF_F_LOCK + - bpf: Fix stale offload->prog pointer after constant blinding + - net: ethernet: ti-cpsw:: rename soft_reset() function + - net: ethernet: ti-cpsw: fix linking built-in code to modules + - wifi: brcmfmac: Fix error pointer dereference + - wifi: mac80211: handle VHT EXT NSS in ieee80211_determine_our_sta_mode() + - bpf: Drop task_to_inode and inet_conn_established from lsm sleepable + hooks + - bpf: Prefer vmlinux symbols over module symbols for unqualified kprobes + - wifi: ath10k: fix station lookup failure during disconnect + - bpf: Fix linked reg delta tracking when src_reg == dst_reg + - net: dropreason: add SKB_DROP_REASON_RECURSION_LIMIT + - net: plumb drop reasons to __dev_queue_xmit() + - net: qdisc_pkt_len_segs_init() cleanup + - net: pull headers in qdisc_pkt_len_segs_init() + - arm64: entry: Don't preempt with SError or Debug masked + - ACPI: AGDI: fix missing newline in error message + - arm64: kexec: Remove duplicate allocation for trans_pgd + - bpf: Propagate error from visit_tailcall_insn + - bpf: Fix ld_{abs,ind} failure path analysis in subprogs + - bpf: Remove static qualifier from local subprog pointer + - mptcp: better mptcp-level RTT estimator + - bpf: Fix use-after-free in offloaded map/prog info fill + - macsec: Support VLAN-filtering lower devices + - net: bcmgenet: fix off-by-one in bcmgenet_put_txcb + - net: bcmgenet: fix leaking free_bds + - net: bcmgenet: fix racing timeout handler + - net: airoha: Add dma_rmb() and READ_ONCE() in airoha_qdma_rx_process() + - eth: fbnic: Use wake instead of start + - netfilter: xt_socket: enable defrag after all other checks + - netfilter: nft_fwd_netdev: check ttl/hl before forwarding + - bpf: fix mm lifecycle in open-coded task_vma iterator + - bpf: switch task_vma iterator from mmap_lock to per-VMA locks + - bpf: return VMA snapshot from task_vma iterator + - bpf: Fix RCU stall in bpf_fd_array_map_clear() + - net: hamradio: 6pack: fix uninit-value in sixpack_receive_buf + - net: airoha: Fix FE_PSE_BUF_SET configuration if PPE2 is available + - bpf: Enforce regsafe base id consistency for BPF_ADD_CONST scalars + - selftests/bpf: fix __jited_unpriv tag name + - net/sched: cls_fw: fix NULL dereference of "old" filters before change() + - net/sched: act_ct: Only release RCU read lock after ct_ft + - selftests: netfilter: nft_tproxy.sh: adjust to socat changes + - net: mana: Use pci_name() for debugfs directory naming + - net: mana: Move current_speed debugfs file to mana_init_port() + - net: airoha: Add missing RX_CPU_IDX() configuration in + airoha_qdma_cleanup_rx_queue() + - net_sched: fix skb memory leak in deferred qdisc drops + - bpf: Fix same-register dst/src OOB read and pointer leak in sock_ops + - bpf: Allow instructions with arena source and non-arena dest registers + - selftests/bpf: Fix reg_bounds to match new tnum-based refinement + - net/rds: Optimize rds_ib_laddr_check + - net/rds: Restrict use of RDS/IB to the initial network namespace + - bpf: Fix OOB in pcpu_init_value + - ppp: require CAP_NET_ADMIN in target netns for unattached ioctls + - net: ipa: Fix programming of QTIME_TIMESTAMP_CFG + - net: ipa: Fix decoding EV_PER_EE for IPA v5.0+ + - dt-bindings: net: dsa: nxp,sja1105: make spi-cpol optional for sja1110 + - net: phy: fix a return path in get_phy_c45_ids() + - net/mlx5e: Fix features not applied during netdev registration + - net/mlx5e: IPsec, fix ASO poll timeout with read_poll_timeout_atomic() + - net: ethernet: mtk_eth_soc: initialize PPE per-tag-layer MTU registers + - net: fix skb_ext_total_length() BUILD_BUG_ON with + CONFIG_GCOV_PROFILE_ALL + - bpf: reject short IPv4/IPv6 inputs in bpf_prog_test_run_skb + - Bluetooth: L2CAP: Fix printing wrong information if SDU length exceeds + MTU + - Bluetooth: hci_ldisc: Clear HCI_UART_PROTO_INIT on error + - Bluetooth: fix locking in hci_conn_request_evt() with HCI_PROTO_DEFER + - Bluetooth: l2cap: Add missing chan lock in l2cap_ecred_reconf_rsp + - Bluetooth: SCO: check for codecs->num_codecs == 1 before assigning to + sco_pi(sk)->codec + - net: phy: qcom: at803x: Use the correct bit to disable extended next + page + - udp: Force compute_score to always inline + - tcp: Don't set treq->req_usec_ts in cookie_tcp_reqsk_init(). + - sctp: fix missing encap_port propagation for GSO fragments + - sctp: disable BH before calling udp_tunnel_xmit_skb() + - selftests/namespaces: remove unused utils.h include from + listns_efault_test + - net, bpf: fix null-ptr-deref in xdp_master_redirect() for down master + - net: airoha: Fix VIP configuration for AN7583 SoC + - net: airoha: Add missing PPE configurations in airoha_ppe_hw_init() + - drm/panel: ilitek-ili9882t: Select DRM_DISPLAY_DSC_HELPER + - selftests/futex: Fix incorrect result reporting of futex_requeue test + item + - drm/komeda: fix integer overflow in AFBC framebuffer size check + - dma-fence: Fix sparse warnings due __rcu annotations + - drm/gpusvm: Fix unbalanced unlock in drm_gpusvm_scan_mm() + - drm/virtio: Allow importing prime buffers when 3D is enabled + - ASoC: soc-compress: use function to clear symmetric params + - PCI/TPH: Allow TPH enable for RCiEPs + - PCI: endpoint: pci-epf-vntb: Fix MSI doorbell IRQ unwind + - PCI: endpoint: pci-epf-test: Don't free doorbell IRQ unless requested + - PCI: endpoint: pci-ep-msi: Fix error unwind and prevent double alloc + - drm/sun4i: mixer: Fix layer init code + - drm/sun4i: backend: fix error pointer dereference + - drm/xe: Consolidate workaround entries for Wa_14019877138 + - drm/xe: Consolidate workaround entries for Wa_14019386621 + - drm/xe/xe2_hpg: Drop invalid workaround Wa_15010599737 + - gpu: nova-core: gsp: use empty slices instead of [0..0] ranges + - gpu: nova-core: gsp: fix improper handling of empty slot in cmdq + - drm/amdkfd: Removed commented line for MQD queue priority + - PCI: imx6: Fix device node reference leak in imx_pcie_probe() + - crypto: inside-secure/eip93 - fix register definition + - ASoC: sti: Return errors from regmap_field_alloc() + - ASoC: sti: use managed regmap_field allocations + - dm cache: fix null-deref with concurrent writes in passthrough mode + - dm cache: fix write path cache coherency in passthrough mode + - dm cache: fix write hang in passthrough mode + - dm cache policy smq: fix missing locks in invalidating cache blocks + - dm cache: fix concurrent write failure in passthrough mode + - dm cache: fix dirty mapping checking in passthrough mode switching + - dm-mpath: don't stop probing paths at presuspend + - drm/amd/ras: Fix type size of remainder argument + - dt-bindings: mmc: dwcmshc-sdhci: Fix resets array validation + - drm/amdgpu: GFX12.1 scratch memory limit up to 57-bit + - platform/chrome: chromeos_tbmc: Drop wakeup source on remove + - gpu: nova-core: use checked arithmetic in FWSEC firmware parsing + - gpu: nova-core: create falcon firmware DMA objects lazily + - gpu: nova-core: falcon: rename load parameters to reflect DMA dependency + - gpu: nova-core: firmware: fix and explain v2 header offsets computations + - PCI: dwc: ep: Fix MSI-X Table Size configuration in + dw_pcie_ep_set_msix() + - PCI: dwc: ep: Mirror the max link width and speed fields to all + functions + - PCI: dwc: Perform cleanup in the error path of dw_pcie_resume_noirq() + - dm cache metadata: fix memory leak on metadata abort retry + - dm log: fix out-of-bounds write due to region_count overflow + - iopoll: fix function parameter names in read_poll_timeout_atomic() + - drm/bridge: cadence: cdns-mhdp8546-core: Set the mhdp connector earlier + in atomic_enable() + - drm/bridge: cadence: cdns-mhdp8546-core: Add mode_valid hook to + drm_bridge_funcs + - drm/bridge: cadence: cdns-mhdp8546-core: Handle HDCP state in bridge + atomic check + - spi: nxp-xspi: Use reinit_completion() for repeated operations + - spi: nxp-fspi: Use reinit_completion() for repeated operations + - spi: fsl-qspi: Use reinit_completion() for repeated operations + - spi: axiado: Remove redundant pm_runtime_mark_last_busy() call + - media: i2c: og01a1b: Fix V4L2 subdevice data initialization on probe + - media: synopsys: VIDEO_DW_MIPI_CSI2RX should depend on ARCH_ROCKCHIP + - [Config] VIDEO_DW_MIPI_CSI2RX depends on ARCH_ROCKCHIP + - drm/amd/pm: Fix xgmi max speed reporting + - spi: atcspi200: fix mutex initialization order + - selftests/sched_ext: Add missing error check for exit__load() + - drm/v3d: Handle error from drm_sched_entity_init() + - drm/sun4i: Fix resource leaks + - crypto: inside-secure/eip93 - register hash before authenc algorithms + - PCI: rzg3s-host: Fix reset handling in probe error path + - PCI: rzg3s-host: Reorder reset assertion during suspend + - dt-bindings: PCI: renesas,r9a08g045s33-pcie: Fix naming properties + - iommu/riscv: Add IOTINVAL after updating DDT/PDT entries + - iommu/riscv: Skip IRQ count check when using MSI interrupts + - iommu/riscv: Add missing GENERIC_MSI_IRQ + - iommu/riscv: Stop polling when CQCSR reports an error + - drm/amdkfd: Update queue properties for metadata ring + - drm/amd/ras: Fix NULL deref in ras_core_ras_interrupt_detected() + - drm/amdgpu: Add default case in DVI mode validation + - regulator: dt-bindings: fp9931: Make vin-supply property as required + - regulator: fp9931: Fix handling of mandatory "vin" supply + - drm/amd/ras: Fix NULL deref in ras_core_get_utc_second_timestamp() + - drm/amdgpu: Drop redundant queue NULL check in hang detect worker + - drm/amdgpu: Remove dead negative offset check in + amdgpu_virt_init_critical_region() + - dm init: ensure device probing has finished in dm-mod.waitfor= + - fbdev: matroxfb: Mark variable with __maybe_unused to avoid W=1 build + break + - crypto: simd - reject compat registrations without __ prefixes + - crypto: tegra - Disable softirqs before finalizing request + - crypto: atmel-aes - guard unregister on error in atmel_aes_register_algs + - padata: Remove cpu online check from cpu add and removal + - padata: Put CPU offline callback in ONLINE section to allow failure + - PCI: dwc: rcar-gen4: Change EPC BAR alignment to 4K as per the + documentation + - accel/amdxdna: fix missing newline in pr_err message + - drm/amd/ras: Remove redundant NULL check in pending bad-bank list + iteration + - drm/amdgpu/gfx10: look at the right prop for gfx queue priority + - drm/amdgpu/gfx11: look at the right prop for gfx queue priority + - spi: hisi-kunpeng: prevent infinite while() loop in hisi_spi_flush_fifo + - PCI: sky1: Fix missing cleanup of ECAM config on probe failure + - drm/imagination: Switch reset_reason fields from enum to u32 + - iommu/tegra241-cmdqv: Set supports_cmd op in tegra241_vcmdq_hw_init() + - iommu/tegra241-cmdqv: Update uAPI to clarify HYP_OWN requirement + - drm/msm: add missing MODULE_DEVICE_ID definitions + - drm/msm/dpu: fix mismatch between power and frequency + - drm/msm/dsi: add the missing parameter description + - drm/msm/dpu: don't try using 2 LMs if only one DSC is available + - drm/msm/dsi: fix bits_per_pclk + - drm/msm/dsi: fix hdisplay calculation for CMD mode panel + - drm/msm/dsi: rename MSM8998 DSI version from V2_2_0 to V2_0_0 + - ASoC: rockchip: rockchip_sai: Set slot width for non-TDM mode + - tools/sched_ext: scx_pair: fix pair_ctx indexing for CPU pairs + - drm/panel: sharp-ls043t1le01: make use of prepare_prev_first + - drm/panel: simple: Correct G190EAN01 prepare timing + - PCI: qcom: Advertise Hotplug Slot Capability with no Command Completion + support + - PCI: Prevent shrinking bridge window from its required size + - PCI: Fix premature removal from realloc_head list during resource + assignment + - crypto: hisilicon/sec2 - prevent req used-after-free for sec + - PCI: Fix alignment calculation for resource size larger than align + - iommu/riscv: Fix signedness bug + - ALSA: core: Validate compress device numbers without dynamic minors + - drm/amd/display: Avoid NULL dereference in dc_dmub_srv error paths + - ASoC: amd: acp: update dmic_num logic for acp pdm dmic + - drm/amd/pm/ci: Use highest MCLK on CI when MCLK DPM is disabled + - drm/amd/pm/ci: Disable MCLK DPM on problematic CI ASICs + - drm/amd/pm/smu7: Fix SMU7 voltage dependency on display clock + - drm/amd/pm/ci: Fix powertune defaults for Hawaii 0x67B0 + - drm/amd/pm/ci: Clear EnabledForActivity field for memory levels + - drm/amd/pm/ci: Fill DW8 fields from SMC + - drm/amd/pm/smu7: Add SCLK cap for quirky Hawaii board + - drm/amdgpu/uvd4.2: Don't initialize UVD 4.2 when DPM is disabled + - PCI/DPC: Log AER error info for DPC/EDR uncorrectable errors + - hwmon: (aspeed-g6-pwm-tach): remove redundant driver remove callback + - ALSA: hda/realtek: fix bad indentation for alc269 + - ALSA: hda/realtek: fix code style (ERROR: else should follow close brace + '}') + - ASoC: SOF: Intel: hda: Place check before dereference + - drm/msm/vma: Avoid lock in VM_BIND fence signaling path + - drm/msm/a6xx: Add missing aperture_lock init + - drm/msm: Reject fb creation from _NO_SHARE objs + - drm/msm: Fix VM_BIND UNMAP locking + - drm/msm/a6xx: Fix HLSQ register dumping + - drm/msm/shrinker: Fix can_block() logic + - drm/msm/a6xx: Fix dumping A650+ debugbus blocks + - drm/msm/a6xx: Use barriers while updating HFI Q headers + - drm/msm/a8xx: Fix the ticks used in submit traces + - drm/msm/a6xx: Switch to preemption safe AO counter + - drm/msm/a6xx: Correct OOB usage + - drm/msm/adreno: Implement gx_is_on() for A8x + - drm/msm/a6xx: Fix gpu init from secure world + - ALSA: hda/cmedia: Remove duplicate pin configuration parsing + - pmdomain: ti: omap_prm: Fix a reference leak on device node + - pmdomain: imx: scu-pd: Fix device_node reference leak during ->probe() + - PM: domains: De-constify fields in struct dev_pm_domain_attach_data + - drm/msm/dpu: drop INTF_0 on MSM8953 + - ASoC: fsl_micfil: Add access property for "VAD Detected" + - ASoC: fsl_micfil: Fix event generation in hwvad_put_enable() + - ASoC: fsl_micfil: Fix event generation in hwvad_put_init_mode() + - ASoC: fsl_micfil: Fix event generation in micfil_range_set() + - ASoC: fsl_micfil: Fix event generation in micfil_put_dc_remover_state() + - ASoC: fsl_micfil: Fix event generation in micfil_quality_set() + - ASoC: fsl_xcvr: Fix event generation in fsl_xcvr_arc_mode_put() + - ASoC: fsl_xcvr: Fix event generation in fsl_xcvr_mode_put() + - ASoC: fsl_easrc: Check the variable range in fsl_easrc_iec958_put_bits() + - ASoC: fsl_easrc: Fix value type in fsl_easrc_iec958_get_bits() + - ASoC: fsl_easrc: Change the type for iec958 channel status controls + - iommu/amd: Fix clone_alias() to use the original device's devid + - iommu/riscv: Remove overflows on the invalidation path + - ASoC: qcom: qdsp6: topology: check widget type before accessing data + - PCI: dwc: Fix type mismatch for kstrtou32_from_user() return value + - crypto: qat - disable 4xxx AE cluster when lead engine is fused off + - crypto: qat - disable 420xx AE cluster when lead engine is fused off + - crypto: qat - fix compression instance leak + - crypto: qat - fix type mismatch in RAS sysfs show functions + - crypto: iaa - fix per-node CPU counter reset in rebalance_wq_table() + - crypto: qat - use swab32 macro + - ALSA: hda: Notify IEC958 Default PCM switch state changes + - ASoC: rsnd: Fix potential out-of-bounds access of component_dais[] + - PCI: Enable AtomicOps only if Root Port supports them + - PCI: imx6: Keep Root Port MSI capability with iMSI-RX to work around + hardware bug + - PCI: aspeed: Fix IRQ domain leak on platform_get_irq() failure + - dt-bindings: PCI: imx6q-pcie: Fix maxItems of clocks and clock-names + - PCI: mediatek-gen3: Prevent leaking IRQ domains when IRQ not found + - gpu: nova-core: bitfield: fix broken Default implementation + - selftests/mm: skip migration tests if NUMA is unavailable + - Documentation: fix a hugetlbfs reservation statement + - Docs/admin-guide/mm/damn/lru_sort: fix intervals autotune parameter name + - Docs/mm/damon/index: fix typo: autoamted -> automated + - zram: do not permit params change after init + - selftest: memcg: skip memcg_sock test if address family not supported + - gpu: nova-core: remove redundant `.as_ref()` for `dev_*` print + - gpu: nova-core: fix missing colon in SEC2 boot debug message + - ALSA: scarlett2: Add missing sentinel initializer field + - ASoC: qcom: audioreach: explicitly enable speaker protection modules + - ASoC: SOF: compress: return the configured codec from get_params + - PCI/NPEM: Set LED_HW_PLUGGABLE for hotplug-capable ports + - tools/sched_ext: Fix off-by-one in scx_sdt payload zeroing + - ASoC: soc-component: re-add pcm_new()/pcm_free() + - ASoC: amd: name back to pcm_new()/pcm_free() + - ASoC: amd: ps: fix the pcm device numbering for acp pdm dmic + - ALSA: usb-audio: qcom: Fix incorrect type in enable_audio_stream + - PCI: tegra194: Fix polling delay for L2 state + - PCI: tegra194: Increase LTSSM poll time on surprise link down + - PCI: tegra194: Disable LTSSM after transition to Detect on surprise link + down + - PCI: tegra194: Don't force the device into the D0 state before L2 + - PCI: tegra194: Disable PERST# IRQ only in Endpoint mode + - PCI: tegra194: Use devm_gpiod_get_optional() to parse "nvidia,refclk- + select" + - PCI: tegra194: Disable direct speed change for Endpoint mode + - PCI: tegra194: Set LTR message request before PCIe link up in Endpoint + mode + - PCI: tegra194: Allow system suspend when the Endpoint link is not up + - PCI: tegra194: Free up Endpoint resources during remove() + - PCI: tegra194: Use DWC IP core version + - PCI: dwc: Apply ECRC workaround to DesignWare 5.00a as well + - PCI: tegra194: Disable L1.2 capability of Tegra234 EP + - PCI: tegra194: Fix CBB timeout caused by DBI access before core power-on + - drm/fb-helper: Fix a locking bug in an error path + - PCI: cadence: Add flags for disabling ASPM capability for broken Root + Ports + - PCI: sg2042: Avoid L0s and L1 on Sophgo 2042 PCIe Root Ports + - ASoC: SDCA: Fix cleanup inversion in class driver + - spi: rzv2h-rspi: Fix invalid SPR=0/BRDV=0 clock configuration + - spi: mtk-snfi: unregister ECC engine on probe failure and remove() + callback + - ALSA: sc6000: Keep the programmed board state in card-private data + - dm cache: fix missing return in invalidate_committed's error path + - sched_ext: Track @p's rq lock across set_cpus_allowed_scx -> + ops.set_cpumask + - sched_ext: Fix ops.cgroup_move() invocation kf_mask and rq tracking + - spi: cadence-qspi: Revert the filtering of certain opcodes in ODTR + - crypto: jitterentropy - replace long-held spinlock with mutex + - ALSA: usb-audio: Exclude Scarlett 18i20 1st Gen from SKIP_IFACE_SETUP + - ALSA: hda/realtek - fixed speaker no sound update + - gfs2: Call unlock_new_inode before d_instantiate + - fanotify: avoid/silence premature LSM capability checks + - fanotify: call fanotify_events_supported() before path_permission() and + security_path_notify() + - fuse: fix uninit-value in fuse_dentry_revalidate() + - ktest: Avoid undef warning when WARNINGS_FILE is unset + - ktest: Honor empty per-test option overrides + - ktest: Run POST_KTEST hooks on failure and cancellation + - vfio: selftests: fix crash in vfio_dma_mapping_mmio_test + - rtla/utils: Fix resource leak in set_comm_sched_attr() + - tools/rtla: Generate optstring from long options + - rtla: Fix segfault on multiple SIGINTs + - vfio: selftests: Build tests on aarch64 + - gfs2: less aggressive low-memory log flushing + - quota: Fix race of dquot_scan_active() with quota deactivation + - vfio: unhide vdev->debug_root + - gfs2: add some missing log locking + - gfs2: prevent NULL pointer dereference during unmount + - efi/capsule-loader: fix incorrect sizeof in phys array reallocation + - ksmbd: fix use-after-free from async crypto on Qualcomm crypto engine + - arm64: dts: mediatek: mt8365: Describe infracfg-nao as a pure syscon + - ARM: dts: mediatek: mt7623: fix efuse fallback compatible + - memory: tegra124-emc: Fix dll_change check + - memory: tegra30-emc: Fix dll_change check + - arm64: dts: imx8-apalis: Fix LEDs name collision + - arm64: dts: imx91-11x11-evk: change usdhc tuning step for eMMC and SD + - riscv: dts: spacemit: pcie: fix missing power regulator + - arm64: dts: mediatek: mt7988a-bpi-r4pro: fix model string + - arm64: dts: rockchip: Make Jaguar PCIe-refclk pin use pull-up config + - arm64: dts: imx8mp-evk: Enable pull select bit for PCIe regulator GPIO + (M.2 W_DISABLE1) + - iommufd: vfio compatibility extension check for noiommu mode + - arm64: dts: qcom: sm6125-xiaomi-ginkgo: Remove board-id + - arm64: dts: qcom: sm6125-xiaomi-ginkgo: Correct reserved memory ranges + - arm64: dts: qcom: sm6125-xiaomi-ginkgo: Remove extcon + - arm64: dts: qcom: sm6125-xiaomi-ginkgo: Fix reserved gpio ranges + - arm64: dts: qcom: qcs6490-rubikpi3: Use lt9611 DSI Port B + - arm64: dts: qcom: talos: Add missing clock-names to GCC + - arm64: dts: ti: k3-am62l: include WKUP_UART0 in wakeup peripheral window + - arm64: dts: mediatek: mt6795: Fix gpio-ranges pin count + - arm64: dts: mediatek: mt7981b: Fix gpio-ranges pin count + - arm64: dts: mediatek: mt7986a: Fix gpio-ranges pin count + - iommufd/selftest: Fix page leaks in mock_viommu_{init,destroy} + - arm64: dts: imx8mp-kontron: Fix touch reset configuration on DL devices + - arm64: dts: imx8mp-kontron: Drop vmmc-supply to fix SD card on SMARC + eval carrier + - arm64: dts: imx8mp-hummingboard-pulse/cubox-m: fix vmmc gpio polarity + - arm64: dts: imx8mp-hummingboard-pulse: fix mini-hdmi dsi port reference + - ARM: dts: BCM5301X: Drop extra NAND controller compatible + - arm64: dts: qcom: msm8953-xiaomi-vince: correct wled ovp value + - arm64: dts: qcom: msm8937-xiaomi-land: correct wled ovp value + - arm64: dts: qcom: msm8953-xiaomi-daisy: fix backlight + - firmware: qcom_scm: don't opencode kmemdup + - soc: qcom: ubwc: disable bank swizzling for Glymur platform + - arm64: dts: rockchip: Fix Bluetooth stability on LCKFB TaiShan Pi + - Revert "arm64: dts: rockchip: add SPDIF audio to Beelink A1" + - arm64: dts: rockchip: Correct Fan Supply for Gameforce Ace + - arm64: dts: rockchip: Correct Joystick Axes on Gameforce Ace + - soc: qcom: ocmem: make the core clock optional + - soc: qcom: ocmem: register reasons for probe deferrals + - soc: qcom: ocmem: return -EPROBE_DEFER is ocmem is not available + - riscv: dts: spacemit: drop incorrect pinctrl for combo PHY + - arm64: dts: rockchip: Fix RK3562 EVB2 model name + - arm64: dts: rockchip: Add mphy reset to ufshc node + - bus: rifsc: fix RIF configuration check for peripherals + - arm64: dts: qcom: arduino-imola: fix faulty spidev node + - Revert "UBUNTU: SAUCE: arm64: dts: add missing denali-oled.dtb to + Makefile" + - arm64: dts: qcom: add missing denali-oled.dtb to Makefile + - arm64: dts: qcom: hamoa: correct Iris corners for the MXC rail + - arm64: dts: qcom: lemans: correct Iris corners for the MXC rail + - arm64: dts: qcom: monaco: correct Iris corners for the MXC rail + - arm64: dts: qcom: sm8550: correct Iris corners for the MXC rail + - arm64: dts: qcom: sm8650: correct Iris corners for the MXC rail + - arm64: dts: qcom: sm8750: correct Iris corners for the MXC rail + - arm64: dts: qcom: kaanapali: Fix GIC_ITS range length + - arm64: dts: qcom: milos: Fix GIC_ITS range length + - arm64: dts: qcom: sm8450: Fix GIC_ITS range length + - arm64: dts: qcom: sm8550: Fix GIC_ITS range length + - arm64: dts: qcom: sm8650: Fix GIC_ITS range length + - arm64: dts: qcom: sm8750: Fix GIC_ITS range length + - arm64: dts: qcom: sm8550: Fix xo clock supply of platform SD host + controller + - arm64: dts: qcom: sm8650: Fix xo clock supply of SD host controller + - arm64: dts: qcom: hamoa: Fix xo clock supply of platform SD host + controller + - arm64: dts: qcom: sm8450: Enable UHS-I SDR50 and SDR104 SD card modes + - arm64: dts: qcom: sm8550: Enable UHS-I SDR50 and SDR104 SD card modes + - arm64: dts: qcom: sm8650: Enable UHS-I SDR50 and SDR104 SD card modes + - arm64: dts: qcom: sm7225-fairphone-fp4: Fix conflicting bias pinctrl + - arm64: dts: qcom: sdm845-xiaomi-beryllium: Mark l1a regulator as powered + during boot + - arm64: dts: qcom: msm8917-xiaomi-riva: Fix board-id for all bootloader + - arm64: dts: ti: k3-am62p5-sk: Disable MMC1 internal pulls on data pins + - arm64: dts: ti: k3-am62l3-evm: Disable MMC1 internal pulls on data pins + - arm64: dts: ti: k3-am62-lp-sk: Enable internal pulls for MMC0 data pins + - arm64: dts: ti: k3-am62-verdin: Fix SPI_1 GPIO CS pinctrl label + - arm64: dts: freescale: imx8mp-tqma8mpql-mba8mp-ras314: fix UART1 RTS/CTS + muxing + - arm64: dts: imx91: Remove TMU's superfluous sensor ID + - arm64: dts: imx8mp-kontron: Fix boot order for PMIC and RTC + - arm64: dts: imx8dxl-evk: Use audio-graph-card2 for wm8960-2 and wm8960-3 + - arm64: dts: imx8mp-evk: Specify ADV7535 register addresses + - arm64: dts: lx2160a: change i2c0 (iic1) pinmux mask to one bit + - arm64: dts: lx2160a: remove duplicate pinmux nodes + - arm64: dts: lx2160a: rename pinmux nodes for readability + - arm64: dts: lx2160a: add sda gpio references for i2c bus recovery + - arm64: dts: lx2160a: change zeros to hexadecimal in pinmux nodes + - arm64: dts: lx2160a: complete pinmux for rcwsr12 configuration word + - arm64: dts: imx8qm-mek: switch Type-C connector power-role to dual + - arm64: dts: imx8qxp-mek: switch Type-C connector power-role to dual + - soc/tegra: cbb: Set ERD on resume for err interrupt + - soc/tegra: cbb: Fix incorrect ARRAY_SIZE in fabric lookup tables + - soc/tegra: cbb: Fix cross-fabric target timeout lookup + - arm64: tegra: Fix RTC aliases + - soc/tegra: pmc: Add kerneldoc for reboot notifier + - soc/tegra: pmc: Correct function names in kerneldoc + - soc/tegra: pmc: Add kerneldoc for wake-up variables + - unshare: fix nsproxy leak in ksys_unshare() on set_cred_ucounts() + failure + - ocfs2/dlm: validate qr_numregions in dlm_match_regions() + - ocfs2/dlm: fix off-by-one in dlm_match_regions() region comparison + - soc: qcom: llcc: fix v1 SB syndrome register offset + - soc: qcom: aoss: compare against normalized cooling state + - arm64: dts: qcom: milos: Add missing CX power domain to GCC + - arm64: dts: qcom: sm8250: Add missing CPU7 3.09GHz OPP + - ARM: OMAP1: Fix DEBUG_LL and earlyprintk on OMAP16XX + - arm64/xor: fix conflicting attributes for xor_block_template + - lib: kunit_iov_iter: fix memory leaks + - ARM: dts: imx27-eukrea: replace interrupts with interrupts-extended + - firmware: arm_ffa: Use the correct buffer size during RXTX_MAP + - fwctl: Fix class init ordering to avoid NULL pointer dereference on + device removal + - ocfs2: fix listxattr handling when the buffer is full + - ocfs2: validate bg_bits during freefrag scan + - ocfs2: validate group add input before caching + - dmaengine: dw-axi-dmac: fix Alignment should match open parenthesis + - dmaengine: dw-axi-dmac: Remove unnecessary return statement from void + function + - phy: apple: apple: Use local variable for ioremap return value + - soundwire: bus: demote UNATTACHED state warnings to dev_dbg() + - soundwire: Intel: test bus.bpt_stream before assigning it + - dmaengine: mxs-dma: Fix missing return value from + of_dma_controller_register() + - soundwire: cadence: Clear message complete before signaling waiting + thread + - tracing: move __printf() attribute on __ftrace_vbprintk() + - tracing: Rebuild full_name on each hist_field_name() call + - hte: tegra194: remove Kconfig dependency on Tegra194 SoC + - [Config] Enable CONFIG_HTE_TEGRA194 on armhf + - remoteproc: xlnx: Fix sram property parsing + - stop_machine: Fix the documentation for a NULL cpus argument + - remoteproc: imx_rproc: Check return value of regmap_attach_dev() in + imx_rproc_mmio_detect_mode() + - ima: check return value of crypto_shash_final() in boot aggregate + - HID: asus: make asus_resume adhere to linux kernel coding standards + - HID: asus: do not abort probe when not necessary + - workqueue: devres: Add device-managed allocate workqueue + - power: supply: max77705: Drop duplicated IRQ error message + - power: supply: max77705: Free allocated workqueue and fix removal order + - mtd: physmap_of_gemini: Fix disabled pinctrl state check + - ima_fs: Correctly create securityfs files for unsupported hash algos + - dt-bindings: interrupt-controller: arm,gic-v3: Fix EPPI range + - mtd: spi-nor: core: correct the op.dummy.nbytes when check read + operations + - mtd: spi-nor: update spi_nor_fixups::post_sfdp() documentation + - mtd: spi-nor: micron-st: add SNOR_CMD_PP_8_8_8_DTR sfdp fixup for + mt35xu512aba + - mtd: spi-nor: swp: check SR_TB flag when getting tb_mask + - mtd: parsers: ofpart: call of_node_put() only in ofpart_fail path + - mtd: parsers: ofpart: call of_node_get() for dedicated subpartitions + - cxl/pci: Check memdev driver binding status in cxl_reset_done() + - mtd: rawnand: sunxi: fix sunxi_nfc_hw_ecc_read_extra_oob + - mtd: spinand: winbond: Clarify when to enable the HS bit + - HID: usbhid: fix deadlock in hid_post_reset() + - ext4: fix miss unlock 'sb->s_umount' in extents_kunit_init() + - ext4: call deactivate_super() in extents_kunit_exit() + - ext4: fix the error handling process in extents_kunit_init). + - ext4: fix possible null-ptr-deref in extents_kunit_exit() + - ext4: fix possible null-ptr-deref in mbt_kunit_exit() + - bpf, arm64: Reject out-of-range B.cond targets + - bpf, arm64: Fix off-by-one in check_imm signed range check + - bpf, arm64: Remove redundant bpf_flush_icache() after pack allocator + finalize + - bpf, riscv: Remove redundant bpf_flush_icache() after pack allocator + finalize + - bpf, sockmap: Fix af_unix iter deadlock + - bpf, sockmap: Fix af_unix null-ptr-deref in proto update + - bpf, sockmap: Take state lock for af_unix iter + - bpf: Fix precedence bug in convert_bpf_ld_abs alignment check + - bpf: Fix NULL deref in map_kptr_match_type for scalar regs + - bpf: allow UTF-8 literals in bpf_bprintf_prepare() + - libbpf: Prevent double close and leak of btf objects + - bpf: Validate node_id in arena_alloc_pages() + - bpf, arm32: Reject BPF-to-BPF calls and callbacks in the JIT + - perf trace: Fix IS_ERR() vs NULL check bug + - dt-bindings: pinctrl: marvell,armada3710-xb-pinctrl: add missing items + keyword + - pinctrl: pinctrl-pic32: Fix resource leak + - perf trace: Avoid an ERR_PTR in syscall_stats + - pinctrl: microchip-mssio: Fix missing return in probe + - perf test type profiling: Remote typedef on struct + - pinctrl: cy8c95x0: remove duplicate error message + - pinctrl: cy8c95x0: Unify messages with help of dev_err_probe() + - pinctrl: cy8c95x0: Avoid returning positive values to user space + - perf branch: Avoid incrementing NULL + - perf: tools: cs-etm: Fix print issue for Coresight debug in ETE/TRBE + trace + - pinctrl: pinconf-generic: Fully validate 'pinmux' property + - pinctrl: realtek: Fix function signature for config argument + - pinctrl: abx500: Fix type of 'argument' variable + - pinctrl: renesas: rzg2l: Fix save/restore of {IOLH,IEN,PUPD,SMT} + registers + - tools build: Correct link flags for libopenssl + - perf lock: Fix option value type in parse_max_stack + - perf stat: Fix opt->value type for parse_cache_level + - memblock: reserve_mem: fix end caclulation in + reserve_mem_release_by_name() + - perf stat: Fix crash on arm64 + - perf tools: Fix module symbol resolution for non-zero .text sh_addr + - perf test: Fix ratio_to_prev event parsing test + - perf test: Skip perf data type profiling tests for s390 + - perf expr: Return -EINVAL for syntax error in expr__find_ids() + - perf metrics: Make common stalled metrics conditional on having the + event + - ipmi: ssif_bmc: fix missing check for copy_to_user() partial failure + - ipmi: ssif_bmc: fix message desynchronization after truncated response + - ipmi: ssif_bmc: change log level to dbg in irq callback + - perf cgroup: Update metric leader in evlist__expand_cgroup + - pinctrl: sophgo: pinctrl-sg2042: Fix wrong module description + - pinctrl: sophgo: pinctrl-sg2044: Fix wrong module description + - perf maps: Fix fixup_overlap_and_insert that can break sorted by name + order + - perf maps: Fix copy_from that can break sorted by name order + - perf util: Kill die() prototype, dead for a long time + - i3c: master: dw-i3c: Fix missing reset assertion in remove() callback + - i3c: master: dw-i3c: Balance PM runtime usage count on probe failure + - i3c: master: renesas: Fix memory leak in renesas_i3c_i3c_xfers() + - i3c: dw: Fix memory leak in dw_i3c_master_i3c_xfers() + - i3c: master: adi: Fix error propagation for CCCs + - i3c: mipi-i3c-hci: fix IBI payload length calculation for final status + - fs/ntfs3: prevent uninitialized lcn caused by zero len + - backlight: sky81452-backlight: Check return value of + devm_gpiod_get_optional() in sky81452_bl_parse_dt() + - platform/surface: surfacepro3_button: Drop wakeup source on remove + - leds: lgm-sso: Remove duplicate assignments for priv->mmap + - usb: typec: Fix error pointer dereference + - tty: hvc_iucv: fix off-by-one in number of supported devices + - usb: typec: ps883x: Fix Oops at unbind + - platform/x86: panasonic-laptop: Fix OPTD notifier registration and + cleanup + - platform/x86: barco-p50-gpio: normalize return value of gpio_get + - fs/ntfs3: fix missing run load for vcn0 in attr_data_get_block_locked() + - mfd: mc13xxx-core: Fix memory leak in mc13xxx_add_subdevice_pdata() + - nfs/blocklayout: Fix compilation error (`make W=1`) in + bl_write_pagelist() + - sunrpc: Kill RPC_IFDEBUG() + - sunrpc: Fix compilation error (`make W=1`) when dprintk() is no-op + - NFSD: fix nfs4_file access extra count in nfsd4_add_rdaccess_to_wrdeleg + - nfsd: use dynamic allocation for oversized NFSv4.0 replay cache + - RDMA/umem: Use consistent DMA attributes when unmapping entries + - greybus: raw: fix use-after-free on cdev close + - greybus: raw: fix use-after-free if write is called after disconnect + - platform/x86: asus-wmi: adjust screenpad power/brightness handling + - platform/x86: asus-wmi: fix screenpad brightness range + - tty: serial: ip22zilog: Fix section mispatch warning + - fs/ntfs3: terminate the cached volume label after UTF-8 conversion + - platform/x86: hp-wmi: fix ignored return values in fan settings + - platform/x86: hp-wmi: avoid cancel_delayed_work_sync from work handler + - platform/x86: hp-wmi: use mod_delayed_work to reset keep-alive timer + - platform/x86: hp-wmi: fix u8 underflow in gpu_delta calculation + - platform/x86: hp-wmi: add locking for concurrent hwmon access + - platform/x86: dell_rbu: avoid uninit value usage in packet_size_write() + - platform/x86: dell-wmi-sysman: bound enumeration string aggregation + - RDMA/core: Prefer NLA_NUL_STRING + - platform/x86: hp-wmi: fix fan table parsing + - dt-bindings: clock: qcom: Add GCC video axi reset clock for Glymur + - clk: qcom: gcc-glymur: Add video axi clock resets for glymur + - clk: qcom: dispcc-glymur: use RCG2 ops for DPTX1 AUX clock source + - clk: qcom: dispcc-sm8450: use RCG2 ops for DPTX1 AUX clock source + - clk: renesas: r9a09g057: Fix ordering of module clocks array + - clk: renesas: r9a09g056: Fix ordering of module clocks array + - clk: sunxi-ng: sun55i-a523-r: Add missing r-spi module clock + - scsi: sg: Fix sysctl sg-big-buff register during sg_init() + - scsi: sg: Resolve soft lockup issue when opening /dev/sgX + - clk: qcom: dispcc-sc8280xp: remove CLK_SET_RATE_PARENT from + byte_div_clk_src dividers + - clk: qcom: dispcc-glymur: Fix DSI byte clock rate setting + - clk: qcom: dispcc-kaanapali: Fix DSI byte clock rate setting + - clk: qcom: dispcc-milos: Fix DSI byte clock rate setting + - clk: qcom: dispcc-sm4450: Fix DSI byte clock rate setting + - clk: qcom: dispcc[01]-sa8775p: Fix DSI byte clock rate setting + - clk: renesas: r9a09g057: Remove entries for WDT{0,2,3} + - scsi: target: core: Fix integer overflow in UNMAP bounds check + - scsi: ufs: rockchip,rk3576-ufshc: dt-bindings: Add new mphy reset item + - dt-bindings: clock: qcom,gcc-sc8180x: Add missing GDSCs + - clk: qcom: gcc-sc8180x: Add missing GDSCs + - clk: qcom: gcc-sc8180x: Use retention for USB power domains + - clk: qcom: gcc-sc8180x: Use retention for PCIe power domains + - clk: qcom: dispcc-sm8250: Use shared ops on the mdss vsync clk + - clk: qcom: dispcc-sm8250: Enable parents for pixel clocks + - clk: imx: imx6q: Fix device node reference leak in pll6_bypassed() + - clk: imx: imx6q: Fix device node reference leak in + of_assigned_ldb_sels() + - clk: imx8mq: Correct the CSI PHY sels + - um: Fix potential race condition in TLB sync + - x86/um: fix vDSO installation + - clk: qoriq: avoid format string warning + - clk: xgene: Fix mapping leak in xgene_pllclk_init() + - clk: spacemit: ccu_mix: fix inverted condition in ccu_mix_trigger_fc() + - f2fs: avoid reading already updated pages during GC + - printk_ringbuffer: Fix get_data() size sanity check + - clk: qcom: gdsc: Fix error path on registration of multiple pm + subdomains + - lib/hexdump: print_hex_dump_bytes() calls print_hex_dump_debug() + - f2fs: fix data loss caused by incorrect use of nat_entry flag + - f2fs: fix to preserve previous reserve_{blocks,node} value when remount + - scsi: hpsa: Enlarge controller and IRQ name buffers + - drm/amd/display: Fix parameter mismatch in panel self-refresh helper + - clk: qcom: gcc-x1e80100: Keep GCC USB QTB clock always ON + - clk: visconti: pll: initialize clk_init_data to zero + - f2fs: allow empty mount string for Opt_usr|grp|projjquota + - f2fs: protect extension_list reading with sb_lock in f2fs_sbi_show() + - drm/i915/wm: Verify the correct plane DDB entry + - virt: arm-cca-guest: fix error check for RSI_INCOMPLETE + - crypto: eip93 - fix hmac setkey algo selection + - crypto: sa2ul - Fix AEAD fallback algorithm names + - crypto: ccp - copy IV using skcipher ivsize + - sh: Include in dac.h + - erofs: unify lcn as u64 for 32-bit platforms + - tools: hv: Fix cross-compilation + - Drivers: hv: vmbus: fix hyperv_cpuhp_online variable shadowing + - arm64: dts: imx8mp-debix-model-a: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-debix-som-a: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-navqp: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-icore-mx8mp: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-edm-g: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-aristainetos3a-som-v1: Correct PAD settings for + PMIC_nINT + - arm64: dts: imx8mp-nitrogen-som: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-sr-som: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-ultra-mach-sbc: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-dhcom-som: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mp-data-modul-edm-sbc: Correct PAD settings for + PMIC_nINT + - PCMCIA: Fix garbled log messages for KERN_CONT + - reset: amlogic: t7: Fix null reset ops + - arm64: dts: imx8mm-emtop-som: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mn-tqma8mqnl: Correct PAD settings for PMIC_nINT + - arm64: dts: imx8mm-tqma8mqml: Correct PAD settings for PMIC_nINT + - arm64: dts: marvell: armada-37xx: use 'usb2-phy' in USB3 controller's + phy-names + - pwm: stm32: Fix rounding issue for requests with inverted polarity + - net/sched: act_mirred: fix wrong device for mac_header_xmit check in + tcf_blockcast_redir + - macvlan: fix macvlan_get_size() not reserving space for + IFLA_MACVLAN_BC_CUTOFF + - net/sched: sch_cake: fix NAT destination port not being updated in + cake_update_flowkeys + - nexthop: fix IPv6 route referencing IPv4 nexthop + - net: airoha: Wait for NPU PPE configuration to complete in + airoha_ppe_offload_setup() + - net/sched: taprio: fix use-after-free in advance_sched() on schedule + switch + - net: dsa: remove redundant netdev_lock_ops() from conduit ethtool ops + - net: enetc: correct the command BD ring consumer index + - net: enetc: fix NTMP DMA use-after-free issue + - ksmbd: fix use-after-free in smb2_open during durable reconnect + - tcp: move tp->chrono_type next tp->chrono_stat[] + - tcp: inline tcp_chrono_start() + - tcp: annotate data-races in tcp_get_info_chrono_stats() + - tcp: add data-race annotations around tp->data_segs_out and + tp->total_retrans + - tcp: add data-races annotations around tp->reordering, tp->snd_cwnd + - tcp: annotate data-races around tp->snd_ssthresh + - tcp: annotate data-races around tp->delivered and tp->delivered_ce + - tcp: add data-race annotations for TCP_NLA_SNDQ_SIZE + - tcp: annotate data-races around tp->bytes_sent + - tcp: annotate data-races around tp->bytes_retrans + - tcp: annotate data-races around tp->dsack_dups + - tcp: annotate data-races around tp->reord_seen + - tcp: annotate data-races around tp->srtt_us + - tcp: annotate data-races around tp->timeout_rehash + - tcp: annotate data-races around (tp->write_seq - tp->snd_nxt) + - tcp: annotate data-races around tp->plb_rehash + - ice: fix 'adjust' timer programming for E830 devices + - ice: update PCS latency settings for E825 10G/25Gb modes + - ice: fix double-free of tx_buf skb + - ice: fix PHY config on media change with link-down-on-close + - ice: fix ICE_AQ_LINK_SPEED_M for 200G + - ice: fix race condition in TX timestamp ring cleanup + - ice: fix potential NULL pointer deref in error path of + ice_set_ringparam() + - i40e: don't advertise IFF_SUPP_NOFCS + - iavf: fix wrong VLAN mask for legacy Rx descriptors L2TAG2 + - e1000e: Unroll PTP in probe error handling + - ipv6: fix possible UAF in icmpv6_rcv() + - af_unix: Drop all SCM attributes for SOCKMAP. + - sctp: fix OOB write to userspace in sctp_getsockopt_peer_auth_chunks + - pppoe: drop PFC frames + - net/mlx5: Fix HCA caps leak on notifier init failure + - net: airoha: Fix possible TX queue stall in airoha_qdma_tx_napi_poll() + - netfilter: nft_osf: restrict it to ipv4 + - netfilter: conntrack: remove sprintf usage + - netfilter: xtables: restrict several matches to inet family + - netfilter: nat: use kfree_rcu to release ops + - ipvs: fix MTU check for GSO packets in tunnel mode + - netfilter: nfnetlink_osf: fix out-of-bounds read on option matching + - netfilter: nfnetlink_osf: fix potential NULL dereference in ttl check + - net/sched: sch_dualpi2: drain both C-queue and L-queue in + dualpi2_change() + - arm64: dts: amlogic: meson-axg: Add missing cache information to cpu0 + - arm64: dts: meson-gxl-p230: fix ethernet PHY interrupt number + - vfio/pci: Clean up DMABUFs before disabling function + - pwm: atmel-tcb: Cache clock rates and mark chip as atomic + - ksmbd: destroy tree_conn_ida in ksmbd_session_destroy() + - ksmbd: destroy async_ida in ksmbd_conn_free() + - ksmbd: fix durable fd leak on ClientGUID mismatch in durable v2 open + - ksmbd: scope conn->binding slowpath to bound sessions only + - net: validate skb->napi_id in RX tracepoints + - bnge: fix initial HWRM sequence + - bnge: remove unsupported backing store type + - sctp: fix sockets_allocated imbalance after sk_clone() + - net/rds: zero per-item info buffer before handing it to visitors + - ice: fix timestamp interrupt configuration for E825C + - ice: perform PHY soft reset for E825C ports at initialization + - ice: fix ready bitmap check for non-E822 devices + - ice: fix ice_ptp_read_tx_hwtstamp_status_eth56g + - net_sched: sch_hhf: annotate data-races in hhf_dump_stats() + - net/sched: sch_pie: annotate data-races in pie_dump_stats() + - net/sched: sch_fq_codel: remove data-races from fq_codel_dump_stats() + - net/sched: sch_red: annotate data-races in red_dump_stats() + - net/sched: sch_sfb: annotate data-races in sfb_dump_stats() + - net: airoha: Move ndesc initialization at end of airoha_qdma_init_tx() + - net: airoha: Add missing bits in airoha_qdma_cleanup_tx_queue() + - net: dsa: realtek: rtl8365mb: fix mode mask calculation + - net: airoha: Move ndesc initialization at end of + airoha_qdma_init_rx_queue() + - net: airoha: Rework the code flow in airoha_remove() and in + airoha_probe() error path + - net: airoha: Add size check for TX NAPIs in airoha_qdma_cleanup() + - net: mana: Init link_change_work before potential error paths in probe + - net: mana: Init gf_stats_work before potential error paths in probe + - net: mana: Guard mana_remove against double invocation + - net: mana: Don't overwrite port probe error with add_adev result + - net: mana: Fix EQ leak in mana_remove on NULL port + - vsock/virtio: fix MSG_ZEROCOPY pinned-pages accounting + - virtio_net: sync rss_trailer.max_tx_vq on queue_pairs change via + VQ_PAIRS_SET + - nfp: fix swapped arguments in nfp_encode_basic_qdr() calls + - tcp: send a challenge ACK on SEG.ACK > SND.NXT + - tipc: fix double-free in tipc_buf_append() + - vhost_net: fix sleeping with preempt-disabled in vhost_net_busy_poll() + - nstree: fix func. parameter kernel-doc warnings + - eventpoll: use hlist_is_singular_node() in __ep_remove() + - eventpoll: split __ep_remove() + - eventpoll: kill __ep_remove() + - eventpoll: drop vestigial __ prefix from ep_remove_{file,epi}() + - eventpoll: move epi_fget() up + - fs/adfs: validate nzones in adfs_validate_bblk() + - rtc: abx80x: Disable alarm feature if no interrupt attached + - kbuild: builddeb - avoid recompiles for non-cross-compiles + - tools/power turbostat: Fix AMD RAPL regression on big systems + - fbdev: offb: fix PCI device reference leak on probe failure + - tools/power turbostat: Fix unrecognized option '-P' + - tools/power turbostat: Fix --cpu-set 0 regression on HT systems + - tools/power turbostat: Fix --cpu-set 1 regression on HT systems + - kbuild: Never respect CONFIG_WERROR / W=e to fixdep + - mailbox: mtk-vcp-mailbox: Fix the return value in mtk_vcp_mbox_xlate() + - mailbox: mtk-cmdq: Fix CURR and END addr for task insert case + - mailbox: mailbox-test: free channels on probe error + - sched/psi: fix race between file release and pressure write + - cgroup/rdma: fix integer overflow in rdmacg_try_charge() + - cgroup/cpuset: record DL BW alloc CPU for attach rollback + - mailbox: add sanity check for channel array + - mailbox: mailbox-test: handle channel errors consistently + - mailbox: mailbox-test: don't free the reused channel + - mailbox: mailbox-test: initialize struct earlier + - mailbox: mailbox-test: make data_ready a per-instance variable + - fsnotify: fix inode reference leak in fsnotify_recalc_mask() + - btrfs: fix bytes_may_use leak in move_existing_remap() + - btrfs: fix bytes_may_use leak in do_remap_reloc_trans() + - btrfs: don't clobber errors in add_remap_tree_entries() + - btrfs: fix double-decrement of bytes_may_use in + submit_one_async_extent() + - tracing: branch: Fix inverted check on stat tracer registration + - nvmet-tcp: propagate nvmet_tcp_build_pdu_iovec() errors to its callers + - netfilter: nf_tables: use list_del_rcu for netlink hooks + - rculist: add list_splice_rcu() for private lists + - netfilter: nf_tables: join hook list via splice_list_rcu() in commit + phase + - netfilter: nf_tables: add hook transactions for device deletions + - nvme-pci: fix missed admin queue sq doorbell write + - drm/amdgpu: avoid double drm_exec_fini() in userq validate + - drm/amdgpu/gmc: Fix AMDGPU_GART_PLACEMENT_LOW to not overlap with VRAM + - drm/amd/pm: fix missing fine-grained dpm table flag on aldebaran + - drm/amdgpu: fix AMDGPU_INFO_READ_MMR_REG + - drm/amdgpu/uvd3.1: Don't validate the firmware when already validated + - drm/amdgpu/gfx6: Support harvested SI chips with disabled TCCs (v2) + - drm/amdgpu: Only send RMA CPER when threshold is exceeded + - netfilter: xt_policy: fix strict mode inbound policy matching + - netfilter: nf_conntrack_sip: don't use simple_strtoul + - spi: rzv2h-rspi: Fix silent failure in clock setup error path + - ASoC: amd: acp: Add DMI quirk for Valve Steam Deck OLED + - ASoC: SOF: Intel: add an empty adr_link + - spi: rockchip: Read ISR, not IMR, to detect cs-inactive IRQ + - ASoC: tas2764: Mark die temp register as volatile + - ASoC: tas2770: Fix order of operations for temperature calculation + - drm/sysfb: ofdrm: fix PCI device reference leaks + - drm/color-mgmt: Typo s/R332/RGB332/ + - arm64/scs: Fix potential sign extension issue of advance_loc4 + - spi: spi-mem: Add a packed command operation + - mtd: spinand: Add support for packed read data ODTR commands + - mtd: spinand: winbond: Set the packed page read flag to W35N02/04JW + - mtd: spinand: winbond: Fix ODTR write VCR on W35NxxJW + - ACPICA: Provide #defines for EINJV2 error types + - ACPI: APEI: EINJ: Fix EINJV2 memory error injection + - cdrom, scsi: sr: propagate read-only status to block layer via + set_disk_ro() + - spi: axiado: replace usleep_range() with udelay() in IRQ path + - netdevsim: zero initialize struct iphdr in dummy sk_buff + - net/sched: netem: fix probability gaps in 4-state loss model + - net/sched: netem: fix queue limit check to include reordered packets + - net/sched: netem: only reseed PRNG when seed is explicitly provided + - net/sched: netem: validate slot configuration + - net/sched: netem: fix slot delay calculation overflow + - net/sched: netem: check for negative latency and jitter + - net: airoha: fix BQL imbalance in TX path + - net: airoha: stop net_device TX queue before updating CPU index + - net: airoha: fix typo in function name + - net: airoha: Do not wake all netdev TX queues in + airoha_qdma_wake_netdev_txqs() + - net: airoha: Do not read uninitialized fragment address in + airoha_dev_xmit() + - net/sched: sch_choke: annotate data-races in choke_dump_stats() + - net/sched: sch_fq_pie: annotate data-races in fq_pie_dump_stats() + - vrf: Fix a potential NPD when removing a port from a VRF + - net: usb: rtl8150: fix use-after-free in rtl8150_start_xmit() + - net: usb: rtl8150: free skb on usb_submit_urb() failure in xmit + - spi: amlogic-spisg: initialize completion before requesting IRQ + - NFC: trf7970a: Ignore antenna noise when checking for RF field + - net: phonet: do not BUG_ON() in pn_socket_autobind() on failed bind + - neigh: let neigh_xmit take skb ownership + - tcp: make probe0 timer handle expired user timeout + - netpoll: fix IPv6 local-address corruption + - ALSA: usb-audio: Fix potential leak of pd at parsing UAC3 streams + - sched/fair: Fix wakeup_preempt_fair() vs delayed dequeue + - sched/fair: Clear rel_deadline when initializing forked entities + - net: mctp i2c: check length before marking flow active + - md/raid1,raid10: don't fail devices for invalid IO errors + - md: add fallback to correct bitmap_ops on version mismatch + - md: factor bitmap creation away from sysfs handling + - md/md-bitmap: split bitmap sysfs groups + - md/md-bitmap: add a none backend for bitmap grow + - s390/mm: Fix phys_to_folio() usage in do_secure_storage_access() + - net: phy: dp83869: fix setting CLK_O_SEL field. + - drm/amd/display: properly handle family setting for early GC 11.5.4 + - drm/amdgpu/vcn: set no_user_fence for VCN v2.0 enc/dec rings + - drm/amdgpu/vcn: set no_user_fence for VCN v2.5 enc/dec rings + - drm/amdgpu/vcn: set no_user_fence for VCN v3.0 enc/dec rings + - drm/amdgpu/vcn: set no_user_fence for VCN v4.0 enc ring + - drm/amdgpu/vcn: set no_user_fence for VCN v4.0.3 enc ring + - drm/amdgpu/vcn: set no_user_fence for VCN v4.0.5 enc ring + - drm/amdgpu/vcn: set no_user_fence for VCN v5.0.0 enc ring + - drm/amdgpu/vcn: set no_user_fence for VCN v5.0.1 enc ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v2.0 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v2.5 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v3.0 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v4.0 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v4.0.3 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v4.0.5 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v5.0.0 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v5.0.1 ring + - drm/amdgpu/jpeg: set no_user_fence for JPEG v5.3.0 ring + - drm/amd/pm: Add fine grained flag to SMU v13.0.6 + - io_uring/napi: cap busy_poll_to 10 msec + - ASoC: cs35l56: Fix illegal writes to OTP_MEM registers + - net: psp: check for device unregister when creating assoc + - net: psp: require admin permission for dev-set and key-rotate + - ASoC: codecs: ab8500: Fix casting of private data + - netfilter: skip recording stale or retransmitted INIT + - sctp: discard stale INIT after handshake completion + - net/sched: sch_cake: annotate data-races in cake_dump_stats() (I) + - net/sched: sch_cake: annotate data-races in cake_dump_stats() (II) + - net/sched: sch_cake: annotate data-races in cake_dump_stats() (III) + - net/sched: sch_cake: annotate data-races in cake_dump_stats() (IV) + - net/sched: sch_cake: annotate data-races in cake_dump_stats() (V) + - netconsole: return count instead of strnlen(buf, count) from store + callbacks + - netconsole: avoid clobbering userdatum value on truncated write + - netconsole: propagate device name truncation in dev_name_store() + - netconsole: restore userdatum value on update_userdata() failure + - ALSA: hda/conexant: Fix missing error check for jack detection + - ALSA: hda: cs35l56: Fix uninitialized value in cs35l56_hda_read_acpi() + - ALSA: hda/tas2781: Fix incorrect bit update for non-book-zero or book 0 + pages >1 + - futex: Prevent lockup in requeue-PI during signal/ timeout wakeup + - drm/amd/display: Allow embedded connectors without DDC + - drm/amd/display: Allow DCE link encoder without AUX registers + - drm/amd/display: Allow constructing DCE6 link encoder without DDC + - drm/amd/display: Allow constructing DCE8 link encoder without DDC + - drm/amd/display: Read EDID from VBIOS embedded panel info + - drm/amd/display: Use EDID from VBIOS embedded panel info + - drm/xe: Use XE_WEDGED_MODE_UPON_ANY_HANG_NO_RESET enum instead of magic + number + - drm/xe: Drop registration of guc_submit_wedged_fini from + xe_guc_submit_wedge() + - drm/xe/debugfs: Correct printing of register whitelist ranges + - drm/xe: Fix potential NULL deref in + xe_exec_queue_tlb_inval_last_fence_put_unlocked + - drm/xe: Fix error cleanup in xe_exec_queue_create_ioctl() + - drm/xe/eustall: Fix drm_dev_put called before stream disable in close + - drm/xe/gsc: Fix BO leak on error in query_compatibility_version() + - drm/xe/xelp: Fix Wa_18022495364 + - net: airoha: Do not return err in ndo_stop() callback + - bonding: print churn state via netlink + - bonding: 3ad: implement proper RCU rules for port->aggregator + - page_pool: fix memory-provider leak in page_pool_create_percpu() error + path + - iavf: rename IAVF_VLAN_IS_NEW to IAVF_VLAN_ADDING + - iavf: stop removing VLAN filters from PF on interface down + - iavf: wait for PF confirmation before removing VLAN filters + - iavf: add VIRTCHNL_OP_ADD_VLAN to success completion handler + - ice: fix NULL pointer dereference in ice_reset_all_vfs() + - ice: fix infinite recursion in ice_cfg_tx_topo via ice_init_dev_hw + - ice: fix missing SMA pin initialization in DPLL subsystem + - ice: fix SMA and U.FL pin state changes affecting paired pin + - ice: fix missing dpll notifications for SW pins + - dpll: export __dpll_pin_change_ntf() for use under dpll_lock + - ice: add dpll peer notification for paired SMA and U.FL pins + - net: tls: fix strparser anchor skb leak on offload RX setup failure + - sfc: fix error code in efx_devlink_info_running_versions() + - net/sched: cls_flower: revert unintended changes + - kselftest/arm64: Include for user_gcs definition + - arm64: Reserve an extra page for early kernel mapping + - futex: Drop CLONE_THREAD requirement for private default hash alloc + - PCI: Initialize temporary device in new_id_store() + - workqueue: fix devm_alloc_workqueue() va_list misuse + - net/sched: sch_pie: annotate more data-races in pie_dump_stats() + - sched/fair: Fix wakeup_preempt_fair() for not waking up task + - crypto: af_alg - Cap AEAD AD length to 0x80000000 + - i40e: Cleanup PTP pins on probe failure + - workqueue: Fix wq->cpu_pwq leak in alloc_and_link_pwqs() WQ_UNBOUND path + - net: ena: PHC: Fix potential use-after-free in get_timestamp + - cgroup/cpuset: Reset DL migration state on can_attach() failure + - netfilter: nf_conntrack_sip: get helper before allocating expectation + - audit: fix incorrect inheritable capability in CAPSET records + - net: ena: PHC: Check return code before setting timestamp output + - cgroup/dmem: Return -ENOMEM on failed pool preallocation + - idpf: fix double free and use-after-free in aux device error paths + - cgroup/cpuset: Reserve DL bandwidth only for root-domain moves + - Revert "ACPI: CPPC: Adjust debug messages in amd_set_max_freq_ratio() to + warn" + - netfilter: nft_ct: fix missing expect put in obj eval + - net: atlantic: preserve PCI wake-from-D3 on shutdown when WOL enabled + - audit: enforce AUDIT_LOCKED for AUDIT_TRIM and AUDIT_MAKE_EQUIV + - cgroup/cpuset: Return only actually allocated CPUs during partition + invalidation + - KVM: x86: Swap the dst and src operand for MOVNTDQA + - KVM: Reject wrapped offset in kvm_reset_dirty_gfn() + - KVM: s390: pci: fix GAIT table indexing due to double-scaling pointer + arithmetic + - KVM: x86: Fix Xen hypercall tracepoint argument assignment + - HID: pass the buffer size to hid_report_raw_event + - HID: core: introduce hid_safe_input_report() + - rseq: Revert to historical performance killing behaviour + - rseq: Implement read only ABI enforcement for optimized RSEQ V2 mode + - rseq: Reenable performance optimizations conditionally + - HID: core: Fix size_t specifier in hid_report_raw_event() + - ata: libata-scsi: fix requeue of deferred ATA PASS-THROUGH commands + - media: staging: imx: configure src_mux in csi_start + - Bluetooth: btmtk: accept too short WMT FUNC_CTRL events + - nvme-apple: Reset q->sq_tail during queue init + - smb/client: fix possible infinite loop and oob read in symlink_data() + - drm/loongson: Use managed KMS polling + - drm: Replace old pointer to new idr + - drm/bridge: imx8qxp-pxl2dpi: avoid ERR_PTR with device_node cleanup + - drm/i915/dp: Fix VSC dynamic range signaling for RGB formats + - drm/amd/display: Wrap DCN32 phantom-plane allocation in + DC_RUN_WITH_PREEMPTION_ENABLED + - drm/ttm: Fix ttm_bo_swapout() infinite LRU walk on swapout failure + - platform/x86: intel: Move debugfs register before creating devices + - platform/x86: lenovo-wmi-helpers: Move gamezone enums to wmi-helpers + - platform/x86: lenovo-wmi-helpers: Fix memory leak in + lwmi_dev_evaluate_int() + - platform/x86: lenovo-wmi-other: Balance IDA id allocation and free + - platform/x86: lenovo-wmi-other: Balance component bind and unbind + - platform/x86: lenovo-wmi-other: Zero initialize WMI arguments + - platform/x86: lenovo-wmi-other: Fix tunable_attr_01 struct members + - platform/x86: lenovo-wmi-other: Add Attribute ID helper functions + - platform/x86: lenovo-wmi-other: Limit adding attributes to supported + devices + - accel/rocket: Fix prep_bo ioctl leaking positive return from + dma_resv_wait_timeout() + - ALSA: hda/realtek: Add mute LED quirk for HP Pavilion Laptop 16-ag0xxx + - ALSA: hda/realtek: Add quirk for Samsung Galaxy Book5 360 headphone + - ALSA: usb-audio: Bound MIDI 2.0 endpoint descriptor scans + - ALSA: usb-audio: Bound MIDI endpoint descriptor scans + - ALSA: usb-audio: qcom: Check offload mapping failures + - btrfs: only release the dirty pages io tree after successful writes + - ceph: fix a buffer leak in __ceph_setxattr() + - ceph: fix BUG_ON in __ceph_build_xattrs_blob() due to stale blob size + - ceph: put folios not suitable for writeback + - io-wq: check that the predecessor is hashed in io_wq_remove_pending() + - iommu/amd: Bounds-check devid in __rlookup_amd_iommu() + - x86/kexec: Push kjump return address even for non-kjump kexec + - xfs: fix memory leak on error in xfs_alloc_zone_info() + - virt: sev-guest: Do not use host-controlled page order in cleanup path + - powerpc/warp: Fix error handling in pika_dtm_thread + - netfs: fix error handling in netfs_extract_user_iter() + - nfsd: fix GET_DIR_DELEGATION when VFS leases are disabled + - nfsd: fix file change detection in CB_GETATTR + - nfsd: update mtime/ctime on CLONE in presense of delegated attributes + - nfsd: update mtime/ctime on COPY in presence of delegated attributes + - irqchip/riscv-imsic: Clear interrupt move state during CPU offlining + - irqchip/meson-gpio: Use the correct register in + meson_s4_gpio_irq_set_type() + - irqchip/gic-v5: Move LPI allocation into the LPI domain + - irqchip/gic-v5: Support range allocation for LPIs + - irqchip/gic-v5: Allocate ITS parent LPIs as a range + - libceph: Fix potential out-of-bounds access in osdmap_decode() + - libceph: Fix potential null-ptr-deref in decode_choose_args() + - libceph: Fix potential out-of-bounds access in __ceph_x_decrypt() + - libceph: Fix potential out-of-bounds access in crush_decode() + - libceph: handle rbtree insertion error in decode_choose_args() + - iommu/vt-d: Disable DMAR for Intel Q35 IGFX + - iommu/vt-d: Fix oops due to out of scope access + - iommu/vt-d: Avoid NULL pointer dereference or refcount corruption + - iommu: Fix NULL group->domain dereference in pci_dev_reset_iommu_done() + - iommu: Replace per-group resetting_domain with per-gdev blocked flag + - iommu: Fix WARN_ON in __iommu_group_set_domain_nofail() due to reset + - iommu: Fix pasid attach in pci_dev_reset_iommu_prepare/done() + - iommu: Fix nested pci_dev_reset_iommu_prepare/done() + - iommu: Fix ATS invalidation timeouts during __iommu_remove_group_pasid() + - drm/i915: skip __i915_request_skip() for already signaled requests + - drm/panfrost: Fix wait_bo ioctl leaking positive return from + dma_resv_wait_timeout() + - drm/xe/dma-buf: handle empty bo and UAF races + - drm/xe/dma-buf: fix UAF with retry loop + - drm/ttm: Fix ttm_bo_shrink() infinite LRU walk on backup failure + - drm/ttm: Convert -EAGAIN from dmem_cgroup_try_charge to -ENOSPC + - drm/gma500/oaktrail_hdmi: fix i2c adapter leak on setup + - drm/gma500/oaktrail_lvds: fix hang on init failure + - drm/gma500/oaktrail_lvds: fix i2c adapter leaks on init + - arm_mpam: Fix monitor instance selection when checking for hardware NRDY + - arm_mpam: Pretend that NRDY is always hardware managed + - arm_mpam: Improve check for whether or not NRDY is hardware managed + - arm_mpam: Fix false positive assert failure during mpam_disable() + - arm_mpam: Check whether the config array is allocated before destroying + it + - eventfs: Simplify code using guard()s + - eventfs: Use list_add_tail_rcu() for SRCU-protected children list + - smb: client: Use FullSessionKey for AES-256 encryption key derivation + - spi: sifive: Simplify clock handling with devm_clk_get_enabled() + - spi: sifive: fix controller deregistration + - tracefs: Removed unused 'ret' variable in eventfs_iterate() + - workqueue: Annotate alloc_workqueue_va() with __printf(1, 0) + - selftests/bpf: Remove test_access_variable_array + - netfs: Fix potential uninitialised var in netfs_extract_user_iter() + - Linux 7.0.10 + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45846 + - bareudp: fix NULL pointer dereference in bareudp_fill_metadata_dst() + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45845 + - net/sched: taprio: fix NULL pointer dereference in class dump + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45844 + - netfilter: arp_tables: fix IEEE1394 ARP payload parsing + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-46242 + - eventpoll: fix ep_remove struct eventpoll / struct file UAF + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45843 + - slip: bound decode() reads against the compressed packet length + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45842 + - slip: reject VJ receive packets on instances with no rstate array + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45841 + - netfilter: nfnetlink_osf: fix divide-by-zero in OSF_WSS_MODULO + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45840 + - openvswitch: cap upcall PID array size and pre-size vport replies + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45839 + - bpf: reject negative CO-RE accessor indices in bpf_core_parse_spec() + * Resolute update: v7.0.10 upstream stable release (LP: #2156385) // + CVE-2026-45838 + - bpf: fix end-of-list detection in cgroup_storage_get_next_key() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) + - Linux 7.0.8 + - HID: pidff: Fix integer overflow in pidff_rescale + - media: uvcvideo: Enable VB2_DMABUF for metadata stream + - drm/msm/hdmi: Fix wrong CTRL1 register used in writing info frames + - media: rzv2h-ivc: Avoid double job scheduling + - media: nxp: imx8-isi: Reduce minimum queued buffers from 2 to 0 + - media: rzv2h-ivc: Write AXIRX_PIXFMT once + - media: rzv2h-ivc: Fix FM_STOP register write + - media: rzv2h-ivc: Fix concurrent buffer list access + - media: mali-c55: Initialize the ISP in enable_streams() + - media: mali-c55: Fix Iridix bypass macros + - media: renesas: vsp1: Fix NULL pointer deref on module unload + - media: renesas: vin: Fix RAW8 (again) + - media: i2c: ov8856: free control handler on error in + ov8856_init_controls() + - media: dt-bindings: rockchip,vdec: Add alternative reg-names order for + RK35{76,88} + - media: dt-bindings: rockchip,vdec: Mark reg-names required for + RK35{76,88} + - media: chips-media: wave5: fix a potential memory leak in + wave5_vdi_init() + - media: chips-media: wave5: add missing spinlock protection for + send_eos_event() + - media: chips-media: wave5: add missing spinlock protection for + handle_dynamic_resolution_change() + - arm64: dts: freescale: imx95-toradex-smarc: fix PMIC_SD2_VSEL label + position + - drm/gpusvm: Allow device pages to be mapped in mixed mappings after + system pages + - drm/gpusvm: Force unmapping on error in drm_gpusvm_get_pages + - spi: bcm63xx: fix controller deregistration + - spi: atmel: fix controller deregistration + - arm64: dts: lx2160a-cex7/lx2162a-sr-som: fix usd-cd & gpio pinmux + - regulator: mt6357: fix OF node reference imbalance + - spi: st-ssc4: fix controller deregistration + - regulator: max77650: fix OF node reference imbalance + - media: ti: vpe: Add missing v4l2_device_unregister in vip_remove() + - media: rc: streamzap: Error handling in probe + - media: i2c: imx283: Enter full standby when stopping streaming + - regulator: bq257xx: fix OF node reference imbalance + - regulator: rk808: fix OF node reference imbalance + - media: videobuf2: Set vma_flags in vb2_dma_sg_mmap + - media: mali-c55: Fully reset the ISP configuration + - media: intel/ipu6: fix error pointer dereference + - media: i2c: imx283: Fix hang when going from large to small resolution + - regulator: act8945a: fix OF node reference imbalance + - regulator: s2dos05: fix OF node reference imbalance + - regulator: bd9571mwv: fix OF node reference imbalance + - spi: lantiq-ssc: fix controller deregistration + - spi: meson-spicc: fix controller deregistration + - spi: qup: fix controller deregistration + - arm64: dts: ti: k3-am69-aquila-clover: Fix DP regulator enable GPIO + - spi: at91-usart: fix controller deregistration + - media: ipu-bridge: Add upside-down sensor DMI quirk for Dell XPS 13 9340 + and XPS 14 9440 + - spi: amlogic-spisg: fix controller deregistration + - spi: aspeed-smc: fix controller deregistration + - drm/colorop: Preserve bypass value in duplicate_state() + - drm/atomic: Add affected colorops with affected planes + - platform/x86: hp-wmi: Ignore backlight and FnLock events + - vsock/virtio: fix MSG_PEEK ignoring skb offset when calculating bytes to + copy + - arm64: dts: broadcom: bcm2712-d-rpi-5-b: add fixes for + pinctrl/pinctrl_aon + - arm64: dts: broadcom: bcm2712-d-rpi-5-b: update uart10 interrupt + - media: pci: zoran: fix potential memory leak in zoran_probe() + - media: dib8000: avoid division by 0 in dib8000_set_dds() + - media: i2c: imx412: Assert reset GPIO during probe + - media: staging: imx: request mbus_config in csi_start + - media: i2c: ov08d10: fix image vertical start setting + - media: i2c: ov08d10: fix runtime PM handling in probe + - media: omap3isp: drop the use count of v4l2 pipeline + - media: iris: fix QCOM_MDT_LOADER dependency + - media: qcom: camss: Fix csid clock configuration for sa8775p + - media: qcom: camss: Fix csid IRQ offset for sa8775p + - media: qcom: iris: increase H265D_MAX_SLICE to fix H.265 decoding on + SC7280 + - media: venus: fix QCOM_MDT_LOADER dependency + - media: iris: Fix dma_free_attrs() size in iris_hfi_queues_init() + - media: iris: switch to hardware mode after firmware boot + - media: qcom: camss: Add missing clocks for VFE lite on sa8775p + - spi: mxs: fix controller deregistration + - spi: mt65xx: fix controller deregistration + - spi: dln2: fix controller deregistration + - spi: s3c64xx: fix controller deregistration + - spi: fsl-espi: fix controller deregistration + - spi: omap2-mcspi: fix controller deregistration + - spi: pic32: fix controller deregistration + - spi: ep93xx: fix controller deregistration + - spi: mtk-nor: fix controller deregistration + - spi: pl022: fix controller deregistration + - spi: sh-hspi: fix controller deregistration + - spi: bcmbca-hsspi: fix controller deregistration + - spi: coldfire-qspi: fix controller deregistration + - spi: npcm-pspi: fix controller deregistration + - spi: cavium-thunderx: fix controller deregistration + - spi: pic32-sqi: fix controller deregistration + - spi: sprd: fix controller deregistration + - spi: sh-msiof: fix controller deregistration + - spi: slave-mt27xx: fix controller deregistration + - spi: img-spfi: fix controller deregistration + - spi: mpfs: fix controller deregistration + - spi: octeon: fix controller deregistration + - spi: imx: fix runtime pm leak on probe deferral + - spi: mxic: fix controller deregistration + - spi: orion: fix controller deregistration + - spi: orion: fix runtime pm leak on unbind + - spi: orion: fix clock imbalance on registration failure + - spi: cadence: fix controller deregistration + - spi: cadence-quadspi: fix controller deregistration + - spi: cadence: fix unclocked access on unbind + - spi: cadence: fix clock imbalance on probe failure + - spi: cadence-quadspi: fix runtime pm disable imbalance on probe failure + - spi: cadence-quadspi: fix clock imbalance on probe failure + - spi: cadence-quadspi: fix runtime pm and clock imbalance on unbind + - drm/colorop: Fix blob property reference tracking in state lifecycle + - drm/imx: parallel-display: Prefer bus format set via legacy "interface- + pix-fmt" DT property + - drm/msm: always recover the gpu + - drm/v3d: Reject empty multisync extension to prevent infinite loop + - drm/i915/psr: Init variable to avoid early exit from et alignment loop + - drm/amd/display: fix math_mod() using arg1 instead of arg2 + - drm/amd: Add missing firmware declaration for PSP v15.0.0 + - drm/amdgpu: Use NBIF offset for register RCC_STRAP0_RCC_DEV0_EPF0_STRAP0 + . + - drm/amdgpu: Use SMUIO 15.0.0 offsets for TSC upper and lower count. + - drm/amdgpu: gate VM CPU HDP flush on reset lock + - drm/amd/pm: fix incorrect FeatureCtrlMask setting on smu v14.0.x + - drm/amdkfd: Add upper bound check for num_of_nodes + - drm/amdgpu/vce: Prevent partial address patches + - drm/amd/display: Change dither policy for 10 bpc output back to + dithering + - drm/appletbdrm: Use kvzalloc for big allocations + - drm/amdgpu: Avoid reset in AMDGPU unload path for APUs with GFX V11 and + higher. + - drm/udl: Increase GET_URB_TIMEOUT + - drm/xe/bo: Fix bo leak on GGTT flag validation in xe_bo_init_locked() + - drm/xe/bo: Fix bo leak on unaligned size validation in + xe_bo_init_locked() + - drm/xe/uapi: Reject coh_none PAT index for CPU cached memory in madvise + - drm/radeon: add missing revision check for CI + - drm/amdgpu: zero-initialize GART table on allocation + - drm/exynos: remove bridge when component_add fails + - drm/amdgpu/userq: fix access to stale wptr mapping + - drm/panel: himax-hx83102: restore MODE_LPM after sending disable cmds + - drm/amdgpu/gfx9: drop unnecessary 64-bit fence flag check in KIQ + - drm/bridge: tda998x: Use __be32 for audio port OF property pointer + - drm/sti: remove bridge when sti_hda component_add fails + - drm/panel: boe-tv101wum-nl6: restore MODE_LPM after sending disable cmds + - drm/amdkfd: Make all TLB-flushes heavy-weight + - drm/amdgpu/pm: add missing revision check for CI + - drm/amdgpu/pm: align Hawaii mclk workaround with radeon + - arm64: dts: qcom: kodiak: Fix PCIe1 PHY ref clock voting + - arm64: dts: qcom: lemans: Correct QUP interrupt numbers + - arm64: dts: ti: k3-am62a7-sk: Fix pin name in comment from M19 to N22 + - arm64: dts: ti: k3-am69-aquila-dev: Fix DP regulator enable GPIO + - batman-adv: tp_meter: fix tp_num leak on kmalloc failure + - sched_ext: Use HK_TYPE_DOMAIN_BOOT to detect isolcpus= domain isolation + - usb: typec: tcpm: reset internal port states on soft reset AMS + - io_uring/zcrx: use guards for locking + - io_uring/zcrx: warn on freelist violations + - kho: fix error handling in kho_add_subtree() + - EDAC/versalnet: Refactor memory controller initialization and cleanup + - spi: uniphier: Simplify clock handling with devm_clk_get_enabled() + - spi: uniphier: fix controller deregistration + - cgroup: Increment nr_dying_subsys_* from rmdir context + - sched_ext: Skip tasks with stale task_rq in bypass_lb_cpu() + - perf build: fix "argument list too long" in second location + - mm/vma: do not try to unmap a VMA if mmap_prepare() invoked from mmap() + - vsock/virtio: fix length and offset in tap skb for split packets + - drm/amdgpu/vcn3: Avoid overflow on msg bound check + - drm/amdgpu/vcn4: Avoid overflow on msg bound check + - Linux 7.0.9 + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46214 + - vsock/virtio: fix accept queue count leak on transport mismatch + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46207 + - vsock/virtio: fix empty payload in tap skb for non-linear buffers + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46234 + - vsock: fix buffer size clamping order + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46223 + - cgroup: Defer css percpu_ref kill on rmdir until cgroup is depopulated + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46221 + - EDAC/versalnet: Fix device name memory leak + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46231 + - batman-adv: bla: put backbone reference on failed claim hash insert + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46233 + - batman-adv: bla: only purge non-released claims + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46212 + - batman-adv: bla: prevent use-after-free when deleting claims + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46238 + - batman-adv: stop caching unowned originator pointers in BAT IV + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46208 + - batman-adv: stop tp_meter sessions during mesh teardown + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46206 + - batman-adv: reject new tp_meter sessions during teardown + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46198 + - batman-adv: fix integer overflow on buff_pos + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46227 + - sctp: revalidate list cursor after sctp_sendmsg_to_asoc() in + SCTP_SENDALL + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46220 + - drm/amdgpu/sdma4: replace BUG_ON with WARN_ON in fence emission + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46215 + - drm: Set old handle to NULL before prime swap in change_handle + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46201 + - drm/xe: Fix dma-buf attachment leak in xe_gem_prime_import() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46224 + - drm/xe: Fix bo leak in xe_dma_buf_init_obj() on allocation failure + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46197 + - drm/amdkfd: validate SVM ioctl nattr against buffer size + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46209 + - drm/gem: Fix inconsistent plane dimension calculation in + drm_gem_fb_init_with_funcs() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46230 + - drm/amdgpu/vcn3: Prevent OOB reads when parsing dec msg + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46199 + - drm/amdgpu/vcn4: Prevent OOB reads when parsing dec msg + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46204 + - drm/amdgpu/vcn4: Prevent OOB reads when parsing IB + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46218 + - drm/amdgpu: Add bounds checking to ib_{get,set}_value + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46229 + - drm/amdkfd: Clear VRAM on allocation to prevent stale data exposure + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46211 + - drm/msm/gem: fix error handling in msm_ioctl_gem_info_get_metadata() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46203 + - spi: cadence-quadspi: fix unclocked access on unbind + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46219 + - spi: mpc52xx: fix use-after-free on unbind + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46200 + - spi: mpc52xx: fix controller deregistration + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46241 + - spi: mpc52xx: fix use-after-free on registration failure + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46225 + - spi: rspi: fix controller deregistration + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46226 + - spi: fsl: fix controller deregistration + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46228 + - spi: ch341: fix devres lifetime + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46216 + - drm/xe/hdcp: Add NULL check for media_gt in + intel_hdcp_gsc_check_status() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46210 + - media: iris: fix use-after-free of fmt_src during MBPF check + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46240 + - media: iris: Fix use-after-free in iris_release_internal_buffers() + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46235 + - media: saa7164: add ioremap return checks and cleanups + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46222 + - media: rockchip: rkcif: Add missing MUST_CONNECT flag to pads + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46239 + - media: i2c: ov5647: Fix runtime PM refcount leak in s_ctrl + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46236 + - media: rc: xbox_remote: heed DMA restrictions + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46205 + - staging: media: atomisp: Disallow all private IOCTLs + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46202 + - HID: appletb-kbd: run inactivity autodim from workqueues + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46213 + - HID: appletb-kbd: fix UAF in inactivity-timer cleanup path + * Resolute update: v7.0.9 upstream stable release (LP: #2156006) // + CVE-2026-46232 + - HID: playstation: Clamp num_touch_reports + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) + - ACPI: arm64: cpuidle: Tolerate platforms with no deep PSCI idle states + - ACPI: scan: Use acpi_dev_put() in object add error paths + - ACPI: video: Add backlight=native quirk for Dell OptiPlex 7770 AIO + - ACPI: CPPC: Fix related_cpus inconsistency during CPU hotplug + - ACPI: video: force native backlight on HP OMEN 16 (8A44) + - iommufd: Fix a race with concurrent allocation and unmap + - wifi: mt76: mt7925: fix incorrect TLV length in CLC command + - spi: rockchip: fix controller deregistration + - ksmbd: rewrite stop_sessions() with restartable iteration + - flow_dissector: do not dissect PPPoE PFC frames + - smb: client/smbdirect: fix MR registration for coalesced SG lists + - wifi: mt76: mt7925: fix AMPDU state handling in mt7925_tx_check_aggr + - wifi: mt76: mt7925: fix incorrect length field in txpower command + - wifi: mt76: mt7921: fix ROC abort flow interruption in mt7921_roc_work + - wifi: ath5k: do not access array OOB + - ALSA: usb-audio: midi2: Restart output URBs on resume + - ALSA: usb-audio: Fix UAC3 cluster descriptor size check + - usb: dwc3: Move GUID programming after PHY initialization + - USB: omap_udc: DMA: Don't enable burst 4 mode + - USB: serial: option: add Telit Cinterion LE910Cx compositions + - usb: typec: tcpm: fix debug accessory mode detection for sink ports + - ALSA: hda: cs35l56: Propagate ASP TX source control errors + - ALSA: hda/realtek: Fix speaker silence after S3 resume on Xiaomi Mi + Laptop Pro 15 + - ALSA: firewire-tascam: Do not drop unread control events + - ALSA: core: Serialize deferred fasync state checks + - ALSA: seq: Fix UMP group 16 filtering + - powerpc/kdump: fix KASAN sanitization flag for core_$(BITS).o + - x86/efi: Restore IRQ state in EFI page fault handler + - xfrm: provide message size for XFRM_MSG_MAPPING + - selinux: fix avdcache auditing + - selinux: don't reserve xattr slot when we won't fill it + - selinux: shrink critical section in sel_write_load() + - selinux: prune /sys/fs/selinux/checkreqprot + - selinux: prune /sys/fs/selinux/disable + - selinux: prune /sys/fs/selinux/user + - selinux: allow multiple opens of /sys/fs/selinux/policy + - io_uring/kbuf: support min length left for incremental buffers + - io_uring/tw: serialize ctx->retry_llist with ->uring_lock + - LoongArch: KVM: Fix missing EMULATE_FAIL in kvm_emu_mmio_read() + - rust: drm: gem: clean up GEM state in init failure case + - rust: allow `clippy::collapsible_match` globally + - rust: allow `clippy::collapsible_if` globally + - rust: pin-init: internal: move alignment check to `make_field_check` + - spi: syncuacer: fix controller deregistration + - spi: sun4i: fix controller deregistration + - spi: zynq-qspi: fix controller deregistration + - spi: ti-qspi: fix controller deregistration + - spi: sun6i: fix controller deregistration + - spi: tegra114: fix controller deregistration + - spi: zynqmp-gqspi: fix controller deregistration + - spi: tegra20-sflash: fix controller deregistration + - spi: s3c64xx: fix NULL-deref on driver unbind + - staging: rtl8723bs: os_dep: avoid NULL pointer dereference in + rtw_cbuf_alloc + - staging: vme_user: fix root device leak on init failure + - KVM: arm64: Fix kvm_vcpu_initialized() macro parameter + - arm64: signal: Preserve POR_EL0 if poe_context is missing + - mm/hugetlb_cma: round up per_node before logging it + - LoongArch: Fix SYM_SIGFUNC_START definition for 32BIT + - LoongArch: KVM: Compile switch.S directly into the kernel + - mptcp: pm: ADD_ADDR rtx: skip inactive subflows + - perf/x86/intel: Improve validation and configuration of ACR masks + - selftests/rseq: Don't run tests with runner scripts outside of the + scripts + - rseq: Set rseq::cpu_id_start to 0 on unregistration + - rseq: Protect rseq_reset() against interrupts + - rseq: Don't advertise time slice extensions if disabled + - selftests/rseq: Make registration flexible for legacy and optimized mode + - selftests/rseq: Skip tests if time slice extensions are not available + - selftests/rseq: Validate legacy behavior + - selftests/rseq: Expand for optimized RSEQ ABI v2 + - pseries/papr-hvpipe: Fix race with interrupt handler + - pseries/papr-hvpipe: Fix & simplify error handling in papr_hvpipe_init() + - pseries/papr-hvpipe: Fix the usage of copy_to_user() + - net: libwx: use request_irq for VF misc interrupt + - netpoll: pass buffer size to egress_dev() to avoid MAC truncation + - ovl: fix verity lazy-load guard broken by fsverity_active() semantic + change + - parisc: Fix IRQ leak in LASI driver + - x86/efi: Fix graceful fault handling after FPU softirq changes + - hwmon: (ltc2992) Clamp threshold writes to hardware range + - hwmon: (ltc2992) Fix u32 overflow in power read path + - clk: rk808: fix OF node reference imbalance + - hwmon: (corsair-psu) Close HID device on probe errors + - af_unix: Reject SIOCATMARK on non-stream sockets + - arm64/fpsimd: ptrace: zero target's fpsimd_state, not the tracer's + - pmdomain: mediatek: fix use-after-free in + scpsys_get_bus_protection_legacy() + - block: fix zone write plug removal + - block: only read from sqe on initial invocation of blkdev_uring_cmd() + - cifs: abort open_cached_dir if we don't request leases + - cifs: change_conf needs to be called for session setup + - extcon: ptn5150: handle pending IRQ events during system resume + - gpio: of: clear OF_POPULATED on hog nodes in remove path + - hv: Select CONFIG_SYSFB only for CONFIG_HYPERV_VMBUS + - hv_sock: fix ARM64 support + - hv_sock: Report EOF instead of -EIO for FIN + - hv_sock: Return -EIO for malformed/short packets + - spi: microchip-core-qspi: fix controller deregistration + - spi: microchip-core-spi: fix controller deregistration + - tracefs: Fix default permissions not being applied on initial mount + - udf: reject descriptors with oversized CRC length + - x86/boot/e820: Re-enable BIOS fallback if e820 table is empty + - thermal: core: Free thermal zone ID later during removal + - thermal/drivers/sprd: Fix temperature clamping in + sprd_thm_temp_to_rawdata + - thermal/drivers/sprd: Fix raw temperature clamping in + sprd_thm_rawdata_to_temp + - spi: topcliff-pch: fix controller deregistration + - spi: topcliff-pch: fix use-after-free on unbind + - tracing/fprobe: Avoid kcalloc() in rcu_read_lock section + - tracing/fprobe: Remove fprobe from hash in failure path + - tracing/fprobe: Unregister fprobe even if memory allocation fails + - tracing/probes: Limit size of event probe to 3K + - tracing/fprobe: Check the same type fprobe on table as the unregistered + one + - clk: imx: imx8-acm: fix flags for acm clocks + - clk: microchip: mpfs-ccc: fix out of bounds access during output + registration + - cpuidle: powerpc: avoid double clear when breaking snooze + - ASoC: amd: yc: Add HP OMEN Gaming Laptop 16-ap0xxx product line in quirk + table + - ASoC: ES8389: convert to devm_clk_get_optional() to get clock + - ASoC: fsl_easrc: fix comment typo + - ASoC: Intel: bytcr_wm5102: Fix MCLK leak on platform_clock_control error + - ASoC: qcom: q6apm-dai: reset queue ptr on trigger stop + - ASoC: qcom: q6apm: remove child devices when apm is removed + - btrfs: do not mark inode incompressible after inline attempt fails + - dm: don't report warning when doing deferred remove + - dm: fix a buffer overflow in ioctl processing + - dm-verity-fec: correctly reject too-small FEC devices + - dm-verity-fec: correctly reject too-small hash devices + - dm-verity-fec: fix corrected block count stat + - dm-verity-fec: fix the size of dm_verity_fec_io::erasures + - isofs: validate Rock Ridge CE continuation extent against volume size + - iommufd: Fix return value of iommufd_fault_fops_write() + - iommu/vt-d: Block PASID attachment to nested domain with dirty tracking + - iommu/arm-smmu-v3: Add a missing dma_wmb() for hitless STE update + - lib/crc: tests: Make crc_kunit test only the enabled CRC variants + - lib/scatterlist: fix temp buffer in extract_user_to_sg() + - nvme-apple: drop invalid put of admin queue reference count + - nvmet: avoid recursive nvmet-wq flush in nvmet_ctrl_free + - pmdomain: core: Fix detach procedure for virtual devices in genpd + - psp: strip variable-length PSP header in psp_dev_rcv() + - s390/debug: Reject zero-length input in debug_input_flush_fn() + - s390/debug: Reject zero-length input before trimming a newline + - KVM: x86: Do IRR scan in __kvm_apic_update_irr even if PIR is empty + - mm/damon/lru_sort: detect and use fresh enabled and kdamond_pid values + - mm/damon/reclaim: detect and use fresh enabled and kdamond_pid values + - mm/damon/stat: detect and use fresh enabled value + - PCI: Update saved_config_space upon resource assignment + - PCI/AER: Clear only error bits in PCIe Device Status + - PCI/AER: Stop ruling out unbound devices as error source + - PCI/ASPM: Fix pci_clear_and_set_config_dword() usage + - power: supply: max17042: avoid overflow when determining health + - perf/x86/intel: Always reprogram ACR events to prevent stale masks + - perf/x86/intel: Disable PMI for self-reloaded ACR events + - perf/x86/intel: Enable auto counter reload for DMR + - RDMA/ionic: bound node_desc sysfs read with %.64s + - RDMA/ionic: Fix typo in format string + - remoteproc: imx_rproc: Fix NULL vs IS_ERR() bug in imx_rproc_addr_init() + - remoteproc: k3: Fix NULL vs IS_ERR() bug in k3_reserved_mem_init() + - sched_ext: idle: Recheck prev_cpu after narrowing allowed mask + - sched_ext: Use dsq->first_task instead of list_empty() in + dispatch_enqueue() FIFO-tail + - selftests: mptcp: check output: catch cmd errors + - selftests: mptcp: pm: restrict 'unknown' check to pm_nl_ctl + - mptcp: fastclose msk when linger time is 0 + - mptcp: use MPJoinSynAckHMacFailure for SynAck HMAC failure + - mptcp: use MPTCP_RST_EMPTCP for ACK HMAC validation failure + - mptcp: sockopt: set timestamp flags on subflow socket, not msk + - mptcp: sockopt: increase seq in mptcp_setsockopt_all_sf + - mptcp: fix rx timestamp corruption on fastopen + - mptcp: pm: prio: skip closed subflows + - mptcp: pm: kernel: reset fullmesh counter after flush + - mptcp: pm: kernel: correctly retransmit ADD_ADDR ID 0 + - mptcp: pm: ADD_ADDR rtx: resched blocked ADD_ADDR quicker + - mptcp: pm: ADD_ADDR rtx: return early if no retrans + - f2fs: add READ_ONCE() for i_blocks in f2fs_update_inode() + - f2fs: fix false alarm of lockdep on cp_global_sem lock + - f2fs: fix fiemap boundary handling when read extent cache is incomplete + - f2fs: fix fsck inconsistency caused by incorrect nat_entry flag usage + - f2fs: fix incorrect file address mapping when inline inode is unwritten + - f2fs: fix incorrect multidevice info in trace_f2fs_map_blocks() + - f2fs: fix uninitialized kobject put in f2fs_init_sysfs() + - f2fs: refactor f2fs_move_node_folio function + - f2fs: fix inline data not being written to disk in writeback path + - KVM: arm64: Wake-up from WFI when iqrchip is in userspace + - KVM: arm64: vgic: Fix IIDR revision field extracted from wrong value + - KVM: arm64: Fix initialisation order in __pkvm_init_finalise() + - KVM: arm64: Fix FEAT_SPE_FnE to use PMSIDR_EL1.FnE, not PMSVer + - KVM: arm64: Fix FEAT_Debugv8p9 to check DebugVer, not PMUVer + - LoongArch: KVM: Cap KVM_CAP_NR_VCPUS by KVM_CAP_MAX_VCPUS + - LoongArch: KVM: Fix "unreliable stack" for kvm_exc_entry + - LoongArch: KVM: Fix HW timer interrupt lost when inject interrupt by + software + - LoongArch: KVM: Move unconditional delay into timer clear scenery + - LoongArch: KVM: Use kvm_set_pte() in kvm_flush_pte() + - LoongArch: Use per-root-bridge PCIH flag to skip mem resource fixup + - mmc: core: Adjust MDT beyond 2025 + - mmc: core: Add quirk for incorrect manufacturing date + - mmc: core: Optimize time for secure erase/trim for some Kingston eMMCs + - crypto: qat - fix indentation of macros in qat_hal.c + - crypto: qat - fix firmware loading failure for GEN6 devices + - hfsplus: fix held lock freed on hfsplus_fill_super() + - 8021q: use RCU for egress QoS mappings + - printk: add print_hex_dump_devel() + - crypto: caam - guard HMAC key hex dumps in hash_digest_key + - net: stmmac: rename STMMAC_GET_ENTRY() -> STMMAC_NEXT_ENTRY() + - rust: pin-init: fix incorrect accessor reference lifetime + - Linux 7.0.7 + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43490 + - ksmbd: validate inherited ACE SID length + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46174 + - x86/CPU/AMD: Prevent improper isolation of shared resources in Zen2's op + cache + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46110 + - net: stmmac: Prevent NULL deref when RX memory exhausted + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46153 + - 8021q: delete cleared egress QoS mappings + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46169 + - hfsplus: fix uninit-value by validating catalog record size + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46188 + - octeon_ep_vf: add NULL check for napi_build_skb() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-45837 + - bpf: Fix use-after-free in arena_vm_close on fork + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46156 + - LoongArch: Fix potential ADE in loongson_gpu_fixup_dma_hang() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46147 + - KVM: arm64: Fix pin leak and publication ordering in __pkvm_init_vcpu() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46175 + - f2fs: fix fsck inconsistency caused by FGGC of node block + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46194 + - f2fs: fix node_cnt race between extent node destroy and writeback + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46170 + - mptcp: pm: ADD_ADDR rtx: free sk if last + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46158 + - mptcp: pm: ADD_ADDR rtx: always decrease sk refcount + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46168 + - mptcp: fix scheduling with atomic in timestamp sockopt + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46189 + - RDMA/vmw_pvrdma: Fix double free on pvrdma_alloc_ucontext() error path + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46133 + - RDMA/rxe: Reject unknown opcodes before ICRC processing + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46114 + - RDMA/rxe: Reject non-8-byte ATOMIC_WRITE payloads + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46127 + - RDMA/ocrdma: Don't NULL deref uctx on errors in ocrdma_copy_pd_uresp() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46176 + - RDMA/mlx5: Fix error path fall-through in mlx5_ib_dev_res_srq_init() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46178 + - RDMA/mlx4: Fix resource leak on error in mlx4_ib_create_srq() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46181 + - RDMA/mlx4: Fix mis-use of RCU in mlx4_srq_event() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46145 + - RDMA/mana: Validate rx_hash_key_len + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46117 + - RDMA/mana: Remove user triggerable WARN_ON() in mana_ib_create_qp_rss() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46126 + - RDMA/mana: Fix mana_destroy_wq_obj() cleanup in mana_ib_create_qp_rss() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46144 + - RDMA/mana: Fix error unwind in mana_ib_create_qp_rss() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46141 + - powerpc/xive: fix kmemleak caused by incorrect chip_data lookup + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46183 + - mm/damon/sysfs-schemes: protect path kfree() with damon_sysfs_lock + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46121 + - mm/damon/sysfs-schemes: protect memcg_path kfree() with damon_sysfs_lock + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46131 + - KVM: x86: check for nEPT/nNPT in slow flush hypercalls + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46139 + - smb: client: use kzalloc to zero-initialize security descriptor buffer + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46105 + - scsi: mpt3sas: Limit NVMe request size to 2 MiB + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46171 + - riscv: kvm: fix vector context allocation leak + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46112 + - RDMA/hns: Fix unlocked call to hns_roce_qp_remove() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46165 + - openvswitch: vport: fix self-deadlock on release of tunnel ports + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46161 + - md/raid10: fix divide-by-zero in setup_geo() with zero far_copies + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43492 + - lib/crypto: mpi: Fix integer underflow in mpi_read_raw_from_sgl() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46124 + - isofs: validate block number from NFS file handle in isofs_export_iget + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46130 + - dm-verity-fec: fix reading parity bytes split across blocks (take 3) + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46106 + - eventfs: Hold eventfs_mutex and SRCU when remount walks events + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46107 + - dm-thin: fix metadata refcount underflow + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46160 + - btrfs: fix missing last_unlink_trans update when removing a directory + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46164 + - btrfs: fix double free in create_space_info_sub_group() error path + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46129 + - btrfs: fix double free in create_space_info() error path + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46159 + - btrfs: fix btrfs_ioctl_space_info() slot_count TOCTOU which can lead to + info-leak + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46143 + - ASoC: qcom: q6apm-lpass-dai: Fix multiple graph opens + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46148 + - spi: microchip-core-qspi: control built-in cs manually + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46192 + - spi: microchip-core-qspi: don't attempt to transmit during emulated + read-only dual/quad operations + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46162 + - ice: fix double free in ice_sf_eth_activate() error path + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46273 + - ibmveth: Disable GSO for packets with small MSS + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46191 + - fbcon: Avoid OOB font access if console rotation fails + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46134 + - platform/chrome: cros_ec_typec: Init mutex in Thunderbolt registration + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43495 + - net: wwan: t7xx: validate port_count against message length in + t7xx_port_enum_msg_handler + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43502 + - net/rds: handle zerocopy send cleanup before the message is queued + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46120 + - ip6_gre: Use cached t->net in ip6erspan_changelink(). + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46142 + - net: libwx: fix VF illegal register access + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46118 + - pseries/papr-hvpipe: Fix null ptr deref in + papr_hvpipe_dev_create_handle() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46182 + - pseries/papr-hvpipe: Prevent kernel stack memory leak to userspace + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46184 + - sound: ua101: fix division by zero at probe + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43498 + - accel/ivpu: Disallow re-exporting imported GEM objects + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46132 + - net: rtnetlink: zero ifla_vf_broadcast to avoid stack infoleak in + rtnl_fill_vfinfo + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46190 + - mtd: spi-nor: debugfs: fix out-of-bounds read in spi_nor_params_show() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46150 + - fanotify: fix false positive on permission events + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-45836 + - Bluetooth: L2CAP: Fix null-ptr-deref in l2cap_sock_get_sndtimeo_cb() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-45834 + - Bluetooth: L2CAP: Fix null-ptr-deref in l2cap_sock_state_change_cb() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-45835 + - Bluetooth: L2CAP: Fix null-ptr-deref in l2cap_sock_new_connection_cb() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46138 + - Bluetooth: hci_event: Fix OOB read and infinite loop in + hci_le_create_big_complete_evt + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46111 + - Bluetooth: hci_conn: fix potential UAF in create_big_sync + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46140 + - Bluetooth: btmtk: validate WMT event SKB length before struct access + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46186 + - Bluetooth: virtio_bt: validate rx pkt_type header length + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46123 + - Bluetooth: virtio_bt: clamp rx length before skb_put + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46104 + - selinux: use sk blob accessor in socket permission helpers + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46193 + - xfrm: ah: account for ESN high bits in async callbacks + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46172 + - ipv6: xfrm6: release dst on error in xfrm6_rcv_encap() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46116 + - xfrm: defensively unhash xfrm_state lists in __xfrm_state_delete + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46154 + - sched_ext: Read scx_root under scx_cgroup_ops_rwsem in cgroup setters + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46157 + - ALSA: pcm: oss: Fix data race at accessing runtime.oss.trigger + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46109 + - usb: ulpi: fix memory leak on ulpi_register() error paths + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46146 + - ALSA: usb-audio: Avoid potential endless loop in convert_chmap_v3() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46167 + - usb: usblp: fix uninitialized heap leak via LPGETSTATUS ioctl + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46151 + - usb: usblp: fix heap leak in IEEE 1284 device ID via short response + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46180 + - wifi: brcmfmac: Fix potential use-after-free issue when stopping + watchdog task + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46122 + - wifi: b43: enforce bounds check on firmware key index in b43_rx() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46125 + - wifi: mac80211: remove station if connection prep fails + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46166 + - wifi: mac80211: use safe list iteration in radar detect work + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46187 + - wifi: rsi: fix kthread lifetime race between self-exit and external-stop + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46152 + - wifi: mac80211: drop stray 'static' from fast-RX rx_result + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46163 + - wifi: b43legacy: enforce bounds check on firmware key index in RX path + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46136 + - wifi: mt76: mt7921: fix a potential clc buffer length underflow + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46173 + - exit: prevent preemption of oopsing TASK_DEAD task + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43496 + - net/sched: sch_red: Replace direct dequeue call with peek and + qdisc_dequeue_peeked + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46113 + - KVM: x86: Fix shadow paging use-after-free due to unexpected GFN + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46179 + - ASoC: SOF: Don't allow pointer operations on unconfigured streams + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46196 + - tracepoint: balance regfunc() on func_add() failure in + tracepoint_add_func() + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-43497 + - fbdev: udlfb: add vm_ops to dlfb_ops_mmap to prevent use-after-free + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46108 + - ipmi:si: Return state to normal if message allocation fails + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46128 + - ipmi: Check event message buffer response for bad data + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46177 + - ipmi: Add limits to event and receive message requests + * Resolute update: v7.0.7 upstream stable release (LP: #2155988) // + CVE-2026-46149 + - scsi: target: configfs: Bound snprintf() return in + tg_pt_gp_members_show() + * Ubuntu 26.04 linux kernel has non-functional nova-core GPU driver enabled, + conflicting with nouveau (LP: #2150845) + - [Config] Disable NOVA_CORE + * CVE-2026-46244 + - netfilter: nft_inner: Fix IPv6 inner_thoff desync + * CVE-2026-46137 + - mptcp: pm: ADD_ADDR rtx: allow ID 0 + - mptcp: pm: ADD_ADDR rtx: fix potential data-race + * CVE-2026-46185 + - smb/client: fix out-of-bounds read in symlink_data() + * CVE-2026-46195 + - smb: client: validate dacloffset before building DACL pointers + * CVE-2026-46289 + - lib/scatterlist: fix length calculations in extract_kvec_to_sg + * CVE-2026-46119 + - libceph: Fix slab-out-of-bounds access in auth message processing + * CVE-2026-46135 + - nvmet-tcp: fix race between ICReq handling and queue teardown + * CVE-2026-46155 + - smb/client: fix out-of-bounds read in smb2_compound_op() + * CVE-2026-46115 + - block: add pgmap check to biovec_phys_mergeable + * CVE-2026-46243 + - smb: client: reject userspace cifs.spnego descriptions + + -- Jacob Martin Wed, 22 Jul 2026 10:02:25 -0500 linux-nvidia-bos (7.0.0-2015.15) resolute; urgency=medium diff --git a/debian.nvidia-bos/reconstruct b/debian.nvidia-bos/reconstruct index d3d1c7130e908..39263f0076c35 100644 --- a/debian.nvidia-bos/reconstruct +++ b/debian.nvidia-bos/reconstruct @@ -41,8 +41,14 @@ chmod +x 'drivers/net/ethernet/realtek/r8127/rtl_eeprom.c' chmod +x 'drivers/net/ethernet/realtek/r8127/rtl_eeprom.h' chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.c' chmod +x 'drivers/net/ethernet/realtek/r8127/rtltool.h' +chmod +x 'tools/testing/selftests/rseq/run_legacy_check.sh' +chmod +x 'tools/testing/selftests/rseq/run_timeslice_test.sh' # Remove any files deleted from the orig. +rm -f 'Documentation/ABI/obsolete/sysfs-selinux-user' rm -f 'Documentation/admin-guide/perf/nvidia-pmu.rst' rm -f 'arch/parisc/lib/checksum.c' +rm -f 'drivers/platform/x86/lenovo/wmi-gamezone.h' +rm -f 'tools/testing/selftests/bpf/prog_tests/access_variable_array.c' +rm -f 'tools/testing/selftests/bpf/progs/test_access_variable_array.c' rm -f 'tools/testing/selftests/mqueue/setting' exit 0 From 386a76e8a826de14032bedb9504a514206d89dc2 Mon Sep 17 00:00:00 2001 From: Manish Honap Date: Thu, 9 Jul 2026 16:05:23 +0530 Subject: [PATCH 458/464] NVIDIA: VR: SAUCE: vfio/cxl: advertise DPA mmap and update its selftest BugLink: https://bugs.launchpad.net/bugs/2160591 The DPA region needs VFIO_REGION_INFO_FLAG_MMAP so a VMM can map it into stage-2 for the device to reach over ATS. Without the flag the region stays a slow-path I/O region that the stage-2 listener skips, and the GPU's ATS access to DPA-resident memory never resolves. This is the same mmap capability the upstream series and nvgrace-gpu advertise for coherent device memory. Whether a host CPU coherent access to DPA succeeds is a platform property and is not gated by this flag. Advertising the flag lets any fd holder mmap DPA and touch it from the host CPU, so a memory error on that struct-page-less range would otherwise escalate to a host SError. Register the range with memory_failure() via register_pfn_address_space(), as nvgrace-gpu does, so such an error is contained to unmapping the range and a SIGBUS to the fd holder. The mmap flag stays required for the ATS path regardless; containment does not replace it. The dpa_mmap_fault selftest is skipped when the flag is absent. With the flag set it runs, but its raw host CPU load/store of DPA does not model how a VMM uses the region: the VMM mmaps DPA and maps it into the IOAS, and the guest reaches it through stage-2. Rewrite the test to do the same (mmap plus iommu_map), drop the host dereference, and fail rather than skip when the mmap flag is missing so a future regression is caught. Since the test now performs a different operation renamed the test to dpa_mmap_ioas_map matching what it verifies. Fixes: e658757fac34 ("NVIDIA: VR: SAUCE: vfio/cxl: Register regions with VFIO layer") Signed-off-by: Manish Honap Acked-by: Nirmoy Das Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/vfio/pci/cxl/vfio_cxl_core.c | 73 +++++++++++++++++-- drivers/vfio/pci/cxl/vfio_cxl_priv.h | 2 + .../selftests/vfio/vfio_cxl_type2_test.c | 43 ++++++----- 3 files changed, 92 insertions(+), 26 deletions(-) diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c b/drivers/vfio/pci/cxl/vfio_cxl_core.c index 2d6b804d8537b..31bad3ad0b066 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c @@ -1034,6 +1034,7 @@ static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev, cxl->region_size, true); if (cxl->region_vaddr) { + unregister_pfn_address_space(&cxl->dpa_pfn_space); memunmap(cxl->region_vaddr); cxl->region_vaddr = NULL; } @@ -1045,6 +1046,60 @@ static const struct vfio_pci_regops vfio_cxl_regops = { .release = vfio_cxl_region_release, }; +/* + * Map a poisoned DPA pfn back to the file offset of every user mapping so + * memory_failure() can unmap it and signal the fd holder. The DPA is a + * single linear range at region_hpa; recover the per-vma file offset the + * same way the fault handler derived the pfn. + */ +static int vfio_cxl_pfn_to_vma_pgoff(struct vm_area_struct *vma, + unsigned long pfn, pgoff_t *pgoff) +{ + struct vfio_pci_region *region; + struct vfio_pci_cxl_state *cxl; + pgoff_t vma_off, pfn_off; + unsigned long start_pfn; + + if (vma->vm_ops != &vfio_cxl_region_vm_ops) + return -ENOENT; + + region = vma->vm_private_data; + cxl = region->data; + + start_pfn = PHYS_PFN(cxl->region_hpa); + if (pfn < start_pfn || pfn >= start_pfn + (cxl->region_size >> PAGE_SHIFT)) + return -EFAULT; + + pfn_off = pfn - start_pfn; + vma_off = vma->vm_pgoff & + ((1UL << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + /* Skip VMAs that do not map the pfn, e.g. a partial mmap of DPA */ + if (pfn_off < vma_off || pfn_off - vma_off >= vma_pages(vma)) + return -EFAULT; + + *pgoff = vma->vm_pgoff + (pfn_off - vma_off); + return 0; +} + +/* + * DPA is struct-page-less device memory, so a memory error on it cannot be + * routed through the normal page path. Register the range with + * memory_failure() to contain such errors (unmap + SIGBUS to the fd holder) + * instead of letting them escalate to a host SError. + */ +static int vfio_cxl_register_pfn_space(struct vfio_pci_cxl_state *cxl) +{ + unsigned long start_pfn = PHYS_PFN(cxl->region_hpa); + + cxl->dpa_pfn_space.node.start = start_pfn; + cxl->dpa_pfn_space.node.last = + start_pfn + (cxl->region_size >> PAGE_SHIFT) - 1; + cxl->dpa_pfn_space.mapping = cxl->vdev->vdev.inode->i_mapping; + cxl->dpa_pfn_space.pfn_to_vma_pgoff = vfio_cxl_pfn_to_vma_pgoff; + + return register_pfn_address_space(&cxl->dpa_pfn_space); +} + int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) { struct vfio_pci_cxl_state *cxl = vdev->cxl; @@ -1067,13 +1122,18 @@ int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) if (!cxl->region_vaddr) return -ENOMEM; - /* - * BOS/backport policy: do not advertise DPA mmap until the CXL DPA - * backing is proven safe for userspace CPU mappings. Keep fd - * read/write available via the memremap() kernel mapping. - */ + /* -EOPNOTSUPP means CONFIG_MEMORY_FAILURE is off; run without it */ + ret = vfio_cxl_register_pfn_space(cxl); + if (ret && ret != -EOPNOTSUPP) { + memunmap(cxl->region_vaddr); + cxl->region_vaddr = NULL; + return ret; + } + + /* DPA is mmappable coherent device memory */ flags = VFIO_REGION_INFO_FLAG_READ | - VFIO_REGION_INFO_FLAG_WRITE; + VFIO_REGION_INFO_FLAG_WRITE | + VFIO_REGION_INFO_FLAG_MMAP; ret = vfio_pci_core_register_dev_region(vdev, PCI_VENDOR_ID_CXL | @@ -1083,6 +1143,7 @@ int vfio_cxl_register_cxl_region(struct vfio_pci_core_device *vdev) cxl->region_size, flags, cxl); if (ret) { + unregister_pfn_address_space(&cxl->dpa_pfn_space); memunmap(cxl->region_vaddr); cxl->region_vaddr = NULL; return ret; diff --git a/drivers/vfio/pci/cxl/vfio_cxl_priv.h b/drivers/vfio/pci/cxl/vfio_cxl_priv.h index ac8ea3893c7af..430ddd116deb4 100644 --- a/drivers/vfio/pci/cxl/vfio_cxl_priv.h +++ b/drivers/vfio/pci/cxl/vfio_cxl_priv.h @@ -10,6 +10,7 @@ #include #include +#include #include struct vfio_pci_core_device; @@ -31,6 +32,7 @@ struct vfio_pci_cxl_state { resource_size_t region_hpa; size_t region_size; void *region_vaddr; + struct pfn_address_space dpa_pfn_space; resource_size_t hdm_reg_offset; size_t hdm_reg_size; resource_size_t comp_reg_offset; diff --git a/tools/testing/selftests/vfio/vfio_cxl_type2_test.c b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c index 8d932250864f4..ec39b7e503d4c 100644 --- a/tools/testing/selftests/vfio/vfio_cxl_type2_test.c +++ b/tools/testing/selftests/vfio/vfio_cxl_type2_test.c @@ -627,27 +627,29 @@ TEST_F(cxl_type2, comp_regs_region_info) /* ------------------------------------------------------------------ */ /* - * mmap() the DPA region and verify the first page can be read. - * The region uses lazy fault insertion so the first access triggers the - * vfio_cxl_region_page_fault path. + * Mirror the VMM: mmap the DPA region and map it into the IOAS (stage-2). + * The DPA mmap is required for the device's ATS path, so a missing mmap + * flag is a regression, not a reason to skip. */ -TEST_F(cxl_type2, dpa_mmap_fault) +TEST_F(cxl_type2, dpa_mmap_ioas_map) { struct vfio_region_info reg = { .argsz = sizeof(reg) }; + struct iova_allocator *iova_alloc; + struct dma_region region; size_t map_size; void *ptr; - uint8_t *p; - uint8_t val; reg.index = self->cxl_cap.dpa_region_index; ASSERT_EQ(0, ioctl(self->dev->fd, VFIO_DEVICE_GET_REGION_INFO, ®)); - if (!(reg.flags & VFIO_REGION_INFO_FLAG_MMAP)) - SKIP(return, "DPA region does not advertise mmap"); + ASSERT_NE(0, reg.flags & VFIO_REGION_INFO_FLAG_MMAP); - /* Map just the first 2MB or the full region, whichever is smaller */ - map_size = (size_t)reg.size < (size_t)(2 * SZ_1M) - ? (size_t)reg.size : (size_t)(2 * SZ_1M); + /* Do not touch DPA from the host CPU; the guest reaches it via stage-2 */ + if (reg.size < SZ_2M) + SKIP(return, "DPA region smaller than 2M"); + + /* iova_allocator_alloc() requires a power-of-2 size */ + map_size = SZ_2M; ptr = mmap(NULL, map_size, PROT_READ | PROT_WRITE, MAP_SHARED, self->dev->fd, (off_t)reg.offset); @@ -656,17 +658,18 @@ TEST_F(cxl_type2, dpa_mmap_fault) self->dpa_mmap = ptr; self->dpa_mmap_size = map_size; - /* First access - triggers vmf_insert_pfn */ - p = (uint8_t *)ptr; - val = *p; - (void)val; + iova_alloc = iova_allocator_init(self->iommu); + region.vaddr = ptr; + region.size = map_size; + region.iova = iova_allocator_alloc(iova_alloc, map_size); + + iommu_map(self->iommu, ®ion); - printf("DPA mmap: ptr=%p size=0x%zx first byte=0x%02x\n", - ptr, map_size, (uint8_t)val); + printf("DPA mmap + IOAS map ok: iova=0x%llx size=0x%zx\n", + (unsigned long long)region.iova, map_size); - /* Write a pattern and read it back */ - *p = 0xab; - ASSERT_EQ(0xab, *p); + iommu_unmap(self->iommu, ®ion); + iova_allocator_cleanup(iova_alloc); } /* From c507e79dfaf65959cde121d2a8b2fff02f651286 Mon Sep 17 00:00:00 2001 From: John Hubbard Date: Wed, 27 May 2026 15:55:06 -0700 Subject: [PATCH 459/464] perf/core: out-of-line and export perf_allow_cpu/tracepoint() BugLink: https://bugs.launchpad.net/bugs/2160654 These helpers are static inline in and reach into sysctl_perf_event_paranoid and security_perf_event_open(), neither of which is itself exported. The perf_allow_* trio is therefore asymmetric: built-in callers can use any of the three, but modular code can only call perf_allow_kernel(). Move both bodies into kernel/events/core.c next to perf_allow_kernel() and export them with EXPORT_SYMBOL_GPL, following the shape of commit 5e9629d0ae97 ("drivers/perf: arm_spe: Use perf_allow_kernel() for permissions"). Existing in-tree callers live in built-in arch and tracing code, so the change is invisible to them. Provide !CONFIG_PERF_EVENTS stubs that fall back to perfmon_capable(), so the helpers stay callable when perf is compiled out. Signed-off-by: John Hubbard Reviewed-by: Ashutosh Dixit Link: https://patch.msgid.link/20260527225507.2044027-2-ashutosh.dixit@intel.com Signed-off-by: Ashutosh Dixit (cherry picked from commit d32bf877c0c3ebc345b444cbe009b3f44f9f8073 linux-next) Signed-off-by: Kelsey Steele Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- include/linux/perf_event.h | 31 +++++++++++++++---------------- kernel/events/core.c | 18 ++++++++++++++++++ 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 6cdc23fb7e093..f929a7dd324ec 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -1797,22 +1797,8 @@ static inline int perf_is_paranoid(void) } extern int perf_allow_kernel(void); - -static inline int perf_allow_cpu(void) -{ - if (sysctl_perf_event_paranoid > 0 && !perfmon_capable()) - return -EACCES; - - return security_perf_event_open(PERF_SECURITY_CPU); -} - -static inline int perf_allow_tracepoint(void) -{ - if (sysctl_perf_event_paranoid > -1 && !perfmon_capable()) - return -EPERM; - - return security_perf_event_open(PERF_SECURITY_TRACEPOINT); -} +extern int perf_allow_cpu(void); +extern int perf_allow_tracepoint(void); extern int perf_exclude_event(struct perf_event *event, struct pt_regs *regs); @@ -2029,6 +2015,19 @@ perf_event_pause(struct perf_event *event, bool reset) { return 0; } static inline int perf_exclude_event(struct perf_event *event, struct pt_regs *regs) { return 0; } +static inline int perf_allow_kernel(void) +{ + return perfmon_capable() ? 0 : -EACCES; +} +static inline int perf_allow_cpu(void) +{ + return perfmon_capable() ? 0 : -EACCES; +} +static inline int perf_allow_tracepoint(void) +{ + return perfmon_capable() ? 0 : -EPERM; +} + #endif /* !CONFIG_PERF_EVENTS */ #if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL) diff --git a/kernel/events/core.c b/kernel/events/core.c index b46f849d726dc..eaad93d4493f5 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -14700,6 +14700,24 @@ int perf_allow_kernel(void) } EXPORT_SYMBOL_GPL(perf_allow_kernel); +int perf_allow_cpu(void) +{ + if (sysctl_perf_event_paranoid > 0 && !perfmon_capable()) + return -EACCES; + + return security_perf_event_open(PERF_SECURITY_CPU); +} +EXPORT_SYMBOL_GPL(perf_allow_cpu); + +int perf_allow_tracepoint(void) +{ + if (sysctl_perf_event_paranoid > -1 && !perfmon_capable()) + return -EPERM; + + return security_perf_event_open(PERF_SECURITY_TRACEPOINT); +} +EXPORT_SYMBOL_GPL(perf_allow_tracepoint); + /* * Inherit an event from parent task to child task. * From 57dea08fe397598671770af53afef06b5a483824 Mon Sep 17 00:00:00 2001 From: John Hubbard Date: Wed, 27 May 2026 15:55:07 -0700 Subject: [PATCH 460/464] drm/xe: gate observation streams with perf_allow_cpu() BugLink: https://bugs.launchpad.net/bugs/2160654 xe OA and EU-stall paths open-code a partial copy of the system-wide perf CPU-event permission check: if (xe_observation_paranoid && !perfmon_capable()) return -EACCES; This open-coded check skips two things perf_allow_cpu() handles: the graduated kernel.perf_event_paranoid policy that an administrator may have tuned, and the security_perf_event_open() LSM hook. Introduce xe_observation_paranoid_check() to wrap perf_allow_cpu(), and convert the open-coded sites in xe_oa.c and xe_eu_stall.c. The dev.xe.observation_paranoid sysctl still acts as an escape hatch when cleared. xe observation now consults kernel.perf_event_paranoid and the LSM perf hook on every open. Sites that have already configured an LSM perf policy or tuned the paranoid sysctl will see those settings extend to xe. Signed-off-by: John Hubbard Reviewed-by: Ashutosh Dixit Link: https://patch.msgid.link/20260527225507.2044027-3-ashutosh.dixit@intel.com Signed-off-by: Ashutosh Dixit (backported from commit 6680bf0cb7261b7eb62a7226c6845c5c9ce5a009 linux-next) [kelseys: Drop the new int ret declaration and initialize the existing ret variable used by remap_pfn_range() instead. This tree does not contain commit 41255b2f1e03 ("drm/xe/oa: Use drm_gem_mmap_obj for OA buffer mmap"), which removed the legacy mapping loop and its ret variable.] Signed-off-by: Kelsey Steele Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- drivers/gpu/drm/xe/xe_eu_stall.c | 5 +++-- drivers/gpu/drm/xe/xe_oa.c | 25 +++++++++++++--------- drivers/gpu/drm/xe/xe_observation.c | 32 ++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_observation.h | 3 +-- 4 files changed, 46 insertions(+), 19 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_eu_stall.c b/drivers/gpu/drm/xe/xe_eu_stall.c index 7da14854f688e..3d3490511c37e 100644 --- a/drivers/gpu/drm/xe/xe_eu_stall.c +++ b/drivers/gpu/drm/xe/xe_eu_stall.c @@ -963,9 +963,10 @@ int xe_eu_stall_stream_open(struct drm_device *dev, u64 data, struct drm_file *f return -ENODEV; } - if (xe_observation_paranoid && !perfmon_capable()) { + ret = xe_observation_paranoid_check(); + if (ret) { drm_dbg(&xe->drm, "Insufficient privileges for EU stall monitoring\n"); - return -EACCES; + return ret; } /* Initialize and set default values */ diff --git a/drivers/gpu/drm/xe/xe_oa.c b/drivers/gpu/drm/xe/xe_oa.c index 449a431ec1d4e..6f69ded6dc911 100644 --- a/drivers/gpu/drm/xe/xe_oa.c +++ b/drivers/gpu/drm/xe/xe_oa.c @@ -1676,9 +1676,10 @@ static int xe_oa_mmap(struct file *file, struct vm_area_struct *vma) unsigned long start = vma->vm_start; int i, ret; - if (xe_observation_paranoid && !perfmon_capable()) { + ret = xe_observation_paranoid_check(); + if (ret) { drm_dbg(&stream->oa->xe->drm, "Insufficient privilege to map OA buffer\n"); - return -EACCES; + return ret; } /* Can mmap the entire OA buffer or nothing (no partial OA buffer mmaps) */ @@ -2070,10 +2071,12 @@ int xe_oa_stream_open_ioctl(struct drm_device *dev, u64 data, struct drm_file *f privileged_op = true; } - if (privileged_op && xe_observation_paranoid && !perfmon_capable()) { - drm_dbg(&oa->xe->drm, "Insufficient privileges to open xe OA stream\n"); - ret = -EACCES; - goto err_exec_q; + if (privileged_op) { + ret = xe_observation_paranoid_check(); + if (ret) { + drm_dbg(&oa->xe->drm, "Insufficient privileges to open xe OA stream\n"); + goto err_exec_q; + } } if (!param.exec_q && !param.sample) { @@ -2352,9 +2355,10 @@ int xe_oa_add_config_ioctl(struct drm_device *dev, u64 data, struct drm_file *fi return -ENODEV; } - if (xe_observation_paranoid && !perfmon_capable()) { + err = xe_observation_paranoid_check(); + if (err) { drm_dbg(&oa->xe->drm, "Insufficient privileges to add xe OA config\n"); - return -EACCES; + return err; } err = copy_from_user(¶m, u64_to_user_ptr(data), sizeof(param)); @@ -2454,9 +2458,10 @@ int xe_oa_remove_config_ioctl(struct drm_device *dev, u64 data, struct drm_file return -ENODEV; } - if (xe_observation_paranoid && !perfmon_capable()) { + ret = xe_observation_paranoid_check(); + if (ret) { drm_dbg(&oa->xe->drm, "Insufficient privileges to remove xe OA config\n"); - return -EACCES; + return ret; } ret = get_user(arg, ptr); diff --git a/drivers/gpu/drm/xe/xe_observation.c b/drivers/gpu/drm/xe/xe_observation.c index e3f9b546207e4..39e05b9131a74 100644 --- a/drivers/gpu/drm/xe/xe_observation.c +++ b/drivers/gpu/drm/xe/xe_observation.c @@ -4,6 +4,7 @@ */ #include +#include #include #include @@ -12,9 +13,28 @@ #include "xe_oa.h" #include "xe_observation.h" -u32 xe_observation_paranoid = true; +static u32 xe_observation_paranoid = true; static struct ctl_table_header *sysctl_header; +/** + * xe_observation_paranoid_check - Gate access to xe observation streams. + * + * When the xe-specific observation_paranoid sysctl is enabled (the + * default), defer to perf_allow_cpu() so that access is governed by the + * same policy as system-wide perf CPU events: kernel.perf_event_paranoid + * plus the security_perf_event_open() LSM hook. When the sysctl has been + * cleared by a privileged user, observation is open to all callers. + * + * Return: 0 if access is permitted, a negative errno otherwise. + */ +int xe_observation_paranoid_check(void) +{ + if (!xe_observation_paranoid) + return 0; + + return perf_allow_cpu(); +} + static int xe_oa_ioctl(struct drm_device *dev, struct drm_xe_observation_param *arg, struct drm_file *file) { @@ -83,11 +103,13 @@ static const struct ctl_table observation_ctl_table[] = { }; /** - * xe_observation_sysctl_register - Register xe_observation_paranoid sysctl + * xe_observation_sysctl_register - Register the observation_paranoid sysctl * - * Normally only superuser/root can access observation stream - * data. However, superuser can set xe_observation_paranoid sysctl to 0 to - * allow non-privileged users to also access observation data. + * When dev.xe.observation_paranoid is set (the default), access to + * observation streams follows the system-wide perf_allow_cpu() policy: + * kernel.perf_event_paranoid plus the security_perf_event_open() LSM + * hook. A privileged user can clear the sysctl to bypass that gate and + * allow unprivileged access to observation data. * * Return: always returns 0 */ diff --git a/drivers/gpu/drm/xe/xe_observation.h b/drivers/gpu/drm/xe/xe_observation.h index 17816998e9666..73a03e03c96a7 100644 --- a/drivers/gpu/drm/xe/xe_observation.h +++ b/drivers/gpu/drm/xe/xe_observation.h @@ -11,8 +11,7 @@ struct drm_device; struct drm_file; -extern u32 xe_observation_paranoid; - +int xe_observation_paranoid_check(void); int xe_observation_ioctl(struct drm_device *dev, void *data, struct drm_file *file); int xe_observation_sysctl_register(void); void xe_observation_sysctl_unregister(void); From b02b764a5e06fc0ceff05d770a1720a99b80fe33 Mon Sep 17 00:00:00 2001 From: Cosmin Ratiu Date: Thu, 7 May 2026 10:56:04 +0300 Subject: [PATCH 461/464] ipv4: Provide a FIB flushing signal from nexthop removal functions BugLink: https://bugs.launchpad.net/bugs/2158449 Plumb a bool value throughout the various nexthop removal functions, determined in the innermost __remove_nexthop_fib() (which still does the FIB flushing) and propagated up all callers. The next patch will make use of this signal to optimize the removal of multiple nexthops by moving the FIB flushing up the call hierarchy. Signed-off-by: Cosmin Ratiu Reviewed-by: Ido Schimmel Reviewed-by: David Ahern Link: https://patch.msgid.link/20260507075606.322405-2-cratiu@nvidia.com Signed-off-by: Jakub Kicinski (cherry picked from commit 31c777be2a2efd8980a660724955ba795ef751de) Signed-off-by: Benjamin Poirier Acked-by: Omer Barak Acked-by: Aya Levin Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- net/ipv4/nexthop.c | 50 +++++++++++++++++++++++++++------------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c index 11a763cbc8482..994859ee8eb16 100644 --- a/net/ipv4/nexthop.c +++ b/net/ipv4/nexthop.c @@ -20,7 +20,7 @@ #define NH_RES_DEFAULT_IDLE_TIMER (120 * HZ) #define NH_RES_DEFAULT_UNBALANCED_TIMER 0 /* No forced rebalancing. */ -static void remove_nexthop(struct net *net, struct nexthop *nh, +static bool remove_nexthop(struct net *net, struct nexthop *nh, struct nl_info *nlinfo); #define NH_DEV_HASHBITS 8 @@ -2016,7 +2016,7 @@ static void nh_hthr_group_rebalance(struct nh_group *nhg) } } -static void remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, +static bool remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, struct nl_info *nlinfo, struct list_head *deferred_free) { @@ -2033,10 +2033,8 @@ static void remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, newg = nhg->spare; /* last entry, keep it visible and remove the parent */ - if (nhg->num_nh == 1) { - remove_nexthop(net, nhp, nlinfo); - return; - } + if (nhg->num_nh == 1) + return remove_nexthop(net, nhp, nlinfo); newg->has_v4 = false; newg->is_multipath = nhg->is_multipath; @@ -2093,22 +2091,26 @@ static void remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, if (nlinfo) nexthop_notify(RTM_NEWNEXTHOP, nhp, nlinfo); + + return false; } -static void remove_nexthop_from_groups(struct net *net, struct nexthop *nh, +static bool remove_nexthop_from_groups(struct net *net, struct nexthop *nh, struct nl_info *nlinfo) { struct nh_grp_entry *nhge, *tmp; LIST_HEAD(deferred_free); + bool need_flush = false; /* If there is nothing to do, let's avoid the costly call to * synchronize_net() */ if (list_empty(&nh->grp_list)) - return; + return false; list_for_each_entry_safe(nhge, tmp, &nh->grp_list, nh_list) - remove_nh_grp_entry(net, nhge, nlinfo, &deferred_free); + need_flush |= remove_nh_grp_entry(net, nhge, nlinfo, + &deferred_free); /* make sure all see the newly published array before releasing rtnl */ synchronize_net(); @@ -2118,6 +2120,8 @@ static void remove_nexthop_from_groups(struct net *net, struct nexthop *nh, list_del(&nhge->nh_list); free_percpu(nhge->stats); } + + return need_flush; } static void remove_nexthop_group(struct nexthop *nh, struct nl_info *nlinfo) @@ -2142,17 +2146,15 @@ static void remove_nexthop_group(struct nexthop *nh, struct nl_info *nlinfo) } /* not called for nexthop replace */ -static void __remove_nexthop_fib(struct net *net, struct nexthop *nh) +static bool __remove_nexthop_fib(struct net *net, struct nexthop *nh) { + bool need_flush = !list_empty(&nh->fi_list); struct fib6_info *f6i; - bool do_flush = false; struct fib_info *fi; - list_for_each_entry(fi, &nh->fi_list, nh_list) { + list_for_each_entry(fi, &nh->fi_list, nh_list) fi->fib_flags |= RTNH_F_DEAD; - do_flush = true; - } - if (do_flush) + if (need_flush) fib_flush(net); spin_lock_bh(&nh->lock); @@ -2173,12 +2175,14 @@ static void __remove_nexthop_fib(struct net *net, struct nexthop *nh) } spin_unlock_bh(&nh->lock); + + return need_flush; } -static void __remove_nexthop(struct net *net, struct nexthop *nh, +static bool __remove_nexthop(struct net *net, struct nexthop *nh, struct nl_info *nlinfo) { - __remove_nexthop_fib(net, nh); + bool need_flush = __remove_nexthop_fib(net, nh); if (nh->is_group) { remove_nexthop_group(nh, nlinfo); @@ -2189,13 +2193,17 @@ static void __remove_nexthop(struct net *net, struct nexthop *nh, if (nhi->fib_nhc.nhc_dev) hlist_del(&nhi->dev_hash); - remove_nexthop_from_groups(net, nh, nlinfo); + need_flush |= remove_nexthop_from_groups(net, nh, nlinfo); } + + return need_flush; } -static void remove_nexthop(struct net *net, struct nexthop *nh, +static bool remove_nexthop(struct net *net, struct nexthop *nh, struct nl_info *nlinfo) { + bool need_flush; + call_nexthop_notifiers(net, NEXTHOP_EVENT_DEL, nh, NULL); /* remove from the tree */ @@ -2204,10 +2212,12 @@ static void remove_nexthop(struct net *net, struct nexthop *nh, if (nlinfo) nexthop_notify(RTM_DELNEXTHOP, nh, nlinfo); - __remove_nexthop(net, nh, nlinfo); + need_flush = __remove_nexthop(net, nh, nlinfo); nh_base_seq_inc(net); nexthop_put(nh); + + return need_flush; } /* if any FIB entries reference this nexthop, any dst entries From 4aefaa7f1e6a764cc9f7307f4b620139a7da1e70 Mon Sep 17 00:00:00 2001 From: Cosmin Ratiu Date: Thu, 7 May 2026 10:56:05 +0300 Subject: [PATCH 462/464] ipv4: Flush the FIB once on multiple nexthop removal BugLink: https://bugs.launchpad.net/bugs/2158449 When a device is going down or when a net namespace is deleted, all nexthops on it are removed, and for each nexthop being removed the FIB table is flushed, which does a full trie traversal looking for entries marked RTNH_F_DEAD and removing them. This is O(N x R), with N being number of dev nexthops and R being number of IPv4 routes. The RTNL is held the entire time. When there are many nexthops to be removed and many routing entries, this can result in the RTNL being held for multiple minutes, which causes unhappiness in other processes trying to acquire the RTNL (e.g. systemd-networkd for DHCP renewals). In a complicated deployment with multiple vxlan devices, each having 16K nexthops and a total of 128K ipv4 routes, this is exactly what happens: nexthop_flush_dev() # loops over 16K nexthops -> remove_nexthop() -> __remove_nexthop() -> __remove_nexthop_fib() # marks fi->fib_flags |= RTNH_F_DEAD -> fib_flush() # for EACH nexthop! -> fib_table_flush() # walks the ENTIRE FIB, 128K entries This patch makes use of the previously added FIB flushing signal to only do a single FIB flush after all nexthops to be removed are marked as RTNH_F_DEAD: - __remove_nexthop_fib() no longer flushes the FIB. - nexthop_flush_dev() and flush_all_nexthops() now keep track whether any nexthop was removed and trigger a FIB flush at the end. - a new wrapper is defined, remove_one_nexthop() which calls remove_nexthop() and flushes if necessary. This is intended for places which must remove a single nexthop and shouldn't worry about the need to trigger a FIB flush. For now, the only caller is rtm_del_nexthop(). - The two direct callers of __remove_nexthop() get a WARN_ON_ONCE, since the nh about to be removed should not have any FIB entries referencing it when replacing or inserting a new one. This dramatically improves performance from O(N x R) to O(N + R). Releasing a nexthop reference in remove_nexthop() now no longer frees it. Instead, it is deleted when the last fib_info pointing to it gets freed via free_fib_info_rcu(). All routing code is already careful not to take into consideration routes marked with RTNH_F_DEAD. Tested with: DEV=eth2 ip link set up dev $DEV ip link add testnh0 link $DEV type macvlan mode bridge ip addr add 198.51.100.1/24 dev testnh0 ip link set testnh0 up seq 1 65536 | \ sed 's/.*/nexthop add id & via 198.51.100.2 dev testnh0/' | \ ip -batch - i=1 for a in $(seq 0 255); do for b in $(seq 0 255); do echo "route add 10.${a}.${b}.0/32 nhid $i" i=$((i + 1)) done done | ip -batch - time ip link set testnh0 down ip link del testnh0 Without this patch: real 0m32.601s user 0m0.000s sys 0m32.511s With this patch: real 0m0.209s user 0m0.000s sys 0m0.153s Signed-off-by: Cosmin Ratiu Reviewed-by: Ido Schimmel Reviewed-by: David Ahern Link: https://patch.msgid.link/20260507075606.322405-3-cratiu@nvidia.com Signed-off-by: Jakub Kicinski (cherry picked from commit 35ce55100c61270eb8234bcc8ac87fec1d8e4ff9) Signed-off-by: Benjamin Poirier Acked-by: Omer Barak Acked-by: Aya Levin Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- net/ipv4/nexthop.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c index 994859ee8eb16..4766b4f9cee1e 100644 --- a/net/ipv4/nexthop.c +++ b/net/ipv4/nexthop.c @@ -2154,8 +2154,6 @@ static bool __remove_nexthop_fib(struct net *net, struct nexthop *nh) list_for_each_entry(fi, &nh->fi_list, nh_list) fi->fib_flags |= RTNH_F_DEAD; - if (need_flush) - fib_flush(net); spin_lock_bh(&nh->lock); @@ -2220,6 +2218,13 @@ static bool remove_nexthop(struct net *net, struct nexthop *nh, return need_flush; } +static void remove_one_nexthop(struct net *net, struct nexthop *nh, + struct nl_info *nlinfo) +{ + if (remove_nexthop(net, nh, nlinfo)) + fib_flush(net); +} + /* if any FIB entries reference this nexthop, any dst entries * need to be regenerated */ @@ -2599,7 +2604,7 @@ static int replace_nexthop(struct net *net, struct nexthop *old, if (!err) { nh_rt_cache_flush(net, old, new); - __remove_nexthop(net, new, NULL); + WARN_ON_ONCE(__remove_nexthop(net, new, NULL)); nexthop_put(new); } @@ -2706,6 +2711,7 @@ static void nexthop_flush_dev(struct net_device *dev, unsigned long event) unsigned int hash = nh_dev_hashfn(dev->ifindex); struct net *net = dev_net(dev); struct hlist_head *head = &net->nexthop.devhash[hash]; + bool need_flush = false; struct hlist_node *n; struct nh_info *nhi; @@ -2717,22 +2723,28 @@ static void nexthop_flush_dev(struct net_device *dev, unsigned long event) (event == NETDEV_DOWN || event == NETDEV_CHANGE)) continue; - remove_nexthop(net, nhi->nh_parent, NULL); + need_flush |= remove_nexthop(net, nhi->nh_parent, NULL); } + + if (need_flush) + fib_flush(net); } /* rtnl; called when net namespace is deleted */ static void flush_all_nexthops(struct net *net) { struct rb_root *root = &net->nexthop.rb_root; + bool need_flush = false; struct rb_node *node; struct nexthop *nh; while ((node = rb_first(root))) { nh = rb_entry(node, struct nexthop, rb_node); - remove_nexthop(net, nh, NULL); + need_flush |= remove_nexthop(net, nh, NULL); cond_resched(); } + if (need_flush) + fib_flush(net); } static struct nexthop *nexthop_create_group(struct net *net, @@ -3002,7 +3014,7 @@ static struct nexthop *nexthop_add(struct net *net, struct nh_config *cfg, err = insert_nexthop(net, nh, cfg, extack); if (err) { - __remove_nexthop(net, nh, NULL); + WARN_ON_ONCE(__remove_nexthop(net, nh, NULL)); nexthop_put(nh); nh = ERR_PTR(err); } @@ -3371,7 +3383,7 @@ static int rtm_del_nexthop(struct sk_buff *skb, struct nlmsghdr *nlh, nh = nexthop_find_by_id(net, id); if (nh) - remove_nexthop(net, nh, &nlinfo); + remove_one_nexthop(net, nh, &nlinfo); else err = -ENOENT; From 265816a43353c9ee37a41f7a1342f44b38fdb92f Mon Sep 17 00:00:00 2001 From: Cosmin Ratiu Date: Thu, 7 May 2026 10:56:06 +0300 Subject: [PATCH 463/464] ipv4: Add __must_check to nexthop removal functions BugLink: https://bugs.launchpad.net/bugs/2158449 These functions return a signal whether FIB flushing is required which must not be ignored. Use the compiler to help with enforcing this requirement in the future. Signed-off-by: Cosmin Ratiu Reviewed-by: Ido Schimmel Reviewed-by: David Ahern Link: https://patch.msgid.link/20260507075606.322405-4-cratiu@nvidia.com Signed-off-by: Jakub Kicinski (cherry picked from commit 5dcbd64e66ba36fc7abd433d9bbba660dc0c473d) Signed-off-by: Benjamin Poirier Acked-by: Omer Barak Acked-by: Aya Levin Acked-by: Jamie Nguyen Acked-by: Carol L Soto Signed-off-by: Brad Figg --- net/ipv4/nexthop.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c index 4766b4f9cee1e..d28aa3f7ec37b 100644 --- a/net/ipv4/nexthop.c +++ b/net/ipv4/nexthop.c @@ -20,8 +20,8 @@ #define NH_RES_DEFAULT_IDLE_TIMER (120 * HZ) #define NH_RES_DEFAULT_UNBALANCED_TIMER 0 /* No forced rebalancing. */ -static bool remove_nexthop(struct net *net, struct nexthop *nh, - struct nl_info *nlinfo); +static bool __must_check remove_nexthop(struct net *net, struct nexthop *nh, + struct nl_info *nlinfo); #define NH_DEV_HASHBITS 8 #define NH_DEV_HASHSIZE (1U << NH_DEV_HASHBITS) @@ -2016,9 +2016,9 @@ static void nh_hthr_group_rebalance(struct nh_group *nhg) } } -static bool remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, - struct nl_info *nlinfo, - struct list_head *deferred_free) +static bool __must_check +remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, + struct nl_info *nlinfo, struct list_head *deferred_free) { struct nh_grp_entry *nhges, *new_nhges; struct nexthop *nhp = nhge->nh_parent; @@ -2095,8 +2095,9 @@ static bool remove_nh_grp_entry(struct net *net, struct nh_grp_entry *nhge, return false; } -static bool remove_nexthop_from_groups(struct net *net, struct nexthop *nh, - struct nl_info *nlinfo) +static bool __must_check +remove_nexthop_from_groups(struct net *net, struct nexthop *nh, + struct nl_info *nlinfo) { struct nh_grp_entry *nhge, *tmp; LIST_HEAD(deferred_free); @@ -2146,7 +2147,8 @@ static void remove_nexthop_group(struct nexthop *nh, struct nl_info *nlinfo) } /* not called for nexthop replace */ -static bool __remove_nexthop_fib(struct net *net, struct nexthop *nh) +static bool __must_check __remove_nexthop_fib(struct net *net, + struct nexthop *nh) { bool need_flush = !list_empty(&nh->fi_list); struct fib6_info *f6i; @@ -2177,8 +2179,8 @@ static bool __remove_nexthop_fib(struct net *net, struct nexthop *nh) return need_flush; } -static bool __remove_nexthop(struct net *net, struct nexthop *nh, - struct nl_info *nlinfo) +static bool __must_check __remove_nexthop(struct net *net, struct nexthop *nh, + struct nl_info *nlinfo) { bool need_flush = __remove_nexthop_fib(net, nh); @@ -2197,8 +2199,8 @@ static bool __remove_nexthop(struct net *net, struct nexthop *nh, return need_flush; } -static bool remove_nexthop(struct net *net, struct nexthop *nh, - struct nl_info *nlinfo) +static bool __must_check remove_nexthop(struct net *net, struct nexthop *nh, + struct nl_info *nlinfo) { bool need_flush; From 08eb251b260a737cf65e7d258d68b66542c96762 Mon Sep 17 00:00:00 2001 From: Koba Ko Date: Wed, 29 Jul 2026 01:20:54 +0800 Subject: [PATCH 464/464] NVIDIA: VR: SAUCE: firmware: tegra: bpmp: grow debugfs read buffer BPMP firmware debugfs listings can exceed the fixed 16 KiB buffer. After CMD_DEBUG_OPEN reports the listing length, the host-side size check returns -EFBIG and prevents the firmware debugfs trees from being populated. Allocate the listing buffer using the firmware-reported length and populate debugfs in one pass. Validate every read chunk for forward progress, response-payload bounds, and remaining-length bounds before copying or writing it. Bound peak firmware data retained along a recursive in-band population path to 512 KiB, matching the existing MRQ_DEBUGFS whole-tree buffer size. Reject listing lengths that exceed the remaining path budget before allocation. Fixes: 5e37b9c137ee ("firmware: tegra: Add support for in-band debug") Signed-off-by: Koba Ko --- drivers/firmware/tegra/bpmp-debugfs.c | 97 ++++++++++++++++++--------- 1 file changed, 66 insertions(+), 31 deletions(-) diff --git a/drivers/firmware/tegra/bpmp-debugfs.c b/drivers/firmware/tegra/bpmp-debugfs.c index 9898aa9396c56..421a6ed5becfc 100644 --- a/drivers/firmware/tegra/bpmp-debugfs.c +++ b/drivers/firmware/tegra/bpmp-debugfs.c @@ -11,6 +11,12 @@ #include #include +/* + * Bound BPMP debugfs firmware data retained while mirroring directory trees. + * The same limit also sizes the legacy MRQ_DEBUGFS whole-tree dump buffer. + */ +#define BPMP_DEBUGFS_FIRMWARE_DATA_MAX SZ_512K + static DEFINE_MUTEX(bpmp_debug_lock); struct seqbuf { @@ -175,8 +181,19 @@ static int mrq_debug_close(struct tegra_bpmp *bpmp, u32 fd) return 0; } +static int bpmp_debug_validate_read_chunk(const char *caller, u32 readlen, + u32 remaining) +{ + if (!readlen || readlen > DEBUG_READ_MAX_SZ || readlen > remaining) { + pr_err("%s: read data length invalid\n", caller); + return -EINVAL; + } + + return 0; +} + static int mrq_debug_read(struct tegra_bpmp *bpmp, const char *name, - char *data, size_t sz_data, u32 *nbytes) + size_t remaining_budget, void **datap, u32 *nbytes) { struct mrq_debug_request req = { .cmd = CMD_DEBUG_READ, @@ -194,20 +211,34 @@ static int mrq_debug_read(struct tegra_bpmp *bpmp, const char *name, }, }; u32 fd = 0, len = 0; - int remaining, err, close_err; + char *data = NULL, *ptr; + u32 remaining; + int err, close_err; + + *datap = NULL; + *nbytes = 0; mutex_lock(&bpmp_debug_lock); err = mrq_debug_open(bpmp, name, &fd, &len, 0); if (err) goto out; - if (len > sz_data) { + if (len > remaining_budget) { err = -EFBIG; goto close; } + if (len) { + data = kvmalloc(len, GFP_KERNEL); + if (!data) { + err = -ENOMEM; + goto close; + } + } + req.frd.fd = fd; remaining = len; + ptr = data; while (remaining > 0) { err = tegra_bpmp_transfer(bpmp, &msg); @@ -218,23 +249,27 @@ static int mrq_debug_read(struct tegra_bpmp *bpmp, const char *name, goto close; } - if (resp.frd.readlen > remaining) { - pr_err("%s: read data length invalid\n", __func__); - err = -EINVAL; + err = bpmp_debug_validate_read_chunk(__func__, + resp.frd.readlen, + remaining); + if (err) goto close; - } - memcpy(data, resp.frd.data, resp.frd.readlen); - data += resp.frd.readlen; + memcpy(ptr, resp.frd.data, resp.frd.readlen); + ptr += resp.frd.readlen; remaining -= resp.frd.readlen; } - *nbytes = len; - close: close_err = mrq_debug_close(bpmp, fd); if (!err) err = close_err; + if (err) { + kvfree(data); + } else { + *datap = data; + *nbytes = len; + } out: mutex_unlock(&bpmp_debug_lock); return err; @@ -322,7 +357,8 @@ static int bpmp_debug_show(struct seq_file *m, void *p) }, }; u32 fd = 0, len = 0; - int remaining, err, close_err; + u32 remaining; + int err, close_err; filename = get_filename(bpmp, file, fnamebuf, sizeof(fnamebuf)); if (!filename) @@ -345,11 +381,11 @@ static int bpmp_debug_show(struct seq_file *m, void *p) goto close; } - if (resp.frd.readlen > remaining) { - pr_err("%s: read data length invalid\n", __func__); - err = -EINVAL; + err = bpmp_debug_validate_read_chunk(__func__, + resp.frd.readlen, + remaining); + if (err) goto close; - } seq_write(m, resp.frd.data, resp.frd.readlen); remaining -= resp.frd.readlen; @@ -403,34 +439,31 @@ static const struct file_operations bpmp_debug_fops = { static int bpmp_populate_debugfs_inband(struct tegra_bpmp *bpmp, struct dentry *parent, - char *ppath) + char *ppath, + size_t remaining_budget) { const size_t pathlen = SZ_256; - const size_t bufsize = SZ_16K; + size_t child_budget; struct dentry *dentry; u32 dsize, attrs = 0; struct seqbuf seqbuf; - char *buf, *pathbuf; + void *buf = NULL; + char *pathbuf; const char *name; int err = 0; if (!bpmp || !parent || !ppath) return -EINVAL; - buf = kmalloc(bufsize, GFP_KERNEL); - if (!buf) - return -ENOMEM; - pathbuf = kzalloc(pathlen, GFP_KERNEL); - if (!pathbuf) { - kfree(buf); + if (!pathbuf) return -ENOMEM; - } - err = mrq_debug_read(bpmp, ppath, buf, bufsize, &dsize); + err = mrq_debug_read(bpmp, ppath, remaining_budget, &buf, &dsize); if (err) goto out; + child_budget = remaining_budget - dsize; seqbuf_init(&seqbuf, buf, dsize); while (!seqbuf_eof(&seqbuf)) { @@ -458,7 +491,8 @@ static int bpmp_populate_debugfs_inband(struct tegra_bpmp *bpmp, } err = bpmp_populate_debugfs_inband(bpmp, dentry, - pathbuf); + pathbuf, + child_budget); if (err < 0) goto out; } else { @@ -477,7 +511,7 @@ static int bpmp_populate_debugfs_inband(struct tegra_bpmp *bpmp, out: kfree(pathbuf); - kfree(buf); + kvfree(buf); return err; } @@ -743,7 +777,7 @@ static int bpmp_populate_dir(struct tegra_bpmp *bpmp, struct seqbuf *seqbuf, static int bpmp_populate_debugfs_shmem(struct tegra_bpmp *bpmp) { struct seqbuf seqbuf; - const size_t sz = SZ_512K; + const size_t sz = BPMP_DEBUGFS_FIRMWARE_DATA_MAX; dma_addr_t phys; size_t nbytes; void *virt; @@ -805,7 +839,8 @@ int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp) if (inband) err = bpmp_populate_debugfs_inband(bpmp, bpmp->debugfs_mirror, - "/"); + "/", + BPMP_DEBUGFS_FIRMWARE_DATA_MAX); else err = bpmp_populate_debugfs_shmem(bpmp);