Where is the spec?

Without looking at the spec, just your patch:

1. Same page may be mapped both into KVA and UVA. So this effectively means that all pages except initial kernel allocations for text and data, must be encrypted. Also, all pages are mapped in DMAP, which means that you need to set your pg_sem_c for DMAP pages as needed.
2. Kernel often translates pte->phys address->vm page or pte->phys address->DMAP VA. You need to inspect all that places and ensure that your pg_sem_c bit is not used as part of the physical address. Look at the PG_PS_FRAME and PG_FRAME constants which often used to extract phys address from pte-like structures (we arguably need to make them more dynamic to support more bits of physical address).

hi kib, thanks for taking a look!
i'll try to figure out the dmap bits, thanks for the pointer.

spec is in https://www.amd.com/system/files/TechDocs/24593.pdf, page 275.

I think i've wrapped my head around dmap & friends... it makes things more complicated.
Will have to demote dmap mappings. If i try to encrypt user-mode pages first, i'll need to demote the dmap pages down from 1gb into 4k and set the c-bit for the small pages.
Same for trying to encrypt all pages. Will still have to demote dmap pages because of dma and the existing kernel text and preloaded module pages.

Does that sound about right?

I'll continue with user mode pages only for a v0 proof of concept. Because then I dont have to worry about dma and device drivers.

As the final solution this is not going to work, since you basically would downgrade whole DMAP.
On the other hand, for initial attempt, you could disable superpages at all, which should simplify things and give you a more straightforward playground.

For DMA, I think you would need to bounce a lot. I assume encryption is not done from IO side, which should partially mitigate unlimited access from devices?
Does AMD IOMMU able to handle encryption for IO transactions? (But we do not have AMD IOMMU driver proper)

I'm still working on this but it's really slow going rn, got very little time.

I changed my approach to this. What we really need to know is whether a physical "page" has been encrypted or not. And from that follow the right bits in the PTE etc. And not the other way around, which I attempted here first.
So I'm introducing a PGA_ENCRYPTED for vm_page and will set this bit during alloc, say when the page(s) is allocated as managed. And based on that flag then set the C bit in pmap_enter and friends. Still need to keep dmap in sync.