[RISCV] Add target feature to force-enable atomics

Authored by nikic on Jul 27 2022, 2:29 AM.

Restricted Project
Restricted Project

Subscribers

Description

[RISCV] Add target feature to force-enable atomics

This adds a +forced-atomics target feature with the same semantics
as +atomics-32 on ARM (D130480). For RISCV targets without the +a
extension, this forces LLVM to assume that lock-free atomics
(up to 32/64 bits for riscv32/64 respectively) are available.

This means that atomic load/store are lowered to a simple load/store
(and fence as necessary), as these are guaranteed to be atomic
(as long as they're aligned). Atomic RMW/CAS are lowered to sync
(rather than atomic) libcalls. Responsibility for providing the
__sync libcalls lies with the user (for privileged single-core code
they can be implemented by disabling interrupts). Code using
+forced-atomics and -forced-atomics are not ABI compatible if atomic
variables cross the ABI boundary.

For context, the difference between sync and atomic is that the
former are required to be lock-free, while the latter requires a
shared global lock provided by a shared object library. See
https://llvm.org/docs/Atomics.html#libcalls-atomic for a detailed
discussion on the topic.

This target feature will be used by Rust's riscv32i target family
to support the use of atomic load/store without atomic RMW/CAS.

Differential Revision: https://reviews.llvm.org/D130621

rGf5ed0cb217a9

llvm/docs/Atomics.rst

Show First 20 Lines • Show All 575 Lines • ▼ Show 20 Lines
			(which looks atomic so long as there's only one CPU), and contains actual atomic
			instructions on newer multicore models. This sort of functionality can typically
			be provided on any architecture, if all CPUs which are missing atomic
			compare-and-swap support are uniprocessor (no SMP). This is almost always the
			case. The only common architecture without that property is SPARC -- SPARCV8 SMP
			systems were common, yet it doesn't support any sort of compare-and-swap
			operation.

			+ Some targets (like RISCV) support a ``+forced-atomics`` target feature, which
			+ enables the use of lock-free atomics even if LLVM is not aware of any specific
			+ OS support for them. In this case, the user is responsible for ensuring that
			+ necessary ``__sync_*`` implementations are available. Code using
			+ ``+forced-atomics`` is ABI-incompatible with code not using the feature, if
			+ atomic variables cross the ABI boundary.
			+
			In either of these cases, the Target in LLVM can claim support for atomics of an
			appropriate size, and then implement some subset of the operations via libcalls
			to a ``__sync_`` function. Such functions must* not use locks in their
			implementation, because unlike the ``__atomic_*`` routines used by
			AtomicExpandPass, these may be mixed-and-matched with native instructions by the
			target lowering.

			Further, these routines do not need to be shared, as they are stateless. So,
▲ Show 20 Lines • Show All 52 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCV.td

Show First 20 Lines • Show All 475 Lines • ▼ Show 20 Lines
			def TuneNoDefaultUnroll
			: SubtargetFeature<"no-default-unroll", "EnableDefaultUnroll", "false",
			"Disable default unroll preference.">;

			def TuneSiFive7 : SubtargetFeature<"sifive7", "RISCVProcFamily", "SiFive7",
			"SiFive 7-Series processors",
			[TuneNoDefaultUnroll]>;

			+ // Assume that lock-free native-width atomics are available, even if the target
			+ // and operating system combination would not usually provide them. The user
			+ // is responsible for providing any necessary __sync implementations. Code
			+ // built with this feature is not ABI-compatible with code built without this
			+ // feature, if atomic variables are exposed across the ABI boundary.
			+ def FeatureForcedAtomics : SubtargetFeature<
			+ "forced-atomics", "HasForcedAtomics", "true",
			+ "Assume that lock-free native-width atomics are available">;
			+ def HasAtomicLdSt
			+ : Predicate<"Subtarget->hasStdExtA() \|\| Subtarget->hasForcedAtomics()">;
			+
			//===----------------------------------------------------------------------===//
			// Named operands for CSR instructions.
			//===----------------------------------------------------------------------===//

			include "RISCVSystemOperands.td"

			//===----------------------------------------------------------------------===//
			// Registers, calling conventions, instruction descriptions.
▲ Show 20 Lines • Show All 118 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

This file is larger than 256 KB, so syntax highlighting is disabled by default.

Show First 20 Lines • Show All 405 Lines • ▼ Show 20 Lines
			setOperationAction({ISD::TRAP, ISD::DEBUGTRAP}, MVT::Other, Legal);
			setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::Other, Custom);
			if (Subtarget.is64Bit())
			setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i32, Custom);

			if (Subtarget.hasStdExtA()) {
			setMaxAtomicSizeInBitsSupported(Subtarget.getXLen());
			setMinCmpXchgSizeInBits(32);
			+ } else if (Subtarget.hasForcedAtomics()) {
			+ setMaxAtomicSizeInBitsSupported(Subtarget.getXLen());
			} else {
			setMaxAtomicSizeInBitsSupported(0);
			}

			setBooleanContents(ZeroOrOneBooleanContent);

			if (Subtarget.hasVInstructions()) {
			setBooleanVectorContents(ZeroOrOneBooleanContent);
▲ Show 20 Lines • Show All 502 Lines • ▼ Show 20 Lines
			setOperationAction(ISD::BITCAST, MVT::f16, Custom);
			if (Subtarget.hasStdExtF())
			setOperationAction(ISD::BITCAST, MVT::f32, Custom);
			if (Subtarget.hasStdExtD())
			setOperationAction(ISD::BITCAST, MVT::f64, Custom);
			}
			}

			+ if (Subtarget.hasForcedAtomics()) {
			+ // Set atomic rmw/cas operations to expand to force __sync libcalls.
			+ setOperationAction(
			+ {ISD::ATOMIC_CMP_SWAP, ISD::ATOMIC_SWAP, ISD::ATOMIC_LOAD_ADD,
			+ ISD::ATOMIC_LOAD_SUB, ISD::ATOMIC_LOAD_AND, ISD::ATOMIC_LOAD_OR,
			+ ISD::ATOMIC_LOAD_XOR, ISD::ATOMIC_LOAD_NAND, ISD::ATOMIC_LOAD_MIN,
			+ ISD::ATOMIC_LOAD_MAX, ISD::ATOMIC_LOAD_UMIN, ISD::ATOMIC_LOAD_UMAX},
			+ XLenVT, Expand);
			+ }
			+
			// Function alignments.
			const Align FunctionAlignment(Subtarget.hasStdExtC() ? 2 : 4);
			setMinFunctionAlignment(FunctionAlignment);
			setPrefFunctionAlignment(FunctionAlignment);

			setMinimumJumpTableEntries(5);

			// Jumps are expensive, compared to logic
▲ Show 20 Lines • Show All 11,341 Lines • ▼ Show 20 Lines
			TargetLowering::AtomicExpansionKind
			RISCVTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
			// atomicrmw {fadd,fsub} must be expanded to use compare-exchange, as floating
			// point operations can't be used in an lr/sc sequence without breaking the
			// forward-progress guarantee.
			if (AI->isFloatingPointOperation())
			return AtomicExpansionKind::CmpXChg;

			+ // Don't expand forced atomics, we want to have __sync libcalls instead.
			+ if (Subtarget.hasForcedAtomics())
			+ return AtomicExpansionKind::None;
			+
			unsigned Size = AI->getType()->getPrimitiveSizeInBits();
			if (Size == 8 \|\| Size == 16)
			return AtomicExpansionKind::MaskedIntrinsic;
			return AtomicExpansionKind::None;
			}

			static Intrinsic::ID
			getIntrinsicForMaskedAtomicRMWBinOp(unsigned XLen, AtomicRMWInst::BinOp BinOp) {
▲ Show 20 Lines • Show All 87 Lines • ▼ Show 20 Lines
			if (XLen == 64)
			Result = Builder.CreateTrunc(Result, Builder.getInt32Ty());
			return Result;
			}

			TargetLowering::AtomicExpansionKind
			RISCVTargetLowering::shouldExpandAtomicCmpXchgInIR(
			AtomicCmpXchgInst *CI) const {
			+ // Don't expand forced atomics, we want to have __sync libcalls instead.
			+ if (Subtarget.hasForcedAtomics())
			+ return AtomicExpansionKind::None;
			+
			unsigned Size = CI->getCompareOperand()->getType()->getPrimitiveSizeInBits();
			if (Size == 8 \|\| Size == 16)
			return AtomicExpansionKind::MaskedIntrinsic;
			return AtomicExpansionKind::None;
			}

			Value *RISCVTargetLowering::emitMaskedAtomicCmpXchgIntrinsic(
			IRBuilderBase &Builder, AtomicCmpXchgInst CI, Value AlignedAddr,
▲ Show 20 Lines • Show All 391 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

Show First 20 Lines • Show All 99 Lines • ▼ Show 20 Lines
			defm AMOMAXU_D : AMO_rr_aq_rl<0b11100, 0b011, "amomaxu.d">,
			Sched<[WriteAtomicD, ReadAtomicDA, ReadAtomicDD]>;
			} // Predicates = [HasStdExtA, IsRV64]

			//===----------------------------------------------------------------------===//
			// Pseudo-instructions and codegen patterns
			//===----------------------------------------------------------------------===//

			- let Predicates = [HasStdExtA] in {
			-
			- /// Atomic loads and stores
			-
			+ // Atomic load/store are available under both +a and +force-atomics.
			// Fences will be inserted for atomic load/stores according to the logic in
			// RISCVTargetLowering::{emitLeadingFence,emitTrailingFence}.
			-
			- defm : LdPat<atomic_load_8, LB>;
			- defm : LdPat<atomic_load_16, LH>;
			- defm : LdPat<atomic_load_32, LW>;
			+ let Predicates = [HasAtomicLdSt] in {
			+ defm : LdPat<atomic_load_8, LB>;
			+ defm : LdPat<atomic_load_16, LH>;
			+ defm : LdPat<atomic_load_32, LW>;

			- defm : AtomicStPat<atomic_store_8, SB, GPR>;
			- defm : AtomicStPat<atomic_store_16, SH, GPR>;
			- defm : AtomicStPat<atomic_store_32, SW, GPR>;
			+ defm : AtomicStPat<atomic_store_8, SB, GPR>;
			+ defm : AtomicStPat<atomic_store_16, SH, GPR>;
			+ defm : AtomicStPat<atomic_store_32, SW, GPR>;
			+ }
			+
			+ let Predicates = [HasAtomicLdSt, IsRV64] in {
			+ defm : LdPat<atomic_load_64, LD, i64>;
			+ defm : AtomicStPat<atomic_store_64, SD, GPR, i64>;
			+ }
			+
			+ let Predicates = [HasStdExtA] in {

			/// AMOs

			multiclass AMOPat<string AtomicOp, string BaseInst> {
			def : PatGprGpr<!cast<PatFrag>(AtomicOp#"_monotonic"),
			!cast<RVInst>(BaseInst)>;
			def : PatGprGpr<!cast<PatFrag>(AtomicOp#"_acquire"),
			!cast<RVInst>(BaseInst#"_AQ")>;
▲ Show 20 Lines • Show All 169 Lines • ▼ Show 20 Lines
			GPR:$addr, GPR:$cmpval, GPR:$newval, GPR:$mask, timm:$ordering),
			(PseudoMaskedCmpXchg32
			GPR:$addr, GPR:$cmpval, GPR:$newval, GPR:$mask, timm:$ordering)>;

			} // Predicates = [HasStdExtA]

			let Predicates = [HasStdExtA, IsRV64] in {

			- /// 64-bit atomic loads and stores
			-
			- // Fences will be inserted for atomic load/stores according to the logic in
			- // RISCVTargetLowering::{emitLeadingFence,emitTrailingFence}.
			- defm : LdPat<atomic_load_64, LD, i64>;
			- defm : AtomicStPat<atomic_store_64, SD, GPR, i64>;
			-
			defm : AMOPat<"atomic_swap_64", "AMOSWAP_D">;
			defm : AMOPat<"atomic_load_add_64", "AMOADD_D">;
			defm : AMOPat<"atomic_load_and_64", "AMOAND_D">;
			defm : AMOPat<"atomic_load_or_64", "AMOOR_D">;
			defm : AMOPat<"atomic_load_xor_64", "AMOXOR_D">;
			defm : AMOPat<"atomic_load_max_64", "AMOMAX_D">;
			defm : AMOPat<"atomic_load_min_64", "AMOMIN_D">;
			defm : AMOPat<"atomic_load_umax_64", "AMOMAXU_D">;
▲ Show 20 Lines • Show All 59 Lines • Show Last 20 Lines

llvm/lib/Target/RISCV/RISCVSubtarget.h

Show First 20 Lines • Show All 92 Lines • ▼ Show 20 Lines
			bool HasRV64 = false;
			bool IsRV32E = false;
			bool EnableLinkerRelax = false;
			bool EnableRVCHintInstrs = true;
			bool EnableDefaultUnroll = true;
			bool EnableSaveRestore = false;
			bool EnableUnalignedScalarMem = false;
			bool HasLUIADDIFusion = false;
			+ bool HasForcedAtomics = false;
			unsigned XLen = 32;
			unsigned ZvlLen = 0;
			MVT XLenVT = MVT::i32;
			uint8_t MaxInterleaveFactor = 2;
			RISCVABI::ABI TargetABI = RISCVABI::ABI_Unknown;
			BitVector UserReservedRegister;
			RISCVFrameLowering FrameLowering;
			RISCVInstrInfo InstrInfo;
▲ Show 20 Lines • Show All 80 Lines • ▼ Show 20 Lines
			bool is64Bit() const { return HasRV64; }
			bool isRV32E() const { return IsRV32E; }
			bool enableLinkerRelax() const { return EnableLinkerRelax; }
			bool enableRVCHintInstrs() const { return EnableRVCHintInstrs; }
			bool enableDefaultUnroll() const { return EnableDefaultUnroll; }
			bool enableSaveRestore() const { return EnableSaveRestore; }
			bool enableUnalignedScalarMem() const { return EnableUnalignedScalarMem; }
			bool hasLUIADDIFusion() const { return HasLUIADDIFusion; }
			+ bool hasForcedAtomics() const { return HasForcedAtomics; }
			MVT getXLenVT() const { return XLenVT; }
			unsigned getXLen() const { return XLen; }
			unsigned getFLen() const {
			if (HasStdExtD)
			return 64;

			if (HasStdExtF)
			return 32;
▲ Show 20 Lines • Show All 79 Lines • Show Last 20 Lines

llvm/test/CodeGen/RISCV/forced-atomics.ll

rGf5ed0cb217a9

llvm/docs/Atomics.rst

llvm/lib/Target/RISCV/RISCV.td

llvm/lib/Target/RISCV/RISCVISelLowering.cpp

llvm/lib/Target/RISCV/RISCVInstrInfoA.td

llvm/lib/Target/RISCV/RISCVSubtarget.h

llvm/test/CodeGen/RISCV/forced-atomics.ll

Recommend

移动 5G 比 4G 耗电的直观对比，以及寻求一种 5G MIFI 产品

ROG B760小吹雪主板DDR5版上架支持13代酷睿处理器

播客 | 来一场环球旅行：面对忐忑、享受快乐、体会成长

The Biggest Change On The 2023 Sequoia That Some Toyota Fans Aren't Happy About

User Guide for RISC-V Target — LLVM 16.0.0 documentation

出7制胜！第二代骁龙7+发布

5 Budget Sports Cars You're Probably Too Late To Get A Good Deal On

rG4fc7e9cba24b

GitHub - riscv-non-isa/rvv-intrinsic-doc

The first NeuroNet for DeFi cryptocurrency trading created on the basis of Neuro...

About Joyk