2 itemTouchHelpers for 2 recyclerviews not working

Question

Context: WorkoutStartFragment hosts a RecyclerView (For exercises within the workout) that hosts a RecyclerView (For sets). Each RecyclerView has their own itemTouchHelper. For the exercise itemTouchHelper (1st RecyclerView), it only allows a drag and drop feature for moving around exercises up and down to relocate them. For the set itemTouchHelper (2nd RecyclerView), it only allows a swipe feature for swiping away sets to delete them.

Issue: After dragging and dropping an exercise, followed by adding a set then removing the set via swiping, a crash occurs, giving an IndexOutOfBoundsException. This occurs within SetAdapter's removeSet (Called by onswiped) here, as position is outside of sets:

SetAdapter's itemTouchHelper (Code in WorkoutExerciseAdapter):

// Enable drag-and-drop and swipe-to-delete for sets
val itemTouchHelperCallback = object : ItemTouchHelper.SimpleCallback(0, ItemTouchHelper.RIGHT or ItemTouchHelper.LEFT) {
    override fun onMove(recyclerView: RecyclerView, viewHolder: RecyclerView.ViewHolder, target: RecyclerView.ViewHolder): Boolean {
        return false
    }

    override fun onSwiped(viewHolder: RecyclerView.ViewHolder, direction: Int) {
        val setPosition = viewHolder.bindingAdapterPosition
        setAdapter.removeSet(setPosition)
    }
}

val itemTouchHelper = ItemTouchHelper(itemTouchHelperCallback)
itemTouchHelper.attachToRecyclerView(holder.setsRecyclerView)

WorkoutExerciseAdapter's itemTouchHelper (Code in WorkoutStartFragment)

// Initialize itemTouchHelper with custom callback
val callback = AutoScrollItemTouchHelperCallback(
    adapter, recyclerView, onDragEnd = {
        updateExerciseOrder() // Update exercise order when drag ends
        adapter.setExercises(selectedExercises)
    }
)
itemTouchHelper = ItemTouchHelper(callback)
itemTouchHelper.attachToRecyclerView(recyclerView)

AutoScrollItemTouchHelperCallback:

class AutoScrollItemTouchHelperCallback(
    private val adapter: ItemTouchHelperAdapter,
    private val recyclerView: RecyclerView,
    private val onDragEnd: () -> Unit,
) : ItemTouchHelper.Callback() {

    private val scrollSpeed = 20

    override fun getMovementFlags(
        recyclerView: RecyclerView,
        viewHolder: RecyclerView.ViewHolder
    ): Int {
        val dragFlags = ItemTouchHelper.UP or ItemTouchHelper.DOWN
        val swipeFlags = 0
        return makeMovementFlags(dragFlags, swipeFlags)
    }

    override fun onMove(
        recyclerView: RecyclerView,
        viewHolder: RecyclerView.ViewHolder,
        target: RecyclerView.ViewHolder
    ): Boolean {
        adapter.onItemMove(viewHolder.bindingAdapterPosition, target.bindingAdapterPosition)
        return true
    }

    override fun isItemViewSwipeEnabled(): Boolean {
        return true
    }

    override fun onSwiped(viewHolder: RecyclerView.ViewHolder, direction: Int) {

    }

    override fun isLongPressDragEnabled(): Boolean {
        return true
    }

    override fun onSelectedChanged(viewHolder: RecyclerView.ViewHolder?, actionState: Int) {
        super.onSelectedChanged(viewHolder, actionState)
        if (actionState == ItemTouchHelper.ACTION_STATE_DRAG) {
            viewHolder?.itemView?.alpha = 0.7f
        } else if (actionState == ItemTouchHelper.ACTION_STATE_IDLE) {
            viewHolder?.itemView?.alpha = 1.0f
        }
    }

    override fun clearView(recyclerView: RecyclerView, viewHolder: RecyclerView.ViewHolder) {
        super.clearView(recyclerView, viewHolder)
        viewHolder.itemView.alpha = 1.0f
        onDragEnd()
    }

    override fun onChildDraw(
        c: Canvas,
        recyclerView: RecyclerView,
        viewHolder: RecyclerView.ViewHolder,
        dX: Float,
        dY: Float,
        actionState: Int,
        isCurrentlyActive: Boolean
    ) {
        super.onChildDraw(c, recyclerView, viewHolder, dX, dY, actionState, isCurrentlyActive)
        if (actionState == ItemTouchHelper.ACTION_STATE_DRAG && isCurrentlyActive) {
            autoScroll(viewHolder)
        }
    }

    private fun autoScroll(viewHolder: RecyclerView.ViewHolder) {
        val layoutManager = recyclerView.layoutManager as LinearLayoutManager
        val viewTop = viewHolder.itemView.top
        val viewBottom = viewHolder.itemView.bottom
        val parentHeight = recyclerView.height

        if (viewTop < 0) {
            recyclerView.scrollBy(0, -scrollSpeed)
        } else if (viewBottom > parentHeight) {
            recyclerView.scrollBy(0, scrollSpeed)
        }
    }

}

Demonstration:
Upon swiping a set, the following will run.

Set Adapter's RemoveSet

fun removeSet(position: Int) {
    sets.removeAt(position)
    completions.removeAt(position) // Remove corresponding completion status
    renumberSets()
    notifyItemRemoved(position)
    notifyItemRangeChanged(position, sets.size)
    listener.onSetRemoved(sets)
}

Set Adapter's Interface

class SetAdapter(
    private val sets: MutableList<Triple<Float, Int, String>>,
    private val completions: MutableList<Boolean>,
    private val listener: OnSetChangeListener,
    private val areCheckboxesEnabled: Boolean
) : RecyclerView.Adapter<SetAdapter.SetViewHolder>() {

    interface OnSetChangeListener {
        fun onSetChanged(setIndex: Int, weight: Float, reps: Int)
        fun onSetRemoved(updatedSets: List<Triple<Float, Int, String>>) //This
        fun onSetTypeChanged(setIndex: Int, setType: String)
        fun onSetCompletionChanged(setIndex: Int, isComplete: Boolean)
    }

Calling Set Adapter (From WorkoutExerciseAdapter)

override fun onBindViewHolder(holder: WorkoutExerciseViewHolder, position: Int) {
        val workoutExercise = exercises[holder.bindingAdapterPosition]

        // Set up the nested RecyclerView for sets
        val sets = workoutExercise.weights.zip(workoutExercise.reps).mapIndexed { index, pair ->
            Triple(pair.first, pair.second, workoutExercise.sets.getOrNull(index) ?: "")
        }.toMutableList()

        Log.d("WorkoutExerciseAdapter", "onBindViewHolder - Exercise: ${workoutExercise.title}, Sets: ${workoutExercise.sets}")

        val setAdapter = SetAdapter(sets, workoutExercise.completion,
        object : SetAdapter.OnSetChangeListener {
            override fun onSetChanged(setIndex: Int, weight: Float, reps: Int) {
            }

            override fun onSetRemoved(updatedSets: List<Triple<Float, Int, String>>) {
                //this
                if (holder.bindingAdapterPosition != RecyclerView.NO_POSITION) {
                    workoutExercise.sets.clear()
                    workoutExercise.weights.clear()
                    workoutExercise.reps.clear()
                    updatedSets.forEach { (weight, reps, setType) ->
                        workoutExercise.weights.add(weight)
                        workoutExercise.reps.add(reps)
                        workoutExercise.sets.add(setType)
                    }
                }
            }

This Log:

Log.d("WorkoutExerciseAdapter", "onBindViewHolder - Exercise: ${workoutExercise.title}, Sets: ${workoutExercise.sets}")

Shows that sets are correct upon dragging and dropping or anytime exercises are bound in the workoutExerciseAdapter. Within SetAdapter, sets are the same, until it comes to removeSet (Within SetAdapter). Logging removeSet shows that sets are what it was before dragging and dropping, despite
1. All sets being rebound after drag and drop
2. Adding sets also being bound after drag and drop
3. Logging Set Adapter and finding that all sets are correct in anywhere outside the removeSet function, even in other functions

My belief is that this occurs because the two itemTouchHelpers may have their functions collide with one another, or perhaps only 1 can be activated at a time. Without dragging and dropping, everything works fine. Additionally, I experimented with using a button to remove a set instead, and it works, meaning that it is likely an itemTouchHelper issue. I'm not sure how I'm supposed to fix this issue.