Preheat oven to 350 degrees F (175 degrees C). Spray a 9x13-inch casserole dish with cooking spray.

Bring a large pot of salted water to a boil; cook squash in the boiling water until very tender, 10 to 15 minutes. Drain and transfer squash to a large bowl; mash until evenly smooth.

Heat butter in a skillet over medium heat; cook and stir onion in the melted butter until tender, 5 to 10 minutes. Mix onion, cream of mushroom soup, Cheddar cheese, bread crumbs, milk, mayonnaise, sugar, eggs, salt, and pepper into mashed squash. Spread mixture evenly into the prepared casserole dish.

Bake in the preheated oven until bubbling and edges are lightly browned, 45 minutes to 1 hour.