Below is a large portion of those different avant garde tracks -- including their track titles, the artists who did them, and at which points they are used in the film.
(Spoilers should be assumed from here to the end of the post.)
Family Hanging Out '11 / BBQ '79
Different parts of this 24-minute track are used twice: the first time is when the family is being hung from a tree branch in their own backyard, and the second is when the family is burned in their garage. The BBQ '79 portion of the song contains the infamous and wailing vocalizations.
Song: Silence Teaches You How To Sing
Pool Party '66
A family is tied to lawn chairs and pulled into a pool one by one.
Song: A Body of Water
Sleepy Time '96
Each family member slowly has his/her throat cut.
Lawn Work '86
Song: Fragment # 9
The deranged lawn mower scene.
House Painting '12
Artist: Sunn O))) & Boris
Song: Blood Swamp
Things don't end so well for Ellison and his family with the final "footage" track that plays as his young daughter paints the house in her family's blood.
Artist: Aghast Manor
Song: Call from the Grave
Ellison discovers the attic drawings.
Song: Enter the Hall of Ice
Ellison sees Bughuul in the backyard.
Packing / End Titles
Artist: Boards of Canada
Ellison burns the home movies and his family begins packing to leave, as well as makes a reprisal during the end credits. This is the song with the very unusual dragging drum beat.