Abernathy chronicles his personal experience in the civil rights movement in his autobiography, And the Walls Came Tumbling Down, published by HarperCollins. In the book, Mr. Abernathy gives an insider account of the Civil Rights movement detailing the organization of the Montgomery bus boycott of 1955 and the 1965 March in Selma. Personally criticized for his account of the personal life of his close friend, Dr. Martin Luther King, Abernathy takes the opportunity to explain his motives and his view of the proper accounting of Dr. King and civil rights history.

Also, he discusses his controversial endorsement of Ronald Reagan in the 1980 presidential elections and his subsequent frustrating efforts to communicate with the administration. Additionally, he comments on the status of current black leadership in America.