The following is evidence that Mu’awiyah’s true goal was not to become the caliph, but rather, his right for avenging Uthman.

From Siyar A’alam Al-Nubala:

قال الجعفي: حدثنا يعلى بن عبيد، عن أبيه، قال: جاء أبو مسلم الخولاني وأناس إلى معاوية، وقالوا: أنت تنازع عليا أم أنت مثله ؟ فقال: لا والله، إني لاعلم أنه أفضل مني وأحق بالامر مني، ولكن ألستم تعلمون أن عثمان قتل مظلوما، وأنا ابن عمه، والطالب بدمه، فائتوه، فقولوا له، فليدفع إلي قتلة عثمان، وأسلم له.
فأتوا عليا، فكلموه، فلم يدفعهم إليه

Rough translation: Al-Ju’fi (Yahya bin Sulaiman, from his book “Siffeen”): Ya’la bin Ubaid, from his father: Abu Muslim Al-Khawlani and a group of people entered upon Mu’awiyah, and they asked, “Do you dispute Ali? Are you his equal?” He replied, “No, I am not, and I know that he is better than me, and deserves this (khilafa) more than me, but don’t you know that Uthman was killed unjustly, and that I am his cousin, and that I ask for his blood? So go to him (Ali), and tell him to bring forth the killers of Uthman, and I will submit to him.” So, they went to Ali, and spoke to him, but he didn’t hand them (the killers).

Ironically, the same narration can be found in Shia sources. See Waq’at Siffeen by Nasr bin Muzahim:

وإن أبا مسلم الخولانى قدم إلى معاوية في أناس من قراء أهل الشام، [ قبل مسير أمير المؤمنين عليه السلام إلى صفين، ] فقالوا [ له ]: يا معاوية علام تقاتل عليا، وليس لك مثل صحبته ولا هجرته ولا قرابته ولا سابقته ؟ قال لهم: ما أقاتل عليا وأنا أدعى أن لى في الإسلام مثل صحبته ولا هجرته ولا قرابته ولا سابقته، ولكن خبروني عنكم، ألستم تعلمون أن عثمان قتل مظلوما ؟ قالوا: بلى.
قال: فليدع إلينا قتلته فنقتلهم به، ولا قتال بيننا وبينه.

Rough translation: And Abu Muslim Al-Khawlani came to Mu’awiya with a group of qura’a from the people of Al-Shaam, before Ali went to Siffeen. They said, “Why do you fight Ali? You are not in his level of suhba or hijra or closeness or earliness (in Islam). He said, “I don’t fight Ali with the claim that I am like him in suhba, hijra, closeness or earliness, but tell me, aren’t you aware that Uthman was killed unjustly?” They said, “Yes.” He said, “They if he gives us the killers, we’ll kill them, and there will be nothing between us and him (Ali).”

Also, interestingly, the following, by Al-Tabari implies that Mu’awiyah was never seen as a caliph, until after the death of Ali, nor was he given bay’a:

(وفى هذه السنة) بويع لمعاوية بالخلافة بايلياء حدثنى بذلك موسى بن عبد الرحمن قال حدثنا عثمان بن عبد الرحمن قال أخبرنا اسماعيل بن راشد وكان قبل يدعى بالشأم أميرا وحدثت عن أبى مسهر عن سعيد بن عبد العزيز قال كان على عليه السلام يدعى بالعراق أمير المؤمنين وكان معاوية يدعى بالشأم الامير فلما قتل على عليه السلام دعى معاوية أمير المؤمنين

Rough translation: In this year, Mu’awiyah was given baya’a in Ilya’a. Musa bin Abdulrahman said that Uthman bin Abdulrahman said, Isma’eel bin Rashid told us that “he (Mu’awiyah) used to be called the ameer of Al-Shaam.” I was told that Abu Mushir that Sa’eed bin Abdulaziz said, “Ali used to be called Ameer Al-Mu’mineen in Iraq, and Mu’awiyah was called the ameer in Al-Shaam, but when Ali was killed, Mu’awiyah was later called Ameer Al-Mu’mineen.”

Also notice that Al-Tabari didn’t say that Mu’awiyah was given baya’a before this year. This implies that nobody gave Mu’awiyah a baya’a for the khilafa before this time.

Ibn Katheer also says, “When Ali died, the people of Al-Shaam pledged their allegiance to Mu’awiyah, as the Ameer Al-Mu’mineen, for there was nobody that could dispute them.”